Parquet vs CSV: five signals it is time to switch
Nobody should switch file formats because a blog post said columnar is better. You switch when the format you have is costing you something specific. Here are the five signals I treat as decisive, and, just as importantly, the four cases where staying on CSV is the right call.
Signal 1: you re-declare the same types every single time
If your import step has a dictionary mapping fifteen column names to types, and you paste it into every notebook, that dictionary is a schema. It is a schema stored in your code instead of in the file, which means it drifts, gets copied wrong, and has to be maintained by whoever remembers it exists.
Parquet stores the schema in the file. A date column comes back as a date, a decimal as a decimal, a zip code as a string with its leading zeros intact. Every one of the problems in the rest of this section of the site, the stripped zeros, the scientific notation, the swapped day and month, exists because CSV cannot carry that information.
Fix it: convert once and let the file carry its own types →
Signal 2: you read three columns out of forty
This is the structural argument, and it produces the largest speedups.
CSV is row oriented. To read the
revenue column you
have to read every byte of every row, because the values you want are scattered through
the file with everything else in between. Parquet stores each column contiguously, so a
reader seeks to the revenue chunk and reads only that. On a wide table this is not a ten
percent improvement, it is an order of magnitude.
Parquet also stores minimum and maximum statistics per row group, which lets a query
engine skip whole blocks. A filter of
WHERE order_date >= '2026-06-01'
against a file written in date order can skip most of the file without decompressing it.
That is predicate pushdown, and CSV has no equivalent because there is nothing to push
down into.
Signal 3: storage or transfer is a line item
Three to seven times smaller is the range I see for real business data. The mechanism is worth understanding, because it tells you in advance whether your ratio will be good.
- Dictionary encoding. A region column with five distinct values across two million rows stores five strings and two million small integers. That alone can be a 90 percent reduction on that column.
- Run-length encoding. Sorted or naturally clustered columns compress to almost nothing.
- Type-appropriate storage. The number 1,234,567 is seven characters as text and four bytes as an integer.
- Block compression. Snappy by default, Zstd when you want smaller files at a little more CPU.
The corollary: a table of unique random identifiers and free text compresses poorly, because there is no repetition to exploit. If your CSV is mostly UUIDs and comment bodies, expect a modest gain rather than a dramatic one.
Fix it: convert your own file and compare the two sizes →
Signal 4: you have been bitten by precision loss
If you have ever had an 18-digit identifier rounded, or a currency amount arrive as 12.340000000000001, the type system is the problem and no amount of care at the application layer fixes it.
Parquet has a real DECIMAL type with declared precision and scale, so money is stored exactly rather than as a float that happens to be close. It has 64-bit integers, so a Snowflake-style identifier survives. It distinguishes a date from a timestamp from a timestamp with a time zone, which removes the entire class of "the daily total landed in the wrong day" problems.
None of that is available in CSV at any price, because CSV has exactly one type and it is text.
Fix it: the precision guide, if you are already dealing with the damage →
Signal 5: parsing dominates your runtime
Time a job. If most of the wall clock goes on turning text into values rather than doing anything with them, the format is your bottleneck. Parsing CSV means scanning for delimiters, honoring quoting, decoding text and inferring or casting types, per value. Reading Parquet means decompressing a block and reading typed values that are already in a layout close to what memory wants.
This shows up in our own code. Both directions of Parquet conversion go through a lazily loaded DuckDB build, and the reason is exactly this: writing a correct, fast Parquet reader is a great deal of work, and DuckDB already reads one into Arrow buffers with the types intact.
When CSV is still the right answer
I use CSV constantly and would not want to be talked out of it. Four cases where it wins outright:
| Situation | Why CSV wins |
|---|---|
| A person will open it | Double-clickable, readable in any editor, no tooling required |
| Small files | Under a few thousand rows, Parquet's overhead can make the file larger |
| Unknown recipient | Every system on earth reads CSV; Parquet support is good but not universal |
| Appending rows | A CSV takes a line on the end; a Parquet file has to be rewritten or partitioned |
That last one deserves emphasis, because it is the least discussed. Parquet's footer holds the metadata for the whole file, so you cannot simply append to it. Streaming workloads handle this by writing many small files into a partitioned directory, which is a real architecture and not a five-minute change. If your process is "add today's rows to the end of the file", CSV is doing something Parquet cannot do cheaply.
Check what is really in the file
Before committing to a migration, run a cheap experiment on real data. It takes ten minutes and it answers the question for your data rather than for somebody's benchmark.
- Convert a representative file. Not the first thousand rows; a real one, because compression ratios depend on cardinality across the whole file.
- Compare the sizes. Under 2x, the storage argument is weak and you should decide on types alone.
- Time your actual query. Whatever aggregation you run most often, run it against both.
- Read the types back. Open the Parquet file and look at its schema. If a decimal came through as a float, fix the conversion before you migrate anything.
A middle path worth naming: keep both. Write Parquet as the working format for everything internal, and generate CSV on demand for the people who need a spreadsheet. That is usually cheaper than converting anyone's habits.