← All guides
Guide by Arif Aslam 7 min read

Your CSV is too large to open. Here is what is actually stopping you.

A colleague sends you a 900 MB export. Excel thinks about it for four minutes and then shows a dialog you half-read. Notepad opens it and stops responding. The preview app on your Mac shows the first screen and nothing else. None of those failures are the same failure, and the fix depends on which one you hit.

I spend a lot of time in files like this, and the pattern I see most is not a crash. It is a silent truncation that nobody notices until a total comes out wrong two weeks later. So this guide starts with the thing worth being paranoid about.

Symptom: the file opened, but the numbers are wrong

Cause. Excel did not refuse the file. It imported as much as fits and dropped the rest. A worksheet holds 1,048,576 rows and 16,384 columns, and when a CSV has more, Excel loads the first 1,048,576 data rows and stops. The warning appears once, in a modal, and then the workbook looks completely normal. Every SUM, every pivot, every chart you build on it is computed on a prefix of your data.

Google Sheets fails differently: it counts cells, not rows. The limit is 10 million cells per spreadsheet, so a 40-column file runs out at roughly 250,000 rows. A 7-column file gets to about 1.4 million. Same file, same day, two different ceilings depending on how wide the export is.

Fix. Get an independent row count before you trust anything. The cheapest honest check on macOS or Linux is wc -l yourfile.csv, which counts physical lines. That is not the same as a row count when fields contain embedded newlines, but if wc -l says 4.2 million and your worksheet ends at 1,048,576, you have your answer without any ambiguity at all.

Fix it: open the file in the CSV viewer and read the true row count →

Symptom: the tool refuses outright

Cause. Different tools stop for different reasons, and it is worth knowing which wall you hit, because some are hard limits and some are policy.

Tool Limit What happens at the limit
Excel worksheet 1,048,576 rows × 16,384 columns Loads a prefix, warns once, then looks normal
Google Sheets 10,000,000 cells Import fails or truncates depending on route
Notepad / TextEdit No stated cap Loads the whole file into memory as one string and stalls
ExploreMyData in-page converters 100 MB Refuses with a message and offers the full workbench
ExploreMyData workbench 1 GB, confirmation above 100 MB Parses off the main thread with a progress bar

I want to be honest about our own numbers, because vagueness here is how people lose an afternoon. The 100 MB cap on the small converter widgets is a constant in the source, not a marketing figure: past that size a single-purpose widget that holds the input, the parsed rows and the output string at once is the wrong shape of tool, so it hands you off instead. The workbench takes the same file because it does not hold those three things at once.

Fix it: load the file in the full workbench →

Symptom: the machine crawls, then the tab dies

Cause. A CSV on disk and the same CSV in memory are not the same size, and the gap is bigger than people expect. Text editors typically hold the file as UTF-16 in memory, doubling ASCII content before anything else happens. Spreadsheet applications build a cell object per value with formatting, formula and style slots attached. A 300 MB CSV with 20 million cells does not become 300 MB of RAM; it becomes several gigabytes of objects.

There is a second, sneakier multiplier: rendering. A grid that puts every row in the DOM at once is asking the browser to lay out tens of millions of nodes. That is not a parsing problem, it is a painting problem, and it will kill a tab that had plenty of memory left.

Fix. Use something that keeps the data out of the display layer. In our workbench, the file is parsed into DuckDB running as WebAssembly, the grid asks for a window of rows, and only that window exists as DOM. Scrolling a 5 million row file renders about thirty rows at a time no matter where you are in it. The parse itself runs off the main thread, which is why the page stays responsive while the progress bar moves.

Fix it: explore a large CSV without rendering all of it →

Symptom: you need it in Excel anyway

Cause. Sometimes the destination is fixed. An accountant needs it in a workbook, an upload form takes 50,000 rows at a time, a legacy importer chokes past a certain size. Splitting is the right answer, and it is also where people quietly corrupt their data.

The trap is that CSV rows are not lines. RFC 4180 allows a newline inside a quoted field, so a single logical row can span five physical lines:

order_id,notes,total
1001,"Customer asked for:
 - gift wrap
 - no invoice",249.00
1002,Standard,99.00

That file has five physical lines and two data rows. Any splitter that counts newlines will cut order 1001 in half, and both halves will be unparseable. The split -l command does exactly this, silently.

Fix. Split with a parser, and make sure every output part carries the header row. If the destination is Excel, 100,000 rows per part is a comfortable size: small enough to open instantly, large enough that you do not end up with forty files.

Fix it: CSV Splitter, header preserved on every part →

Symptom: you do not actually need to see all the rows

Cause. This is the one that saves the most time and gets skipped the most often. Most of the time the question is not "show me 4 million rows", it is "what is the total by region", or "which SKUs appear more than once", or "what does the tail of this file look like". None of those need the file open in a viewer.

Fix. Query it. A grouped aggregate over 4 million rows returns a handful of rows, and a handful of rows is trivial for any tool to display.

SELECT region,
       COUNT(*)      AS orders,
       SUM(revenue)  AS revenue
FROM data
GROUP BY region
ORDER BY revenue DESC;

That runs against the loaded file in the browser, with no server and no database to set up. If the file is genuinely too big for one machine, the answer stops being a browser at all, and I would rather say that plainly than pretend otherwise: past a few gigabytes, load it into DuckDB or Postgres locally and query it there.

Fix it: run SQL against the file in your browser →

Check what is really in the file

Before any of the fixes above, spend ninety seconds establishing ground truth. Open the file in a viewer that reads the whole thing rather than a preview of it, and write down three numbers: the row count, the column count, and the size on disk. Then compare them to whatever the failing tool told you.

  • Row count differs from your source system's export log? The export itself was truncated, and no amount of splitting will recover the missing rows.
  • Column count is 1? The file is not being parsed with the right delimiter. That is a different problem with a different fix.
  • Size on disk is enormous but the row count is small? You probably have very wide text columns, or a file with an embedded blob column.

A last note on formats. If this file arrives every week and it is always too big, the file format is the problem, not the tooling. Ask the sender for Parquet. It is typically three to seven times smaller for the same data, it carries its column types, and it can be read one column at a time.

Fix it: convert the CSV to Parquet once and stop fighting it weekly →

Questions people actually ask

Does Excel warn me when it drops rows?

It shows a dialog saying the file is not fully loaded, and that dialog is very easy to click through. After that nothing in the workbook indicates truncation. Check the last row number in the sheet against the line count of the source file before you trust a total.

Is splitting a CSV safe?

It is safe if every part keeps the header row and no row is cut in half. A split done with a text editor or the split command will break a row that contains an embedded newline inside a quoted field. Split with something that parses the CSV rather than counting bytes.

How large a CSV can a browser really handle?

Our in-page converters cap at 100 MB, which is a deliberate line, not a technical wall. The full workbench accepts up to 1 GB and asks you to confirm above 100 MB, because past that point memory pressure depends on your machine more than on the file.

Why is a 200 MB CSV slower than a 200 MB Parquet file?

CSV is text, so every value has to be parsed and typed on read, and there is no way to read one column without scanning every row. Parquet stores each column separately with its type already recorded, so reading three columns out of forty touches only those three.

Can I query a huge CSV without loading all of it?

Yes. Aggregations run in SQL over the whole file while only the result comes back to the screen. A GROUP BY on 5 million rows returns 12 rows, and 12 rows is what the grid renders.

AA

Arif Aslam

Staff engineer in Bangalore. By day at Mammoth Analytics; building ExploreMyData on the side. More on my author page or LinkedIn.

Open the file that would not open

Drop a multi-hundred-megabyte CSV into the workbench. It parses in the page, in a worker, with a real progress bar.

Open the workbench