CSV to Arrow Converter

A CSV to Arrow converter writes an Arrow IPC file that pyarrow, polars, pandas and DuckDB load directly, with types already decided. This one uses the file layout with a footer, so the reader call most people try first actually works, and types each column once so a padded code stays text. It builds the file in your browser and nothing is uploaded.

Need to clean or reshape the table first? Open the app

The layout that decides whether your reader works

Arrow IPC comes in two layouts and they are not interchangeable.

The file layout starts with the magic bytes ARROW1 and ends with a footer holding an index of every record batch. A reader can seek straight to batch seven without touching the first six, which is what makes it useful as a file on disk.

The stream layout has no footer, because it is designed to be written to a socket or a pipe where you cannot go back and add one. It is read front to back.

Both are legal Arrow and both usually get the extension .arrow, which is where the trouble comes from. pyarrow.ipc.open_file is the call people reach for first, and it refuses a stream-layout file with an error about a missing footer. The fix is open_stream, and you have to know that before the error makes sense.

So the file layout is the default here. If you pick the stream layout deliberately, you get a warning saying open_file will not take it, which turns a confusing error into a decision you already made.

Why Float64 and not Int64

A column of whole numbers looks like it should be Int64. It is written as Float64, and the reason is JavaScript.

Arrow's Int64 maps to BigInt in JavaScript, because a 64-bit integer does not fit in a double. BigInt is a separate numeric type with sharp edges: JSON.stringify throws on one, 1n === 1 is false, and mixing a BigInt with a number in arithmetic is a TypeError. A file written in a browser that then round-trips through anything JavaScript-shaped hits all three.

Float64 is exact for every integer up to 2 to the 53, which is about nine quadrillion, comfortably past anything that arrives in a spreadsheet. It is also what pandas and polars produce for the same data when they infer types from a CSV, so the file behaves the way a reader expects.

The warning says this out loud rather than leaving you to notice the dtype. If you genuinely need Int64 semantics, cast the column after loading: table.column("n").cast(pa.int64()) in pyarrow, or .cast(pl.Int64) in polars.

There is also an option to write every column as Utf8 and leave the typing to whatever reads the file, which is the right call when the data is going into something with its own schema and you want it to do the parsing.

Types, nulls and reading it back

Types are decided once per column, by reading the whole column rather than one cell at a time. A column where every non-empty value parses exactly becomes Float64; one where every value is true or false becomes Bool; everything else becomes Utf8.

Two cases stay text on purpose. A column of zero-padded codes like 00412, because a leading zero means an identifier and nobody adds two SKUs together. And a column mixing 9.99 with 12.50, because 12.50 as a float prints back as 12.5 and the cent is gone.

Empty cells become real nulls, recorded in Arrow's validity bitmap rather than as an empty string or a NaN. That is one of the concrete things Arrow gives you over a CSV: a missing value and an empty string are genuinely different, and both survive.

Reading it back, whichever stack you are on:

import pyarrow as pa
table = pa.ipc.open_file("products.arrow").read_all()

import polars as pl
df = pl.read_ipc("products.arrow")

import pandas as pd
df = pd.read_feather("products.arrow")

-- DuckDB
SELECT * FROM read_arrow('products.arrow');

The pandas call says feather because Feather v2 and Arrow IPC are the same format under two names. Feather v1, from before that unification, is a different and now obsolete format; a file written by it will not open with any of these.

Arrow against Parquet, briefly: Arrow IPC is the in-memory layout written straight to disk, so it loads with almost no decoding and is ideal for an intermediate file or a cache. Parquet is compressed and encoded for storage, so it is far smaller and slower to read. For a pipeline step, Arrow. For something you keep, Parquet.

Questions

What is the difference between the file and stream layouts?

The file layout has a magic header and a footer with an index of the record batches, so a reader can seek to any batch without reading the whole thing. The stream layout has no footer and is meant to be appended to a socket or a pipe. The file layout is the default here because pyarrow.ipc.open_file, which is the call most people reach for first, refuses a stream and you have to know to use open_stream instead.

Why are numeric columns Float64 rather than Int64?

Because Arrow's Int64 arrives in JavaScript as a BigInt, which cannot be serialised by JSON.stringify, does not compare equal to a plain number, and throws in arithmetic mixed with numbers. Float64 is exact for every integer up to 2 to the 53, which is beyond anything a spreadsheet holds, and it is what pandas and polars produce for the same data. If you need genuine Int64 semantics, cast the column after loading.

Is Arrow better than Parquet for this?

Different jobs. Arrow IPC is the in-memory layout written to disk, so it loads with almost no work and is ideal for handing data between processes or caching a working set. Parquet is compressed and encoded for storage, so it is much smaller on disk and slower to read. For an intermediate file in a pipeline, Arrow. For something you keep, Parquet.

How do I read the file back?

pyarrow.ipc.open_file then read_all, polars.read_ipc, pandas.read_feather, or DuckDB with read_arrow on the path. Arrow IPC and Feather v2 are the same format under two names, which is why the pandas call says feather. All of them read the file layout this writes.

Are types decided per cell or per column?

Once per column, by reading the whole column. A column where every non-empty value parses exactly becomes Float64, one where every value is true or false becomes Bool, and everything else becomes Utf8. A column of zero-padded codes stays Utf8 because a leading zero means an identifier, and a column mixing 9.99 with 12.50 stays Utf8 because 12.50 as a float prints back as 12.5.

What happens to empty cells?

They become nulls in the Arrow column, which is a real null in the validity bitmap rather than an empty string or a NaN. That is one of the things Arrow does better than CSV: a missing value and an empty string are genuinely different, and both survive the conversion as themselves.

Is the file uploaded?

No. The Arrow writer runs in your browser tab as WebAssembly-adjacent JavaScript, and the file is built in memory and handed to your downloads folder. Nothing is sent to a server, nothing is kept between visits, and there is no row cap beyond what your tab's memory allows.

Convert your CSV to Arrow

No sign-up, no upload, no row cap. The file layout with a footer, typed once per column, nulls kept as nulls.

Back to the converter