Arrow to JSON Converter
An Arrow to JSON converter turns an Arrow IPC or Feather v2 file into a JSON array of objects, or into NDJSON with one object per line. The interesting part is typing: Arrow knows exactly what each column holds, JSON has only four value types to spend, and the gap between the two is where most converters quietly go wrong. The file is read in your browser.
Want to query it rather than export it? Open the app
Arrow has forty types, JSON has four
Arrow's schema is precise to a degree that JSON simply cannot match. A column is not merely a number, it is an Int64 or a Uint16 or a Decimal128 with a declared scale. A time column is not merely a date, it carries the unit its integers are counted in and often a timezone. JSON offers strings, numbers, booleans and null, and that is the whole vocabulary. Every Arrow to JSON converter is therefore making a series of judgement calls, and the useful question is which ones.
The cheap approach is to quote everything. Run the file through a reader, call String() on each value, and emit a tidy object per row where the price is "1234.56" and the row count is "18". Nothing is lost, nothing is wrong exactly, and every consumer downstream now has to parse the numbers back out by hand. Load that into pandas and the arithmetic columns arrive as objects.
The opposite mistake is to type aggressively. Coerce anything that looks numeric into a JSON number and you get clean output for most files and a data corruption bug for the rest, because JSON numbers are IEEE doubles once anything reads them. An order id of 1730629920000123456 becomes 1730629920000123400 and no error is raised at any point in the chain.
What happens here is a middle path with a specific rule. Values are read out of Arrow with the types they were stored with, then each column is examined across every one of its rows before a single character is written. A column is emitted as unquoted JSON numbers only if all of its values pass a round trip: convert to a double, convert back to text, and compare against what came out of the file. If any row in the column fails that comparison, the entire column is written as strings.
The decision being per column rather than per cell matters more than it sounds. Mixed types within one JSON field are the sort of thing that passes tests on a sample and breaks on a full extract, because the consumer inferred a schema from the first thousand rows. Deciding once for the whole column means the field you get is the field you keep.
A worked example
Take an Arrow IPC file written by pyarrow whose schema exercises most of the awkward corners at once:
event_id: Int64
amount: Decimal128(18, 2)
occurred: Timestamp[us, tz=UTC]
region: Dictionary<Int32, Utf8>
tags: List<Utf8>
retries: Int16
notes: Utf8 (nullable)Two rows of that file come out like this:
[
{
"event_id": "1730629920000123456",
"amount": 1234.56,
"occurred": "2024-11-03T09:12:00.000Z",
"region": "EMEA",
"tags": "[\"new\",\"priority\"]",
"retries": 0,
"notes": null
},
{
"event_id": "1730629920000123457",
"amount": -89.1,
"occurred": "2024-11-03T09:14:22.000Z",
"region": "AMER",
"tags": "[\"renewal\"]",
"retries": 2,
"notes": "manual review"
}
]event_id keeps its quotes, and that is the point. Nineteen digits is well past where a double can tell consecutive integers apart. The two ids above differ by one, and as JSON numbers they would both read back as the same value. As quoted digit strings they stay distinct and stay exact. This is also what most receiving systems want from an id field, since an id is a label rather than a quantity and nobody was going to add them together.
amount is an unquoted number. Every value in the column round trips through a double without moving, so the quotes come off. Note that the second row reads -89.1 rather than -89.10: a JSON number has no concept of trailing zeros, and this is the one place where becoming a real number costs you something visible. If the trailing zero is meaningful to you, because the column is currency being rendered somewhere downstream, the CSV route preserves it as written. A Decimal128 carrying more significant digits than a double can hold stays quoted automatically, so genuine high-precision decimals are never silently flattened.
occurred is ISO 8601 text. The raw stored value is 1730625120000000, an integer count of microseconds. The unit lives in the column type, not next to the number, so a converter that assumes milliseconds without checking will place that event tens of thousands of years out and produce a perfectly valid JSON document containing nonsense. The declared unit is read off the schema and applied, and the result is a string because JSON has no date type to put it in.
region was dictionary-encoded and you cannot tell. On disk that column is a run of small integers plus a lookup table, which is how Arrow keeps a low-cardinality string column cheap. Those codes are resolved against the dictionary before anything is written, so what lands in the JSON is "EMEA" rather than 2. Getting this wrong is a distinctive failure: the output is well-formed, plausible looking, and every category has been replaced by an arbitrary integer.
tags is nested, and it arrives as JSON inside a string. This is the honest caveat on the page. The reader here is the same one that feeds the CSV converter, and it serialises List, Struct and Map values to JSON text at the moment they leave the Arrow file, because a CSV cell cannot hold a structure. That escaped string carries through to the JSON output rather than being unwrapped back into a native array. Nothing is lost and one extra JSON.parse on that field gives you the array, but if you are generating a consumer from this output, size that field as a string.
notes shows how nulls are handled. An Arrow null becomes a JSON null, not an empty string. The two are different things, and collapsing them is a common annoyance when the same pipeline later has to tell a missing note from a deliberately blank one.
Shapes, layouts and what the file can be
Two output shapes are available. A single JSON array of objects is the default and is what you want if something is going to read the whole document at once. NDJSON, one complete object per line with no wrapping array and no commas between records, is the format that BigQuery loads, that jq streams without buffering, and that most log and event pipelines assume. Arrow files are frequently large enough that the difference is the difference between a load that works and one that exhausts memory. The typing rules are identical across both, so switching shape can never change a value.
On the input side, both Arrow IPC layouts are accepted. The file layout ends with a footer holding an index of the record batches; the stream layout has no footer and is meant to be read front to back. Both are routinely written with a .arrow extension, so accepting only one of them would mean rejecting valid files for reasons the person holding the file has no way to guess. That is deliberately more permissive than pyarrow.ipc.open_file, which takes the file layout alone.
A .feather file opens too, because Feather v2 and Arrow IPC are the same bytes under two names. Feather v1, from before that merge, is a genuinely separate format and will not open. The error says exactly that and suggests re-exporting as Feather v2 or as Parquet, which is more use than being told the file is invalid when it is merely old.
Arrow is binary, so the input has to be a file rather than pasted text. Everything runs inside the tab, which means nothing is uploaded and there is no row limit beyond the memory the browser gives you. If what you actually want is to look at the data rather than move it, the app will profile the columns and run SQL over them, and Arrow to CSV is the same reader pointed at a spreadsheet instead.
Questions
Are the values real JSON numbers or quoted strings?
Real numbers, wherever that is safe. Every column is examined across all of its rows before anything is written, and a column becomes unquoted JSON numbers only if every value in it survives a round trip through a JavaScript double unchanged. If even one row would shift, the whole column stays as strings. That is why a price column comes out as 1234.56 with no quotes while an identifier column keeps its quotes.
What happens to a 64-bit id column?
It comes out as a JSON string with every digit intact. An Int64 or Uint64 arrives from Arrow as a BigInt, which JSON has no representation for at all. Writing it as a number would push it through a double and quietly round the tail off a Snowflake id or a nanosecond timestamp. A quoted digit string is lossless, and it is what most JSON parsers on the other side expect for an id anyway.
How do timestamps come out?
As ISO 8601 strings, because JSON has no date type. An Arrow timestamp is stored as a bare integer count since the epoch, and whether that count is in seconds, milliseconds, microseconds or nanoseconds is recorded in the column's type rather than beside the value. That declared unit is read and applied, so a microsecond column does not land three orders of magnitude into the future. Date32 and Date64 columns come out as plain calendar dates.
What do dictionary-encoded columns look like?
Like ordinary text. A dictionary-encoded column stores small integer codes plus a lookup table of the actual values, which is how Arrow keeps a low-cardinality string column compact. The reader resolves each code against its dictionary before anything is written, so the JSON carries the label your data actually means and never the integer index. Nothing on the page needs to be told that a column was encoded that way.
How are list and struct columns represented?
As JSON text inside a string field, not as native nested arrays and objects. The Arrow reader shared with the CSV converter serialises every nested value to JSON at the point it leaves the file, so a List of Utf8 reaches the output as an escaped string. Nothing is lost and it parses cleanly with a second JSON.parse on that one field, but it is worth knowing before you write a consumer that expects a real array there.
Can I get NDJSON instead of one big array?
Yes. The output shape switches between a single JSON array of objects and one object per line, which is the format that streams into BigQuery, jq and most log pipelines without loading the whole file into memory first. Pretty-printed and minified are both available on the array shape. The typing rules are identical either way, so switching shape never changes a value.
Is my Arrow file uploaded anywhere?
No. The Arrow reader is compiled into the page and runs inside your browser tab, so the bytes are read from local disk and never cross the network. Nothing persists between visits and there is no row ceiling other than the memory the tab is given. Arrow is a binary format, so it has to arrive as a file rather than as pasted text.
Related
Convert your Arrow file to JSON
No sign-up, no upload, no row cap. Numbers typed where it is safe, ids kept whole, dictionaries resolved, timestamps in ISO 8601.
Back to the converter