How a detached ArrayBuffer rewrote our file loader

There is one line in our CSV loader with a four-line comment above it, and that comment is the most expensive thing I have written this year. It says that registering a buffer with DuckDB transfers it to the worker, which detaches it, so any read after that line throws.

The bug it prevents took a while to find, because it did not look like a memory bug. It looked like some files loading and some files crashing, with no pattern I could see from the outside. This is the story of what was actually happening, and of the three other things in the loader that exist because of it.

The setup

ExploreMyData runs DuckDB compiled to WebAssembly inside the browser tab. The engine lives on a worker thread, and the page talks to it over the standard worker message channel. To let SQL read a file the user dropped, you register it under a virtual filename. There are two ways to do that.

The first is a file handle. You hand DuckDB the File object itself and it reads lazily, calling slice() on it from inside the worker as the query demands bytes. No full copy ever enters the JS or WASM heap. That is what we do for most formats:

async function registerFileHandle(database, name, file) {
  _fileHandles.set(name, file); // prevent GC
  await database.registerFileHandle(
    name,
    file,
    duckdb.DuckDBDataProtocol.BROWSER_FILEREADER,
    true,
  );
}

The _fileHandles map is not bookkeeping, it is a GC pin. DuckDB holds its own reference across the worker boundary, but the blob on our side can be collected while a lazy read is still pending, and then the read fails on a file that visibly still exists in the UI. Keeping the File alive in a map on the page is the cheapest fix, and the map doubles as the source for re-parsing a file with different options later.

The second way is a buffer. You read the whole file into a Uint8Array and register the bytes. That is what we do for CSV and TSV, for two reasons. DuckDB's dialect sniffer is more reliable with full byte access, particularly on files with mixed line endings. And since the CSV path ends in a CREATE TABLE that materializes every row anyway, lazy reading saves nothing.

The bug

Registering a buffer is not a copy. The implementation transfers the underlying ArrayBuffer to the worker, which is what makes it fast: no megabytes are duplicated, ownership simply moves. The consequence, spelled out in the structured clone algorithm, is that the sending side's buffer is left detached. Its byte length becomes zero, and every read of it throws Cannot perform Construct on a detached ArrayBuffer.

Our loader had a natural-looking order of operations. Read the file into bytes. Register the bytes. Then, to decide how to parse, take a sample of the text and look at it.

// The order that crashes
let csvBytes = new Uint8Array(await file.arrayBuffer());
await database.registerFileBuffer(virtualName, csvBytes);
const sampleText = new TextDecoder().decode(csvBytes.subarray(0, 65536)); // throws

That is the whole bug. The sample decode runs against a buffer that no longer exists on this thread.

What made it hard to see was that it did not fail for every file. The sample decode was inside a branch that only ran under certain conditions, and some code paths registered and never looked at the bytes again. So the failure was intermittent in exactly the way that makes you suspect a race condition, and it is not a race: it is deterministic, and the determinism is in a code path you have to notice you are taking.

The fix is one line moved:

// Decoded BEFORE registerFileBuffer: registration transfers the buffer to
// the DuckDB worker, which detaches it — any read after this line throws
// "Cannot perform Construct on a detached ArrayBuffer".
const sampleText = new TextDecoder("utf-8", { fatal: false }).decode(
  csvBytes.subarray(0, 64 * 1024),
);

await database.registerFileBuffer(virtualName, csvBytes);

I left the comment at four lines deliberately. The correct order looks arbitrary without it, and the next person to reorganize this function for readability would move the decode back down.

Everything that has to happen before registration

Once you know the buffer becomes unreadable at registration, the shape of the loader changes. Registration stops being a setup step and becomes a point of no return, and every decision that needs the raw bytes has to be made before you cross it.

There turned out to be three of them.

1. Encoding normalization

DuckDB's CSV reader rejects anything that is not UTF-8, and the person who just dragged in a spreadsheet export has no idea what an encoding is. So the bytes get normalized first. The probe is deliberately strict:

try {
  // Chunked so a large file never holds two full copies of itself as strings.
  const probe = new TextDecoder("utf-8", { fatal: true });
  const CHUNK = 4 * 1024 * 1024;
  for (let i = 0; i < bytes.length; i += CHUNK) {
    probe.decode(bytes.subarray(i, Math.min(i + CHUNK, bytes.length)), { stream: true });
  }
  probe.decode();
  return { bytes };
} catch {
  // BOM-less UTF-16LE looks like ASCII interleaved with NULs.
  const head = bytes.subarray(0, 1024);
  let nuls = 0;
  for (const b of head) if (b === 0) nuls++;
  const label = nuls > head.length / 4 ? "utf-16le" : "windows-1252";
  ...
}

Three things in there are load-bearing. fatal: true means the probe throws on bad bytes instead of quietly inserting replacement characters, which is the difference between detecting a problem and destroying data. The chunking means a 300 MB file never holds two full string copies of itself. And the null-density heuristic catches BOM-less UTF-16, which otherwise gets misread as Latin and produces a column name with a null byte between every letter.

The fallback is Windows-1252 rather than Latin-1 on purpose: it is the superset those files are actually in, and every one of its 256 byte values maps to a character, so the decode cannot fail. The user gets a warning saying it happened and to check that accented characters look right.

One thing this step has to skip: compressed input. Gzip and zstd bytes are not text, and the normalizer would read them as broken UTF-8 and helpfully corrupt the stream. Both the user's compression option and the magic bytes are checked, gzip at 1f 8b and zstd at 28 b5.

2. Finding the actual table

Bank and brokerage exports wrap their table in an address block above and disclaimers below. DuckDB's sniffer samples the top of the file, finds prose, and concludes that the whole thing is a one-column table. It reports success.

So before the sniffer gets a look, we look ourselves, on files under 8 MB and up to the first 5,000 lines. If a delimited region is found, the loader either sets skipRows and the detected delimiter, or, when the trailing prose has to go too, re-encodes a slice of the text so the tail never reaches DuckDB at all. A skip= option cannot do that second thing, which is why the slice path exists.

There is a rule around it: if the user has pinned the layout themselves, their options stand and only the buffer loses its tail. Auto-detection that overrides an explicit choice is worse than no auto-detection.

3. The sample for the degenerate-result check

This is the one that caused the crash, and it exists because of the next section.

A parser that never fails is a parser that loses data

DuckDB's CSV sniffer never raises its hand when it loses. A file with ragged column counts comes back either as one VARCHAR column holding entire lines, or as one enormous row with the newlines swallowed. Both parse successfully. Both are garbage.

So we check for those two shapes explicitly, against the sample text decoded before registration:

export function csvResultLooksDegenerate(columns, rowCount, sampleText) {
  const region = detectTableRegion(sampleText);
  if (!region) return false;
  if (columns.length === 1 && columns[0].name.includes(region.delimiter)) {
    return true;
  }
  // A whole file glued into one or two rows: many physical lines, almost no rows.
  if (rowCount <= 2) {
    let lines = 0;
    for (let i = 0; i < sampleText.length && lines < 10; i++) {
      if (sampleText.charCodeAt(i) === 10) lines++;
    }
    if (lines >= 10) return true;
  }
  return false;
}

The first test is the good one. A single column whose name contains the delimiter is unambiguous: the header row was not split, so nothing was. The second is cruder, a count of physical lines against the row count, and it deliberately stops counting at ten because ten is enough to decide.

Note the dependency this creates. The check needs the raw text, the raw text has to be decoded before registration, and the check runs after the query. That is the whole reason a decoded sample has to survive across the point of no return.

The rescue chain

When the strict parse fails, or produces something degenerate, the loader retries with progressively laxer options. Each attempt carries the warning it will attach if it is the one that succeeds:

const attempts = [{ extra: {} }];
if (!effectiveOptions?.nullPadding) {
  attempts.push({ extra: { nullPadding: true }, warning: PAD_WARNING });
}
if (!effectiveOptions?.ignoreErrors) {
  attempts.push({
    extra: { nullPadding: true, ignoreErrors: true },
    warning: "Some rows could not be parsed and were skipped.",
  });
}
if (!effectiveOptions?.allVarchar) {
  attempts.push({
    extra: { nullPadding: true, ignoreErrors: true, allVarchar: true },
    warning:
      "Loaded with every column as text after stricter parses failed. " +
      "Use type conversion to fix column types.",
  });
}

Two design decisions in there that I would defend.

First, options the user already set are not repeated. If someone has explicitly asked for ignore_errors, the chain does not offer it again as a rescue and does not warn about it as though it were a concession the tool made. Their choice, their consequences, no lecture.

Second, every rescue is loud. The warning travels with the file and shows up in the interface, so "some rows could not be parsed and were skipped" is a thing the user reads, not a thing that happens. The failure mode I care most about is not a crash; it is a table that looks fine and is missing four hundred rows.

What I would tell a past me

Three things came out of this that generalize past the specific bug.

Transfer semantics are API semantics. When a library takes a typed array across a thread boundary, whether it copies or transfers is part of its contract, and it is frequently undocumented. Assume transfer, and design so that everything you need from the bytes is taken before you hand them over.

A decoder configured never to fail is a data loss device. Non-fatal decoding is the default in the platform API, and it silently replaces what it cannot understand. Turning fatal on and catching the throw gives you the same robustness with a place to make a decision.

Check for success, not just for errors. The degenerate-result detector exists because an API's idea of success and a user's idea of success are different things. Any time a parse can return a shape that is technically valid and obviously wrong, that shape deserves a test.

None of this is visible from the outside, which is the point. The measure of it working is that a UTF-16 bank statement with a letterhead and four ragged rows loads, with three notices explaining what was done to it, and nobody has to know why.