Working with Large Files: Sampling and Limiting Rows
Your file has 2 million rows. Maybe it's a year of transaction logs, a full customer database export, or every event from a tracking system. You can't meaningfully scroll through all of it, and you don't need to. What you need is a representative slice - enough data to understand the structure, spot patterns, and decide what to do next.
ExploreMyData can handle large files because DuckDB WASM is a real analytical engine, not a JavaScript array in memory. But it's still running inside a browser tab, and a browser tab has a ceiling. The file gets loaded into that tab's memory, so subsetting doesn't shrink what you loaded, it shrinks what every subsequent step has to scan and what the grid has to render. That's still the difference between exploration that feels instant and exploration that makes you wait, and on a genuinely huge file it's the difference between finishing and crashing the tab.
Four ways to take a slice, starting with the one that's actually built for it.
Strategy 1: Sample Rows, the purpose-built one
Click the green + in the Pipeline panel and select Sample Rows from the Filter & Sort group. Three fields, and each one matters.
Sample by is a choice between
rows and
percent, and it defaults to rows.
Pick rows when you want a predictable working set: 10,000 rows is 10,000 rows whether the file has two
million or two hundred thousand. Pick percent when you want the sample to scale with the input, which is
what you want if the same pipeline will run against files of wildly different sizes. Percent is capped
at 100, sensibly enough.
Sample size is the number, read as rows or as a percentage depending on the mode above.
Random seed (optional, for reproducibility) is the field worth understanding. Leave it blank and every run of the pipeline draws different rows. Reload the page, re-run the step, hand the file to a colleague: different sample each time. Put any number in it and the same seed always produces the same sample from the same input.
The generated SQL makes the distinction visible:
SELECT * FROM events USING SAMPLE 10000 ROWS
SELECT * FROM events USING SAMPLE 10000 ROWS (RESERVOIR, 42)
SELECT * FROM events USING SAMPLE 5 PERCENT
Set a seed as a matter of habit. A sample that changes underneath you means the count you quoted in Slack this morning won't reproduce this afternoon, and you'll spend twenty minutes looking for a bug that isn't there. It costs one number in one field.
Sample or limit?
The two look interchangeable and aren't. The question to ask is whether you're trying to understand the file or estimate something about it.
Limit when you want to see the shape of the data: what the columns are, what the values look like, whether the dates parsed. The first 1,000 rows answer all of that, and they answer it faster because there's no random draw involved.
Sample the moment you start computing anything you'd repeat back to someone. An average, a distribution, a category breakdown, a null rate. The first 1,000 rows of a file sorted by date are all from January, so their average order value is January's average order value, and nothing tells you that on screen. A random 1,000 gets you an estimate of the real number.
A useful rule: if you're going to quote the result, sample. If you're just looking, limit.
Strategy 2: Limit to the first N rows
The simplest approach. If you just want to see what the data looks like - column names, types, sample values - the first 1,000 rows are usually enough.
- Select Limit Rows from the Filter & Sort group.
- Enter the number of rows in "Number of rows", say 1000.
- Apply. The grid now shows only the first 1,000 rows.
This adds a LIMIT 1000
to the pipeline. Every operation you apply after this, filters, transforms, aggregations,
runs against this smaller set, which keeps things responsive.
The panel also has Sort by (optional) and
Sort direction. Those two are more useful than they look,
because there is no standalone Sort operation in the app. Setting Sort by to a date column with
direction DESC gives you
ORDER BY created_at DESC LIMIT 1000,
which is "the thousand most recent events". Leave Sort by empty and you get whatever order the file
happened to be in.
Pipeline with Limit applied (showing first 1,000 of 2,000,000 rows):
| # | event_id | user_id | event_type | created_at | value |
|---|---|---|---|---|---|
| Showing 1,000 of 2,000,000 rows • Pipeline: [Limit: 1,000] | |||||
| 1 | EVT-00001 | USR-4421 | page_view | 2025-01-01 00:00:04 | 1 |
| 2 | EVT-00002 | USR-8803 | click | 2025-01-01 00:00:11 | 1 |
| 3 | EVT-00003 | USR-2210 | purchase | 2025-01-01 00:00:33 | 49.99 |
The full 2M-row file is loaded, but operations only run against 1,000 rows. Instant response. Remove the Limit step when ready to work with the full dataset.
The downside is the one from the previous section: the first N rows aren't random. If your file is sorted by date, you're only seeing the earliest records. If it's sorted by customer ID, you might only see customers in one region. Fine for a quick look, wrong for anything you'd put a number on.
Strategy 3: Top/Bottom N by a column
Instead of arbitrary first rows, you can take the top or bottom N rows based on a specific column. This is useful when you want to focus on extremes: the highest-revenue transactions, the most recent orders, or the oldest unresolved tickets.
- Select Top / Bottom Rows from the Filter & Sort group.
- Choose "top" or "bottom."
- Enter the number of rows (e.g., 500).
- Select the column to rank by (e.g., "revenue" or "created_at").
Top 500 by revenue gives you the biggest deals. Bottom 100 by score gives you the lowest-performing items. Top 10000 by date gives you the most recent records. Each of these is a meaningful sample for different questions.
Top 500 by revenue from the full 2M-row file - meaningful, not arbitrary:
| # | order_id | customer | revenue | order_date |
|---|---|---|---|---|
| Showing 500 of 2,000,000 rows • Pipeline: [Top 500 by revenue desc] | ||||
| 1 | ORD-198204 | Acme Corp | 142,300.00 | 2025-11-03 |
| 2 | ORD-177891 | GlobalTech | 118,050.00 | 2025-09-17 |
| 3 | ORD-203441 | Sunrise Media | 76,400.00 | 2025-12-01 |
Instead of the first 500 rows (which could all be from January), you get the 500 highest-value orders from anywhere in the 2M-row dataset.
Strategy 4: Filter to a meaningful subset
Often the best subset isn't random at all - it's a specific slice. One month of data. One region. One product category. Small enough to explore, coherent enough to draw conclusions from.
Filter by date range. Date columns get their own operator list: is, before, after, on or before, on or after, is Empty, is NOT Empty. No "greater than", no "between". A range is two conditions joined with AND:
- Select Filter from the Filter & Sort group.
- Choose the date column, say "order_date".
- Operator on or after, value 2025-10-01.
- Add a second condition in the same group: before, value 2026-01-01.
That's Q4, with the end boundary exclusive so nothing from January 1st sneaks in. Using "on or before" with 2025-12-31 works too, and is the one to pick if your column carries a time component, since "on or before 2025-12-31" excludes anything logged later that same day.
Filter by category. Text columns use the text operators, so it's is rather than "equals":
- region is "North America".
- Or status is "active" to exclude historical records.
Each filter is a pipeline step, so you can add and remove them freely. The data grid updates immediately.
Combining strategies
These stack, and the order matters. Filter first, then sample: filtering to Q4 and sampling 10,000 gives you a random 10,000 from Q4. Sampling first and then filtering gives you whatever fraction of your 10,000 happened to fall in Q4, which is a smaller and less useful number.
A typical workflow with a large file:
- Limit to 1,000 and look around. Check the type badges, read a few values, open the Column Explorer on the columns that matter. You're learning the shape of the file, and the first thousand rows are fine for that.
- Delete the Limit step. You're done looking.
- Filter to the slice you care about. A quarter, a region, a product line. Now that you know what the columns contain, you can be specific.
- Add Sample Rows with a seed. 10,000 rows, seed 42. Everything from here on runs fast and gives the same answer twice.
- Delete the sample before the final numbers. Explore on the sample, then pull the step out and let the last run go against everything. A sample is for iterating, not for reporting.
Why this matters
Working with a 2-million-row file doesn't mean you need to see all 2 million rows. Most data exploration is about building intuition: what columns exist, what the values look like, where the problems are. You can build that intuition from a well-chosen subset, then apply what you learn to the full dataset.
The pipeline makes this safe. Every limit, sample and filter is a step you can delete. You never modify the original file, and the full dataset is always one deletion away.