A/B test calculator for a CSV

Point this at a CSV with one row per user, name the column holding the variant and the column holding the metric, and it runs Welch's t-test, a Mann-Whitney rank test and, on a conversion column, a two-proportion z-test. You get an effect size, a 95% interval and a conclusion in English. It runs in this tab, so nothing is uploaded.

Comparing two exports rather than two groups? Use the variance report

Three tests, because one is not enough

Most free A/B calculators ask for four numbers, visitors and conversions per variant, and give you back a p-value. That works when your data already is four numbers. It does not work when you have an export with one row per user and a revenue column, which is the situation this page is built for. And a single test carries a single set of assumptions, which your data may not meet.

Welch's t-test compares the two means without assuming the groups have equal variances, which they almost never do and nobody ever checks. The degrees of freedom come from the Welch-Satterthwaite equation and are rarely a whole number. The p-value is computed from a regularized incomplete beta function, not read off a table.
Mann-Whitney U throws away the values and compares the ranks. It assumes nothing about the shape of the distribution, which matters enormously: revenue per user is a long right tail with a large spike at zero, and it is not remotely normal. The variance is tie-corrected, because a column with many repeated values would otherwise report a p-value that is far too confident.
A two-proportion z-test appears automatically when the value column is binary. Running a t-test on a column of zeros and ones is not exactly wrong, but the question was about conversion rates, and the answer should be phrased in percentage points with an interval on the difference in rates.

When Welch and Mann-Whitney disagree about significance, the report says so in a sentence, prints both p-values, points at the two medians and tells you to trust the rank test. That disagreement is not a bug in either test. It is the single most informative thing the page can tell you, and it almost always means one extreme value is pulling a mean around.

Worked example: a checkout test on fifty users

The built-in example is ab-test-checkout.csv: fifty rows, twenty-five in each group, with a revenue column and a converted flag.

user_id,variant,revenue,converted
u001,control,0,0
u002,control,42.50,1
u003,control,0,0
...
u027,treatment,58.00,1
u029,treatment,71.30,1
...

Leave the group column on variant and set the value column to converted. That column holds only 0 and 1, so it is recognised as binary and the proportion test runs. Control converted 7 of 25, which is 28.00 per cent. Treatment converted 10 of 25, which is 40.00 per cent. The difference is 12 percentage points in favour of treatment.

Now the part that matters. The pooled rate across both groups is 17 of 50, or 34 per cent, and the pooled standard error works out at about 9.5 percentage points. The z statistic is therefore about 0.89, and the two-sided p-value is roughly 0.37. The 95 per cent interval on the difference, built from the unpooled standard error, runs from about minus 14 to plus 38 percentage points.

So the conclusion reads: at the 5% level this difference is NOT statistically significant. That means no difference was detected, which is not the same as showing there is none. A twelve-point lift is a large effect commercially, and this test cannot confirm it, because fifty users is nowhere near enough. The interval is the honest summary: treatment could plausibly be fourteen points worse or thirty-eight points better, and you have learned very little. Ship more traffic, not a decision.

Switch the value column to revenue and the framing changes. Now Welch compares the two mean revenues per user, Mann-Whitney compares the ranks, and Cohen's d puts the gap in standard deviations. Because the revenue column is mostly zeros with a scatter of real values on top, it is exactly the shape where the two tests can part company, and the report will tell you if they do.

The p-value is the least useful number on the page

It is the number everybody looks at, and it answers a question almost nobody is asking. A p-value is the probability of seeing a difference at least this large if the two groups genuinely behaved identically. It is not the probability that your result is real, it is not the probability that treatment is better, and it says nothing whatsoever about whether the difference is worth building.

Two other numbers on this page do more work.

The 95% confidence interval is the range of differences the data cannot rule out. An interval from plus 2 to plus 4 per cent and an interval from minus 20 to plus 26 per cent can carry the same p-value while telling you completely different things. Read the interval and ask whether the worst end of it would still be acceptable.
Cohen's d is the difference in pooled standard deviations, with the usual bands: negligible under 0.2, small to 0.5, medium to 0.8, large beyond. On a big enough sample, a d of 0.03 will come back significant. Significant and worthless are not mutually exclusive.

One implementation detail behind the interval, because it is where free calculators go wrong. The multiplier is not 1.96. That is the critical value of the normal distribution, and a t-test on a modest sample lives on a t distribution with fatter tails. At nine degrees of freedom the right multiplier is about 2.26, and an interval built with 1.96 comes out roughly 13 per cent too narrow, overstating what your data can rule out. This page solves for the critical value at your test's actual degrees of freedom, which Welch's formula rarely makes a whole number.

What the page refuses to do

It will not test three groups. Hand it a column with three distinct values and it stops, names the count and lists what it found. Quietly picking two of them, or pooling two together, would produce a plausible number answering a question you did not ask. Filter first.
It will not test a column against itself. Choosing the same column as both group and value is caught and explained.
It will not hide a thin group. Fewer than twenty rows in either group and a warning says so: every number is still computed correctly, but a test on that little data will only detect an enormous effect.
It will not hide a lopsided split. If one group is more than three times the other, a warning suggests checking that the split was deliberate rather than a broken assignment. Welch handles unequal sizes; a broken randomizer is a different problem and no test can fix it.
It counts every row it dropped. Rows with a blank group, or with no usable number in the value column, are excluded and reported. A test quietly run on 70 per cent of your file is not a test you should act on.

Things no calculator can see

The report ends with a paragraph saying this out loud, because the arithmetic is the easy part and the assumptions are where results go wrong.

Assignment. Nothing in your CSV proves users were allocated at random. If treatment quietly went to logged-in users and control to everyone, the test is measuring the login, not the change.
Timing. If the two groups ran over different periods, you have measured the periods. A weekend in one arm and not the other is enough to move most metrics.
Independence. One row per user is the assumption. If a user appears five times, the effective sample is smaller than the row count and every p-value on the page is too small.
Peeking. A p-value assumes you fixed the sample size before you looked. Checking daily and stopping when it crosses 0.05 will get you there eventually on data with no effect in it at all.
Multiple comparisons. Test twenty metrics at the 5 per cent level and one comes back significant on average when nothing is happening. If this is the fourth metric you have tried, say so when you report it.

Frequently Asked Questions

My result is not significant. Does that mean the two groups are the same?

No, and this is the single most misread result in testing. Not significant means no difference was detected at this sample size. A real but modest effect looks exactly like this when the groups are small, and so does an enormous effect measured on thirty rows. The conclusion on this page is worded to say not detected rather than no difference, and the confidence interval tells you what you can actually rule out: if it runs from minus 8 to plus 12 percentage points, you have not ruled out very much at all.

Why does it run three different tests?

Because they answer slightly different questions and can disagree. Welch's t-test compares the two means and assumes those means are roughly normally distributed. Mann-Whitney U compares the ranks and assumes nothing about the shape, which matters because revenue per user is a long right tail with a spike at zero and is not normal at all. When the value column is binary, a two-proportion z-test asks the question you meant to ask, which is about conversion rates rather than means. If the t-test and the rank test disagree, the report says so and tells you to trust the rank test, because a disagreement almost always means an extreme value is dragging a mean around.

What is Cohen's d and why does it matter more than the p-value?

It is the difference between the two means expressed in standard deviations, using the pooled standard deviation of both groups. A p-value only says the difference is probably not chance, and on a large enough sample almost any difference clears that bar. Cohen's d says how big it is: under 0.2 is negligible, up to 0.5 is small, up to 0.8 is medium, and beyond that is large. A statistically significant result with a d of 0.05 is a real difference that is not worth building anything for, and that is a very common outcome.

What happens if my group column has three values?

The page refuses and tells you exactly what it found: how many distinct groups there are and what they are called, up to six of them. An A/B test compares two things. Silently picking two of your three variants, or pooling two of them together, would produce a number that looks fine and answers a question you did not ask. Filter the file down to two groups first, or pick a different column.

Does it detect a conversion column automatically?

Yes. If the value column holds exactly two distinct values and they are a recognisable pair, 0 and 1, true and false, yes and no, or y and n, the page treats it as binary and adds a two-proportion z-test alongside the others. The conclusion is then phrased as conversion rates and percentage points rather than means, which is what you wanted. The test statistic uses the pooled standard error, because the null hypothesis says the rates are equal, and the confidence interval uses the unpooled one, because under the alternative they are not. Mixing those two up is the classic error here.

Why does the confidence interval not use 1.96?

Because 1.96 is the critical value of the normal distribution, and a t-test on a small sample lives on a t distribution with fatter tails. At nine degrees of freedom the correct multiplier is about 2.26, so an interval built with 1.96 would come out roughly 13 per cent too narrow and overstate what the data can rule out. The critical value here is found for the actual degrees of freedom of your test, which Welch's formula rarely makes a whole number.

What can this tool not tell me?

Quite a lot, and the report says so under every run. It cannot know whether users were assigned to the groups at random, whether both groups saw the same conditions over the same period, whether the same person appears in both, or how many other tests you ran before this one. That last one matters: test twenty things at the 5 per cent level and one will come back significant on average even when nothing is happening. It also cannot know whether you stopped the test because the numbers looked good, which invalidates the p-value entirely.

Does the file leave my computer?

No. This page has no upload endpoint. JavaScript in your tab splits the rows into groups and computes every statistic, including the incomplete beta function behind the p-value. Nothing is stored between visits, so reloading the page gives you an empty box again.

Test your two groups properly

Free, no account, no upload. Three tests, an effect size, an interval, and a conclusion that refuses to overclaim.

Back to the calculator