SplitStat decides every test with two readings and one set of bars. The first reading says how likely each variation is to be the best one. The second draws a range around the lift that stays honest however often it is read. No winner is called until every bar holds at once.
Everywhere else the screens use plain words, but a method you cannot name is a method you cannot check, so this page calls the statistics what a textbook calls them. Nothing here is something you must learn to run a test.
The maths reads the same numbers the results screen shows: visitors, conversions and totals per variation, counted from the start of the test. Reading a test's results says what each column means. This page says what is done with them.
The two readings
| Reading | The question it answers | Method |
|---|---|---|
| Chance to beat baseline | How often this variation's true rate beats the baseline's, given what both have done. | Bayesian probability |
| Estimated lift range | How much better the variation is, with a range that is honest at every moment of the run. | Confidence sequence |
Chance to beat baseline is a plain Bayesian probability. For a conversion metric the starting belief is flat, a Beta(1, 1) prior, so the data does all the work and nothing assumed in advance can be read back out of the answer. For a money or count metric the belief is centred on the observed average, using Student's t while the test is small. The chance is worked out by drawing from each variation's belief thousands of times and counting how often each comes out on top.
That number is never adjusted. Some tools bend the probability itself so that daily reading stays safe, which keeps the label and changes the meaning. Here the probability means exactly what a textbook says, and the safety lives in the second reading.
Estimated lift range is a confidence sequence. An ordinary 95% interval is honest once: built at one moment chosen in advance, it contains the truth 95 times in 100. Build one every morning and stop the first time it clears zero, and that promise is gone. A confidence sequence promises more: every interval it draws across the whole run is right, all at once, 95 times in 100. You can look daily and stop whenever you like, and the number still means what it says.
The construction is published and peer reviewed, and the maths section at the end of this page prints it, so your own analyst can check every step.
How a winner is called
All of these must hold at the same time, on the goal the test is judged by:
- Chance to beat baseline reaches your required certainty, 95% unless your project says otherwise.
- Picking that winner is a small risk even if it is wrong: its expected loss is at most 0.5% of the baseline.
- The smallest variation has seen at least 500 visitors. The smallest, not the total, so an uneven split cannot pass on the strength of its large side.
- The smallest variation has at least 100 conversions, asked separately, because a page that converts most visitors reaches the conversion floor long before it has seen enough people.
- The test has run at least 7 days, so one odd weekday cannot decide it.
- The estimated lift range is clear of zero.
The last bar is what makes the others safe to read daily, and we measured why. The study ships in the product's own test suite: 40,000 simulated tests where both pages were the same, read every day for a month and stopped at the first yes. An ordinary interval read that way named a winner in up to 19% of them. The range we ship, through the same bars on the same traffic: at most 1%. Thirty looks at one bar are thirty chances to cross it by luck, and stopping at the first crossing keeps the luckiest one. The chance to beat baseline is not a defence against that, whatever this industry's marketing says about Bayesian methods and peeking, which is why bar 6 exists.
The bars freeze when a test starts. Certainty can be set per metric and per test, and the floors per project, but never on a test that is already running, so nobody can move a bar a result was read against.
What each label means
| Label | What is behind it |
|---|---|
| Chance to beat baseline | The unadjusted Bayesian probability above. |
| Estimated lift range | The confidence sequence, shown as a share of the baseline. |
| Result | Only on a project set to the sequential engine before that choice was removed, in place of the chance: Winner when the whole range is above zero, Worse when below, Not yet otherwise. |
| Day x of y and the chips beside it | Progress against bars 3 to 5. |
| [Variation] is ahead | Bars 1 to 5 hold and the range still includes zero. Not a result. |
| Not enough data yet to measure changes smaller than about X% | Half the width of the estimated lift range, as a share of the baseline. A change smaller than this is still inside the noise. Shown once bars 3 to 5 hold. |
| At most X% chance of a false winner | Under Chance to beat baseline needed in the settings: the error budget of bar 6, which is 100% minus that setting, shared across the variations. An upper bound; measured runs land far below it. |
| Chance to catch a real improvement (power) | Used by the planner only: how often a test should catch a real improvement of the size it was planned for. 80% unless your project says otherwise, and a test can set its own. It never changes a result. |
| Smallest change this test is designed to detect (MDE) | Optional, set per test. With it, the planner says how long the test needs. Without it, the planner says the smallest change a 28-day test can catch. |
| A X-week test can catch improvements of about X% or more | The planner. It plays the test forward 1,000 times, a day at a time, with your bar, minimums and split, and reads traffic from the last 28 days of new visitors to the test's pages. An estimate, never a promise. |
| Visitors a day (optional), Conversion rate (optional) | On How this test is judged, for trying your own numbers in the planner. Never saved. |
| Share the error budget across variations (Bonferroni) | On by default. Splits the budget of bar 6 between the variations compared with the original. Only changes anything with three or more variations. Frozen when a test starts. |
Good to know
- The range's price is width. At its narrowest it is about a third wider than a one-off interval, and early in a test it is wider still. A test read this way needs more data than one read once at a planned ending. In exchange you may stop the moment it clears, and no reading you made along the way was wrong.
- With more than one variation, each range is drawn at a higher confidence so the test as a whole keeps your setting, unless your project turns off Share the error budget across variations. Off, each variation is read at the full budget: tests with three or more variations finish sooner and name more false winners. Reading a test's results lists the figures.
- There is no p-value anywhere in SplitStat. A p-value is not the chance that a variation is better, and any label short enough for a screen implies that it is. The two readings above answer the questions people actually ask.
- Guardrails are read on the same always-honest range, every hour, which is what lets a breach pause a test by itself without crying wolf.
- Every project uses the Bayesian engine, and there is no setting to change it. The range behind bar 6 is not a choice either: every test passes through it, whatever numbers the screen shows, and a running test keeps the bars it started with.
- An A/A test runs the same maths with the reading inverted: a difference is a fault and a draw is the pass.
- The planner counts the pages a test targets, not its audience: an audience rule narrows who enters, so the real traffic can be lower than the plan. A page with no visits in the last 28 days has nothing to plan on until you type your own numbers. Goals that add up an amount are not planned yet.
- A result can be sharpened with what visitors did before the test. Metrics explains the setting.
- Stopping the moment the range clears is safe. Quoting the lift from that moment is another matter: a test stopped on a lucky day keeps the lucky day in its average, so the earlier the stop, the more the printed lift flatters. The range is the honest part of an early readout.
- Watching daily earns its keep in two places: a guardrail catching harm early, and stopping a clear winner so your traffic moves on. Neither needs the lift measured finely, which is what the full run is for.
The maths, for your analyst
Nothing below is needed to run a test. It is here so the statistics can be checked rather than taken on trust.
The beliefs. For a conversion metric, each variation's rate starts flat and after n visitors and c conversions is
\theta \sim \mathrm{Beta}(1 + c,\ 1 + n - c)
For a money or count metric, the mean is believed to sit at the observed average m with standard error s/\sqrt{n}, drawn through Student's t:
\mu \sim m + \frac{s}{\sqrt{n}}, T_{n-1}
switching to a normal above 1,000 degrees of freedom, where the two agree to less than the screen prints. s is the sample standard deviation and the degrees of freedom are n - 1.
The chance and the loss. Both are read off 100,000 paired draws from those beliefs, refined to 1,000,000 when a test sits near its bar. The chance to beat baseline is the share of draws in which the variation's draw beats the baseline's:
P(\text{beat}) = \frac{1}{N} \sum_ \mathbf{1}\left[\theta_v^{(i)} > \theta_b^{(i)}\right]
The expected loss is the average of how far an arm fell short of whichever arm was best in that same draw, taken as a share of the baseline's mean:
\text{loss}v = \frac{1}{N} \sum \max\left(0,\ \max_j \theta_j^{(i)} - \theta_v^{(i)}\right) \Big/\ \bar\theta_b
With two arms the best rival is the baseline and this is the textbook expected loss.
The range. The estimated lift range is an asymptotic confidence sequence. With n visitors across both arms, a difference whose standard error is s, an error rate \alpha of one minus your certainty, and the sequence tuned to be narrowest near n^*, fixed at 5,000 visitors:
\rho^2 = \frac{-2\ln\alpha + \ln(-2\ln\alpha + 1)}{n^*}
\text{half width} = s,\sqrt{n}\ \sqrt{\frac{2,(n\rho^2 + 1)}{n^2\rho^2}\ \ln\frac{\sqrt{n\rho^2 + 1}}{\alpha}}
Read past the algebra and it is the ordinary interval with one change. An ordinary interval is s \times 1.96. This is s times a multiplier that grows slowly with n, roughly \sqrt{2 \ln(\sqrt{n}/\alpha)}, and that slow growth is what pays for the extra looks.
With more than one variation, \alpha is divided by the number of challengers before the range is drawn, which is the Bonferroni correction the results page describes in its own words.
The construction is from Waudby-Smith, Arbour, Sinha, Kennedy and Ramdas, "Time-uniform central limit theory and asymptotic confidence sequences", Annals of Statistics 52(6), 2024, on arXiv as 2103.06476. GrowthBook's documentation prints the same formula. Our own test suite re-measures the false-winner rate of this exact construction, and the figures quoted on this page are read from its results file rather than typed here.
Related articles
- Reading a test's results: every column and verdict line this page is the maths behind.
- Metrics: the goal a test is judged on, the cap on extreme totals, and sharpening a result with earlier activity.
- A/A test: checking your setup with the same engine.