A Frequentist test is planned before it starts and read once, when it reaches the number of visitors it was planned for. The result is a p-value and a confidence interval for each variation against the baseline.
It is one of the two engines a project can use. The other, Bayesian, reports a chance to beat the baseline and can name a winner on any day the bars hold. How the stats work explains that one. A Frequentist test is the right choice when your team already plans tests by sample size, or when a reviewer asks for a p-value.
The engine is set in Project settings, and every new test takes it. A test can choose the other engine on How this test is judged, under Statistics engine, before it starts. It locks the engine, with the rest of its settings, on the day it starts.
How to set one up
- Open Project settings and find Results settings.
- Under Statistics engine, choose Frequentist (fixed sample size).
- Pick a Confidence level: 90%, 95%, 99%.
- Set Chance to catch a real improvement (power) and Smallest change this test is designed to detect (MDE). These are the defaults every new test is planned on.
- Choose a Correction for several variations, then press Save.
- On a test, open How this test is judged. Under Planned sample size, check the sample the screen works out. Type a Baseline conversion rate if these pages have no history, or if the history is wrong.
- Start the test. The planned sample is fixed from that moment.
What each label means
| Label | What it means |
|---|---|
| Statistics engine | Bayesian or Frequentist (fixed sample size). Every new test in the project takes it. |
| Confidence level | 90%, 95%, 99%. At 95%, a test with no real difference names a false winner at most 5 times in 100, if it runs to its planned sample. |
| Chance to catch a real improvement (power) | How often the test should find a real change of the planned size. 80% unless your project says otherwise. It sizes the test and never changes a result. |
| Smallest change this test is designed to detect (MDE) | The change, as a share of the baseline, the test is sized for. 10% unless your project or the test says otherwise. Smaller needs more visitors. |
| Correction for several variations | None, Bonferroni or Šidák. With 3 or more variations, Bonferroni and Šidák split the error budget so the whole test keeps your confidence level. Šidák is a little less strict. Bonferroni is the default. |
| Baseline conversion rate | The rate the sample is worked out from. Empty reads the last 28 days of the test's pages. |
| Visitors per variation | Optional, and needed on a goal that adds up an amount, which no formula can size. A number typed here replaces the worked-out sample. |
| Planned sample | Visitors each variation needs before the result is read. Worked out from the four settings above and fixed when the test starts. |
| p-value | How often a difference at least this big would turn up by chance if the variations were the same. It is not the chance a variation is better. Shown once the planned sample is reached. |
| Significant at X% | Yes when the p-value is below the bar set by your confidence level and the correction. |
| Confidence interval | The range for the lift at the stated confidence. Read at the planned sample, it holds the true lift that often. |
| Not final | Beside the lift until the planned sample is reached. The number moves from day to day and is not to be decided on. |
| No significant difference | The test reached its planned sample and no variation differs from the baseline at your confidence level. That is a result. |
Good to know
- The p-value and the winner appear only at the planned sample. Read before then, a p-value crosses the bar by luck far more often than the confidence level says, so the screen does not show it.
- Every test is two-tailed: it finds a change in either direction.
- The minimum visitors, conversions and days from Results settings still apply. A test is read at its planned sample or after the minimums, whichever comes later.
- A test that needs more than eight weeks at its recent traffic shows a warning on the start checklist. It can still start.
- The engine, the confidence level and the planned sample cannot be changed once a test starts. Stop it and run a copy to change them.
- Guardrails are read every hour on this engine too, held to the one-sided 99% end of the interval.
Related articles
- How the stats work: the Bayesian engine and the minimums both engines share.
- Reading a test's results: every column on the results screen.
- Metrics: the goal a test is judged on.