An A/A test shows both groups the same page. Nothing is changed, so neither group should come out ahead. It is a check on your setup, not a test of an idea.
The reading is upside down from every other test. A draw is the pass. A difference is a fault. If two halves of your traffic seeing the identical page produce different numbers, something between the visitor and the report is treating them differently, and a real test on that page would report the same difference as a win.
Run one when you are new to the tool and want to trust it, after you change how your site records events, or when a past result looked too good and you want to know whether the plumbing was to blame.
The cost is real: an A/A test spends traffic and answers nothing about your product. Run one, read it, move on.
The six steps
The same six steps as any test, with two of them cut down.
| Step | Name | What you do there |
|---|---|---|
| 1 | Setup & page | Name it and pick the page, as usual. |
| 2 | Traffic | Only one number. There is nothing to split. |
| 3 | Metric watched | The metric the check is read on. |
| 4 | Audience & targeting | Same as an A/B test. |
| 5 | Configuration | Same as an A/B test. |
| 6 | Preview & confirm | Same as an A/B test. |
For steps four to six, see Your first A/B test.
How to set one up
- Setup & page. Type a name and pick the page, the same way as an A/B test. Add page rules if the check should cover more than one page.
- Traffic. Set Share of matching traffic, percent. That is the only thing this step asks.
- Read The two groups underneath it. Both are the page as it is, fixed at 50 / 50. Nothing there can be edited.
- Metric watched. Pick the metric the check is read on. Choose one you already trust and that gets plenty of traffic.
- Work through steps four to six as you would for an A/B test, then start it.
There is no visual editor on an A/A test, and there is no button for one. A change would make it a different kind of test.
What each label means
Most of this test is fixed. These are the parts you set.
| Step | Label | What it means |
|---|---|---|
| 2 | Share of matching traffic, percent | How many matching visitors enter the check. The rest are left out of it. |
| 2 | Fixed at 50 / 50 | Not a setting. A check whose halves you can re-weight is not a check. |
| 3 | Metric watched | The metric the check is read on. Pick a busy one. |
The two groups always take half the traffic each, and the split is part of what is being checked. That is why you cannot change it.
How to read the result
The result screen reads the verdict as a check rather than as a winner. Four things it can say:
| What it says | What it means |
|---|---|
| Still collecting | Not enough counted yet. Wait. |
| Setup looks sound | The two groups came out level. This is the pass. The line beneath says how big a difference the check could have caught. |
| A difference where there should be none | One group is ahead by more than chance accounts for, and both saw the same page. Something is treating the groups differently. |
| The traffic did not split evenly | Both groups were set to take half and did not get it. The check failed at the first step. |
"The traffic did not split evenly" is the worse of the two failures. Until the split is even, no number this tool reports on that site can be read at all. Start there before you look at anything else.
A difference is not always a bug in the tool. Common causes are a cache serving one group a different page, a bot filling one half, a redirect that fires for some visitors, or an event that only reports on part of your traffic. Check the Events screen first.
Good to know
- The pass names a size. "A difference bigger than 9% would have shown up" means the check would have caught a gap of that size between the groups and did not find one. It says nothing about smaller gaps. A short run on a quiet site passes at a large size, and that is still a pass, only a coarse one.
- No setting is needed for the pass. If the project has a value in "Difference too small to matter", the check can also pass by proving the two groups are within that value, which is a stronger claim and reads differently.
- A "Setup looks sound" result is not a promise that every future test is clean. It says this page, this metric and this traffic behaved. Change the recording and the answer can change.
- The page is not held back while the test decides anything, because there is nothing to apply. Both groups get the page as it arrives.
- The start checklist is shorter than an A/B test's. There are no changes to resolve, so the checks about changes do not appear.
- Run it long enough. An A/A test that stops early reports "Still collecting" and tells you nothing. Give it the same traffic you would give a real test.
- Expect the occasional false alarm. A difference will turn up now and then by chance alone, which is what a confidence level means. A repeat that comes out level is the cheapest way to tell chance from a fault.
Related articles
- Your first A/B test: steps four to six, and the full checklist.
- Metrics: choosing the metric the check is read on.
- Events: where to look when a difference turns up.