Blog · Testing AEO and GEO changes
A/B Testing for AEO: How to Test AI Visibility Changes
Priya Bothra · September 11, 2026
You cannot run a classic A/B test on AI answers, because you cannot show half of ChatGPT's users one version of your page and half another. What you can do is run controlled experiments: change a group of similar pages or prompts, leave a comparable group unchanged, measure both repeatedly before and after, and compare the difference. Done with enough samples, this tells you whether a change likely caused an improvement or whether you are seeing noise.
Noise is the central problem. AI answers vary substantially between runs. SparkToro and Gumshoe found less than a 1 in 100 chance that ChatGPT or Google's AI would return the same brand list twice. Without a test design, a random swing looks like a win. This guide explains practical experiment designs, the sample sizes they need and how to read the results.
Why classic A/B testing does not fit
In a website A/B test, you control which version each visitor sees and measure their behavior. In AI search:
- Crawlers see one version of a page, not a random split.
- You cannot assign users of an AI assistant to groups.
- Answers change for reasons unrelated to your change: model updates, competitor content and retrieval differences.
- Effects take time to appear, because pages must be recrawled.
So AEO testing borrows from SEO split testing and from quasi-experimental methods used in economics and research.
Three experiment designs that work
Design 1: Page-group split test
Take a set of similar pages, such as 40 product category pages or 30 integration pages. Randomly assign half to receive the change and keep half as a control. Measure AI visibility and citations for prompts related to each page before and after.
Best for: template-level changes such as adding answer-first summaries, comparison tables or FAQ sections.
Requirement: enough similar pages to form two comparable groups.
Design 2: Prompt-cluster comparison
Choose two comparable prompt clusters. Apply a change, such as a new guide plus third-party outreach, to one cluster only. Measure both clusters repeatedly.
Best for: content and authority work that cannot be split at the page level.
Limitation: clusters are never perfectly comparable, so treat results as directional.
Design 3: Pre and post with a control
When you can only change one thing, such as your pricing page, measure related prompts for several weeks before and after. Also measure a control set of unrelated prompts to capture general swings caused by model updates.
Best for: one-off changes.
Limitation: the weakest design, since other events can coincide with your change.
The analysis method: difference-in-differences
For all three designs, compare the change in the test group with the change in the control group:
Effect = (Test after − Test before) − (Control after − Control before)
If visibility in your test group rises from 20% to 32% and the control group rises from 21% to 25% over the same period, the estimated effect is 12 − 4 = 8 percentage points. The control group absorbs general shifts, such as a model update that raised visibility across the board.
Ahrefs used this approach in its study of schema markup and AI citations, comparing 1,885 pages that added schema with 4,000 control pages.
How many samples you need
Because AI answers vary, you need many answers per group to detect a real difference. Standard statistics for comparing two proportions give a useful rule of thumb.
| Baseline visibility | Change you want to detect | Approximate answers needed per group, per period |
|---|---|---|
| 30% | +10 points, to 40% | About 350 |
| 30% | +5 points, to 35% | About 1,400 |
| 10% | +10 points, to 20% | About 200 |
Approximations for 95% confidence and 80% statistical power using standard two-proportion sample size formulas. Answers from the same prompt are not fully independent, so real requirements can be higher.
Practical implications:
- Small tests can only detect large effects.
- Pool answers across many prompts and runs within each group.
- Test changes you expect to matter a lot before testing subtle ones.
Step-by-step test process
- Write a hypothesis. "Adding a summary verdict and dated comparison table to integration pages will increase citation rate for integration prompts by at least 10 points within six weeks."
- Choose the design and groups. Randomize assignment where possible.
- Define metrics. Primary: one metric, such as citation rate. Secondary: visibility, recommendation share and accuracy.
- Measure the baseline. At least two to four weeks of repeated sampling for both groups.
- Ship the change to the test group only. Record the date and exactly what changed.
- Allow time for recrawling. Check logs to confirm AI search crawlers have fetched the changed pages.
- Measure the post period. Use the same prompts, models and sampling method.
- Analyze with difference-in-differences. Report the effect with a confidence interval, not just a point estimate.
- Document and decide. Roll out, iterate or abandon.
What to test first
Good early candidates are changes that are easy to apply to many pages and plausibly important:
- Answer-first summaries at the top of pages.
- Specific comparison tables instead of checkmark tables.
- Adding sourced statistics and citations, which the GEO research paper found effective in its benchmark.
- Clear "who this is for" statements on product pages.
- Server-side rendering of content previously loaded by JavaScript.
Common mistakes
No control group. Model updates can move all results at once.
Changing the prompt set mid-test. Keep prompts, models and sampling identical across periods.
Stopping early. Recrawling and model variation need time.
Testing many changes at once. You will not know which one worked.
Declaring victory on noise. A five-point swing with small samples is often random.
A hypothetical example
A hypothetical integration platform has 60 integration pages. It randomly assigns 30 to receive a new template with a two-sentence summary of what the integration does, a table of supported objects and a setup time estimate. After four baseline weeks and six post-change weeks, citation rate for integration prompts rises from 18% to 29% in the test group and from 17% to 20% in the control group, an estimated effect of about 8 points. With roughly 600 answers per group per period, the team judges the result credible and rolls the template out to all pages.
How Bob Builds AI helps
Bob Builds AI's Visibility Monitoring provides the repeated sampling that experiments depend on, and Analytics & Attribution attributes performance changes to specific optimizations such as content updates and technical fixes.
FAQ
Can you A/B test for AI search?
Not in the classic sense, because you cannot split an AI assistant's users between page versions. You can run controlled experiments by changing a group of pages or prompts, keeping a comparable control group, and comparing changes in both over time.
What is a page-group split test?
A page-group split test randomly assigns similar pages, such as product or integration pages, into test and control groups. The test group receives a change, and results for both groups are compared to estimate the change's effect while controlling for outside factors.
How long should an AEO test run?
Typically two to four weeks of baseline measurement and four to eight weeks after the change, depending on how quickly AI search crawlers recrawl your pages and how many samples you can collect. Check crawler logs to confirm changed pages were fetched.
How many prompts do I need for a valid AEO test?
It depends on the effect size. Detecting a 10-point increase from a 30% baseline needs roughly 350 answers per group per period by standard formulas, more when answers are correlated. Pool answers across many prompts and repeated runs.
What is difference-in-differences?
Difference-in-differences estimates an effect by comparing the change in a test group with the change in a control group over the same period. It removes shifts that affect both groups, such as AI model updates.
What should I test first for AEO?
Start with template-level changes that are easy to apply widely and likely to matter, such as answer-first summaries, specific comparison tables, sourced statistics, clear audience statements and server-side rendering of key content.
Why do my AI visibility results change without any changes on my site?
AI answers vary between runs and change with model updates, retrieval differences and competitor activity. That is why tests need control groups and repeated sampling to separate real effects from noise.
Conclusion
AEO testing is possible, but it requires experimental discipline: comparable test and control groups, repeated sampling, enough answers to detect the effect you care about and difference-in-differences analysis. Without that structure, AI answer variability makes almost any result look meaningful.
Pick one template-level change you believe in, find 30 or more similar pages and run a page-group test for ten weeks. The result will teach you more than a dozen uncontrolled changes. Bob Builds AI can provide the repeated measurement those tests need.