A/B testing sample calculator
A handy little calculator I threw together to — given your current experiment data — calculate your conversion rate, a Bayesian decision view and an estimate of how much more traffic you need to reach a confident decision.
Experiment data
Adjust your experiment counts and settings.
Maximum acceptable drop in conversion rate.
Confidence required before calling the result.
Current results
Neither rule has crossed your threshold yet.
If today’s rates hold, how much more traffic do you need?
Estimate the experiment size at which each probability threshold will be crossed, assuming the conversion rate stays the same.
| Probability threshold | Variant is better | Variant is non-inferior |
|---|---|---|
| 90% | — | — |
| 95% (selected) | — | — |
| 99% | — | — |
Plan a new fixed-sample experiment
Traditional frequentist sample-size planning using the Evan Miller method. This is separate from the Bayesian decision probabilities above.
What do these numbers mean?
Probability variant is better is the Bayesian posterior probability that the variant’s true conversion rate is higher than the control’s.
Probability variant is non-inferior asks how likely is it that the variant is not worse than control by more than your chosen absolute margin?
Decision threshold is the posterior probability you require before calling the result.
Projection is an expected-path forecast, not a guarantee. It assumes future traffic converts at the currently observed rates and continues to split evenly between control and variant.
Fixed-sample planning uses the Evan Miller method to estimate the number of subjects required per variation for a pre-planned frequentist test.