A/B Testing
A controlled experiment that compares two variants of a product or page to determine which performs better.
Definition
A/B testing (also called split testing) is a randomized controlled experiment that shows different variants of a product, page, or feature to different user segments simultaneously, then measures which variant achieves a predefined goal more effectively. Statistical significance — typically 95% confidence — is required before drawing conclusions. Multivariate testing extends this to multiple variables simultaneously.
Why It Matters
A/B testing replaces opinion with data. Companies that systematically run experiments consistently outperform those that rely on intuition. However, A/B tests require sufficient traffic and time to reach statistical significance — running underpowered tests leads to false conclusions.
Example
Testing two landing page headlines: variant A converts 3.2% of visitors; variant B converts 4.1%. With sufficient sample size, variant B is the winner.