When multivariate beats A/B
A/B testing changes one variable at a time. Multivariate testing changes several. The advantage: you discover interactions — maybe headline A wins with image 1, but headline B wins with image 2. If you only A/B tested headlines, you'd miss this entirely. The disadvantage: traffic requirements grow geometrically, and most sites can't supply enough.
The traffic math
To detect a 5% lift in 12 variants at 95% confidence, you need ~120,000 visitors total. Most pages don't have that. If your monthly landing page traffic is below 50,000, multivariate testing will conclude before you have data, producing noise. Stick to sequential A/B tests instead.
The right shape of multivariate test
If you do have the traffic, test combinations that share a common theme (e.g., 3 hero variations × 2 CTA variations focused on objection handling). Don't test 12 unrelated changes at once — even with statistical significance, you can't separate the effects clearly. Run analysis first to identify the highest-leverage elements before testing.