Mistake 1: Ending tests too early
The most common testing error is ending a test when you see the result you hoped for. A test that shows variant B is 25 percent better after 3 days and 200 visits has not produced a reliable result. It has produced noise that happens to point in the direction you wanted.
The minimum duration for a meaningful conversion rate test is 2 weeks, regardless of traffic volume. This controls for day-of-week effects: Monday shoppers behave differently from Friday shoppers. The minimum conversion volume is 100 to 200 conversions per variant, which for most DTC brands with 2 to 4 percent conversion rates requires 2,500 to 10,000 visits per variant. If your traffic volume is below that threshold, you cannot run reliable A/B tests. Focus on other optimization methods first.
Mistake 2: Testing too many things at once
A multivariate test that changes the headline, the hero image, the CTA button, and the price simultaneously cannot attribute a performance difference to any single change. If variant B outperforms variant A, you know something worked. You do not know what, which means you cannot learn from it and apply it elsewhere.
Test one variable at a time. The question should be as specific as possible: does a headline that leads with the benefit outperform a headline that leads with the product name? Does a model-worn hero image outperform a flat-lay hero image? One question, one test, one answer.
Mistake 3: Not calculating sample size before starting
Sample size calculation is a ten-minute exercise that most brands skip. Before starting a test, specify: your current conversion rate, the minimum improvement you want to detect (usually 5 to 15 percent for practical significance), and your desired statistical confidence level (typically 90 to 95 percent). Input these into any sample size calculator to determine how many visits per variant you need.
Running a test without pre-calculating sample size means you do not know how long to run it or when to trust the result. This is the cause of most early test endings and most incorrect winner declarations.
Mistake 4: Ignoring secondary metrics
A test variant that improves add-to-cart rate but decreases checkout completion rate might show a neutral or negative impact on overall conversion rate. If you only track the primary metric (conversion rate or revenue) you might declare a winner that actually damaged a downstream metric.
Track secondary metrics alongside your primary metric: add-to-cart rate, checkout initiation rate, checkout completion rate, AOV, and return rate within 30 days. A variant that wins on conversion rate but generates 15 percent more returns is not necessarily a winner when you account for the full economics.
Mistake 5: Running tests during atypical periods
A test that runs over a major promotional period, a seasonal spike, or a viral moment cannot be generalized to normal conditions. If 40 percent of your test traffic arrived because you ran a promotion that week, the behavior of that traffic is not representative of your standard traffic.
Exclude promotional periods and anomalous weeks from your test calendar. If an anomaly occurs during a test that is already running, flag those days in your results and extend the test period to collect equivalent normal-period data.
---