How Long Should You Run an A/B Test?
You should run an A/B test for at least two full weeks, and keep it running until it reaches the sample size you calculated before it started. For most stores that means two to four weeks, and never past eight. Calling a winner early because the graph looks good is the most common way to get a wrong answer.
Key Points
- Two full weeks is the floor, since weekday and weekend shoppers convert differently and one week catches that cycle only once.
- Set the sample size before you start, then divide it by your daily traffic to get the real length. That number is the test length, not the two-week floor.
- Peeking and stopping early can push your false positive rate from 5% to 26%, per Evan Miller’s math.
- Six to eight weeks is the ceiling, because deleted cookies and seasonality blur the groups after that.
- A broken or clearly harmful variation is the one thing you kill on day one.
Why Two Full Weeks Is the Floor
Two full weeks is the floor for an A/B test, since your store runs on a weekly cycle. Weekday browsers, weekend buyers, and payday spikes all convert at different rates, and one full week captures that cycle once while two weeks capture it twice.
A test that runs Tuesday to Friday only samples one kind of shopper. The person comparing options on a Wednesday lunch break behaves differently from the one buying on a Sunday night, and those customer behavior signals are exactly what the test is trying to average out.
So always end a test on the same weekday you started it. Whole weeks, never parts of them, and if your sample size math says four weeks, four weeks it is. The floor only exists so a lucky weekend never gets to call itself a trend.
Also, pick your start date with the calendar open. A test that needs four clean weeks should not start two weeks before your biggest sale of the quarter, since promo traffic behaves like a different species and neither half of the data will match the other.
How to Calculate Your Test Length
You calculate your test length by dividing the sample size you need by the visitors the tested page gets each day. The sample size comes from three inputs: your current conversion rate, the smallest lift you care about detecting, and the confidence you want, which almost everyone leaves at 95%.
Step 1: Pick the smallest lift worth detecting
Pick the smallest lift that would change what you do, and be honest about it. A 5% lift on a checkout button might be worth chasing at a big store, but on a store doing 300 orders a month it will never reach significance, so the only lifts worth testing are 15% to 20% and up.
Step 2: Run the sample size calculator
Plug the baseline and the lift into Evan Miller’s sample size calculator or Optimizely’s version. The output is the number of visitors each variation needs, at 95% confidence and 80% power. Here is roughly what it returns for a two-way test:
| Baseline conversion rate | 10% lift | 20% lift | 30% lift |
|---|---|---|---|
| 2% | 80,700 per variation | 21,100 per variation | 9,800 per variation |
| 3% | 53,200 per variation | 13,900 per variation | 6,450 per variation |
| 5% | 31,200 per variation | 8,150 per variation | 3,780 per variation |
Read the table left to right and the lesson jumps out: chasing a 20% lift instead of a 10% lift cuts the sample to roughly a quarter. Bold changes are the cheap ones to test.
Step 3: Divide by daily traffic and round up to whole weeks
Divide the per-variation number by half your daily visitors to that page, since a two-way test splits them. A store converting at 3% that wants a 10% lift needs about 53,000 visitors per side, so at 2,000 visitors a day the test runs 53 days, which rounds up to eight full weeks.
If the answer comes back in months, that is its own answer. You do not have a testing problem, you have a traffic problem, and A/B testing with low traffic calls for a different playbook.
Test length is math you do before the test, not a feeling you get during it.
Step 4: Check the answer against your store size
Check the answer against your store size before you build anything, since the same test lands very differently at 500 visitors a day and at 5,000. Using the 3% baseline and a 20% lift from the table, a two-way test needs about 27,800 visitors in total, and the calendar looks like this:
| Visitors per day to the page | Days to reach the sample | What you actually run |
|---|---|---|
| 500 | 56 days | Past the eight-week ceiling, so test a bigger change or a busier page |
| 1,000 | 28 days | Four full weeks |
| 2,000 | 14 days | Two full weeks, the floor |
| 5,000 | 6 days | Still two full weeks, because the floor wins |
The bottom two rows are the ones people get wrong in opposite directions. A big store hits its sample in six days and has to keep running anyway, and a small store hits the ceiling before the sample and has to change the question instead of the length.
What Statistical Significance Actually Means
Statistical significance at 95% means there is a 5% chance the difference you saw came from random noise with no real change behind it. It is not a 95% chance your variation is better, and it says nothing about how big the win is, only that a win of some size is probably there.
The second number, power, is the one most store owners never set. Power at 80%, the default in both calculators above, means a real lift of the size you chose gets caught four times out of five and missed once. Raise it to 90% and the sample grows by about a third, which is why most practitioners leave it at 80% and accept the occasional missed win.
So read a finished test as two questions. Did it clear 95%, and was the sample big enough for the lift you set out to find? A test that clears 95% at half the planned sample is the early-stop trap from the next section wearing a suit.
Why Stopping a Test Early Backfires
Stopping a test early turns your significance number into fiction. Evan Miller showed in How Not To Run an A/B Test that checking daily and stopping the moment one side looks significant can push a 5% false positive rate to 26%, which is a one in four chance of shipping a change that does nothing.
The reason sits in the assumption. Significance testing assumes you picked the sample size ahead of time and looked once at the end. Peek every morning and stop on the first good day, and that assumption dies quietly while the dashboard keeps showing you a green 96%.
Just know that looking is fine and acting is the problem. Check for bugs, broken tracking, and a variation that tanked on one browser as often as you like, and save the winner-picking for the finish line you drew on day one. The same discipline applies to the before-and-after comparisons a small store runs without a testing tool: pick the dates first, then read the numbers.
Can You Watch a Test Without Wrecking It?
Yes, you can watch a test without wrecking it if the tool runs sequential statistics. Spotify’s experimentation team found that even casual repeated checking can roughly double false positives, and their write-up on choosing a sequential testing framework explains how sequential methods make monitoring safe by paying for every look up front.
Most paid testing tools now run some version of that math for you, which is the one honest reason to pay for one. If yours does, you can act on a result the tool calls final even before your calendar date, and if yours does not, the calendar date is the only stopping rule you have.
Whichever tool you run, a safe mid-test check covers four things and nothing else:
- Both variations load on every device and browser your shoppers use.
- Order and revenue events fire for both sides, not just the control.
- Neither side is bleeding revenue from day one.
- The traffic split sits near 50/50 and is not drifting.
Anything past that list is peeking, and the conversion rate column stays closed until the planned end date.
Either way, the tests worth watching are the big ones. The product page tests that move price, add to cart, and shipping above the fold are loud enough to show up early, and a sticky add-to-cart bar published a 7.9% lift at 98% significance in a proper run. Button colors never get there.
When to Run a Test Longer
Run a test longer when something temporary might be inflating the result, like a novelty effect or a holiday sitting in the middle of your data. Returning visitors poke at anything new, which flatters the variation for a week or two before the effect fades, and a test that ends inside that window hands you a lift that is already gone.
On top of the novelty effect, Ron Kohavi’s rules of thumb for experimenters add a sobering one: most real wins are small, so a huge early lift is more likely a fluke than a breakthrough. His later paper with Deng and Vermeer, A/B Testing Intuition Busters, reports that only about a third of experiment ideas at Microsoft improved their target metric, and the rate was lower at Bing, Airbnb, and Netflix.
So when your offer test shows a 40% lift after ten days, add a week. A test between discounts and free gifts is exactly the kind that spikes early, since the first shoppers to see a new gift are your most loyal ones, and the lift with cold traffic is what you are actually paying for.
Beyond that, run longer whenever one segment is carrying the result. Split the test by device before you call it, and if the entire lift comes from desktop while mobile converts the same or worse, the variation needs another week and a closer look on a phone.
When to Pull the Plug
Pull the plug at six to eight weeks, or immediately if the variation is clearly broken and costing sales. Past eight weeks, deleted cookies mix your groups together, returning shoppers land in both variations, and the season quietly changes under the test until neither half of the data matches the other.
Seasonality is the one that gets me, and I have killed tests I was excited about because Black Friday walked into the middle of them. The right move is to log what you learned, note the date you stopped and why, and relaunch the same question as a fresh test in a quieter month. That log is the most valuable page in a CRO audit, since it stops you from re-running a question you already answered.
The one exception to patience is damage control. A variation that breaks checkout on one browser or tanks revenue from day one does not deserve its full run. Kill it, fix it, and relaunch it as a fresh test rather than patching it mid-flight, because a patched test has two different variations in its data and no way to tell them apart. That stop rule, and the rest of the test plan, is something my CRO services for Shopify stores write down before a single test goes live.
Test Length Questions I Get Most
Can I run an A/B test for one week?
No, one week is too short for an A/B test on a store, even when the sample size math says you have enough visitors. One week captures the weekday and weekend cycle once, and a single unusual day, like a payday or a competitor’s sale, can tilt the whole result. Two weeks is the floor for that reason alone.
Should I run tests during Black Friday?
No, you should not run a conversion test across Black Friday, since promo traffic converts on price and urgency, not on the change you are testing. Pause tests the week before your sale and restart them two weeks after, when your traffic mix looks like a normal month again.
What if my test never reaches significance?
A test that never reaches significance after its full planned run is telling you the change did not move the number by the amount you set out to detect. Log it as a real result, since “no meaningful difference” is useful, and move the effort to a bigger change or a page with a bigger leak.
Do I have to send 100% of my traffic into the test?
No, you do not have to send 100% of traffic into a test, but every visitor you hold back stretches the length. A test on 50% of traffic takes twice as long to reach the same sample, so hold traffic back only when the variation carries real risk, like a new checkout flow, and send everything else in.
Want Tests That Actually Settle Arguments?
Test length is not a judgment call. Two full weeks minimum, the sample size you calculated up front, a hard stop before cookie churn and seasonality blur the answer, and a kill switch only for broken variations. Everything else is peeking with extra steps.
Book a call and I’ll look at your traffic and conversion numbers, then tell you whether testing is worth running yet and exactly how long your first test should take. If you want the whole program run for you, that is what my CRO services cover, and you can book through the contact page.
