Can I run an A/B test with low traffic?

Work out the sample size first, because low baselines are expensive. At 80% power and a 5% false positive rate, detecting a 10% lift on a 1% baseline takes about 163,000 users per group. On a 50% baseline it takes about 1,600. If the answer is months of traffic, test a bigger change or a metric with a higher baseline. For returning users, a method called CUPED, which adjusts for how each user behaved before the test, can cut the sample needed by 30 to 50%.

Can an AI analyst analyze my A/B test?

It can do the statistics. In our ChatGPT tests we asked whether a checkout test that was up 8% at p = 0.03 after three days should launch, and it said "No, not based on this readout," naming peeking, novelty and the missing weekend. Asked to design a $1 delivery fee test, it worked out that the expected effect needed 388,664 customers when 4,428 were available. It will not know which metric your team agreed on unless you tell it.

What if my metric is a ratio, like clicks per impression?

If you split by user but the metric counts impressions, one user's impressions are not independent of each other, and a plain two-proportion test will make the result look more certain than it is. The top answer in an r/datascience thread asking this question pointed to the delta method. Computing the metric per user first and comparing users also works.

How to run an A/B test

A product manager who had just moved over from business analysis asked r/ProductManagement in April how to run A/B tests properly. Their team had no testing tool. "For a/b testing what we did is 1 month using variant a, 1 month using variant b, but the analysis is not really apple to apple."

The top reply named the problem. That is two months compared with each other, and the second month had its own campaigns, season and news. The fix is to run both versions over the same days and let chance decide who sees which.

Write the hypothesis with a number in it

The template we teach in week 4 of AI Analytics for Everyone is one sentence: "If we [action], then [metric] will [change] by [amount] because [mechanism]." The worked example is a checkout redesign at NovaMart, the fictional online store our course runs on. If we cut checkout from five steps to three, checkout completion will rise 10% relative, from 15.2% to 16.7%, because fewer steps means fewer places to abandon the cart.

Teams skip two of the slots. The amount is what sets the sample size, and without it nobody agrees in advance what success looks like. The mechanism is what a flat result teaches you, because it tells you which link in the chain to go and check.

Work out how many users you need

For a metric that is a rate, like checkout completion, the users you need in each group are:

n = (z_alpha + z_power)^2 x [p1(1 - p1) + p2(1 - p2)] / (p2 - p1)^2

Here p1 is the current rate and p2 is the rate you expect with the change. z_alpha is 1.960 for a 5% false positive rate, and z_power is 0.8416 for an 80% chance of catching the effect if it is real.

For the checkout example, p1 is 0.152 and p2 is 0.1672. The top of the formula is (1.960 + 0.8416)² = 7.849, times 0.152 × 0.848 + 0.1672 × 0.8328 = 0.26814, which gives 2.1046. The bottom is 0.0152² = 0.00023104. Divide, round up, and you need 9,110 users in each group, 18,220 in total.

In the course demo about 1,300 users a day reach checkout, so the test needs 14 days. Shane gives the same minimum in our workshop on experiments for AI features, where he called two weeks "the rule of thumb for a lot of people". Fourteen days also counts every day of the week twice.

Now expect a 5% lift instead of 10%. p2 becomes 0.1596, and the same formula asks for 35,742 users per group, about 55 days of traffic. Our course rule is that a test needing more than six weeks is not viable at that effect size. Halving the effect you want to detect roughly quadrupled the sample. The options are a bigger change, a metric with a higher baseline, or no test.

Write the rules down before launch

The experiment brief in the course has seven sections, and six of them are filled in before any data comes back. These four do most of the work.

One primary metric, with its definition written out, which we call a metric contract. For the checkout test, orders placed divided by checkout sessions started.
Two or three guardrails with thresholds. Average order value must not drop more than 3% (investigate) or 5% (do not launch). Support tickets must not rise more than 10%. More guardrails than that and one of them will trip by chance.
Split by user. A user who sees three steps on Monday and five on Tuesday belongs to neither group.
The decision rule. What result means launch, what means don't, what means run longer.

Written down first, the brief stops you choosing the winning metric after you have seen the numbers. In our agentic experiment design workshop in June, Shane showed an AI analyst that seals the brief before any outcome data is touched, and then refuses to compute the lift at all if the split check fails. The reason he gave: "you're gonna cheat if it looks like it's up".

Don't read it every day

Checking the p-value daily and stopping when it dips under 0.05 raises the false positive rate on a two-week test from 5% to about 25%, the figure we use in the course. Testing 20 metrics at 5% gives a 64% chance that at least one comes up significant with no real effect behind it.

Early lifts also fade. People try the new thing because it is new. With AI features there is a second reason to run longer. In the same workshop, Shane pointed out that users change what they ask once they trust the feature, so the questions it gets months later are harder than the ones it got at launch, and "a two-week test catches kind of, like, the short-term effect". His suggestion was a small holdout group kept for a couple of months.

Check the split, then read the result

A test that should split 50/50 and doesn't means something upstream is losing users from one arm. This is called a sample ratio mismatch. For our comparison of ChatGPT and Claude we built a $1 delivery fee test file with one planted. Control had 13,915 users and treatment 12,604. An even split would put 13,259.5 in each, and a chi-square test on that gap gives p = 8.2e-16. Both tools caught it. Claude Code opened with "The SRM gate fails first." Any lift computed from that file is meaningless.

If the split holds, pick the test by metric type. A rate gets a two-proportion z-test. An average, like order value, gets Welch's t-test. Read the p-value, the confidence interval and the direction. In the course example, checkout completion rose 1.65 points with p = 0.038 and an interval of +0.09 to +3.22 points. Order value fell $0.71 with p = 0.46 and an interval of −$2.60 to +$1.18. That second result is unresolved. The test was sized for completion rate, and order value is noisier, so a real drop of a couple of dollars could hide inside that interval.

Be careful with overall numbers when the groups were not split at random. In the same comparison we used two marketing campaigns. Campaign A beat B on iOS (15% against 14%) and on Android (5% against 4%), and B still won overall, 12% against 6.6%, because most of B's impressions were on iOS, where both campaigns convert better. That is Simpson's paradox, and it is also why month A against month B tells you so little.