A/B Tests
Compare two versions fairly, read the result honestly, and ask if it's worth it.
Pen and paper is fine · no calculator needed why?
Opens at level 41. You're level 1. You can read and practice here now.
the idea
An A/B test shows two versions to comparable groups and compares their rates. Say the difference two ways. Percentage points subtract the rates: 4% to 5% is 1 percentage point. Relative lift divides that by the first rate: 1 ÷ 4 = 25%.
A difference in a test can be luck. A 95% interval shows the range of effects the data allow. If it includes zero, "no effect" hasn't been ruled out. And "significant" means the result would be unlikely with no real effect, not that the effect is big or worth its cost.
techniques
Percentage points, then lift
- Turn each count into a rate: conversions ÷ recipients.
- Percentage points: B's rate minus A's rate.
- Relative lift: that difference divided by A's rate.
worked example
A gets 30 purchases from 500 recipients; B gets 45 from 500. What is B's relative lift over A?
- A = 30 ÷ 500 = 6%, B = 45 ÷ 500 = 9%.
- Difference: 3 percentage points.
- Lift: 3 ÷ 6 = 0.5 = 50%.
Answer: 50%
Find both ends
- Work out the low and high ends of the interval.
- Both ends above zero: the data point to a real lift. Then check it is worth the cost.
- Zero inside: unclear. It proves neither a win nor no effect.
worked example
An estimated lift is 3 percentage points, give or take 4. What is the low end of the interval, in percentage points? (Use a minus sign if negative.)
- 3 − 4 = −1, 3 + 4 = 7, so zero is inside.
Answer: −1
Significant, then worth it
- Significant means unlikely if there were no real effect.
- Net result: what it adds minus what it costs to run.
- A real effect can still lose money.
worked example
A significant change adds $400 a month of contribution (revenue minus variable costs) and costs $550 a month to run. What is its net monthly result? (Use a minus sign if negative.)
- $400 − $550 = −$150 a month.
- Likely real, but it loses money.
Answer: −$150
watch out for
- Giving the relative lift when asked for percentage points, or the reverse. 30% to 36% is 6 percentage points but a 20% lift.
- Treating an interval that includes zero as proof of no effect, or a positive estimate as proof of a win.
- Assuming a significant result is worth doing. Subtract what it costs to run.
- Comparing different audiences or weeks and crediting every difference to the change.
practice
Difference in points
worked example
Variant A converts 48 of 200 recipients; B converts 350 of 1,000. What is the difference in percentage points (B minus A)? (Use a minus sign if negative.)
Answer: 11 points
- A = 48 ÷ 200 = 24%; B = 350 ÷ 1,000 = 35%.
- The difference is 11 percentage points. On its own this doesn't prove the difference is real.
Relative lift
worked example
A gets 4 purchases from 100 recipients; B gets 5 from 100. What is B's relative lift over A?
Answer: 25%
- Rates: A = 4%, B = 5%.
- (B − A) ÷ A = 1 ÷ 4 = 25%.
- That is a 25% relative lift; in absolute terms the rates differ by 1 percentage points.
Read the interval
worked example
An estimated conversion lift is 1.5 percentage points, with a 95% interval from −2.5 to 5.5 points. What can you conclude?
- B is proven better
- The effect is proven to be zero
- There's a 95% chance the lift is exactly 1.5 points
- It includes zero, so it doesn't rule out no effect
Answer: It includes zero, so it doesn't rule out no effect
- The interval runs from −2.5 to 5.5, so zero is inside it: "no effect" hasn't been ruled out, but it hasn't been proven either.
- An interval shows the range of effects the data are consistent with.
Significant, but worth it?
worked example
A statistically significant change adds $3,270 of expected contribution a month and costs $3,920 a month to run. What is its net monthly result? (Use a minus sign if negative.)
Answer: -$650.00
- $3,270 − $3,920 = −$650.
- "Significant" means the result would be unlikely if there were no real effect, not that the effect is big or worth paying for.
Design a fair test
worked example
You want to compare two pricing pages. Which plan is fair?
- Run A this week and B next week
- Randomly split comparable site visitors between A and B at the same time
- Give A to longtime customers and B to new prospects
- Show B only to people who responded last time
Answer: Randomly split comparable site visitors between A and B at the same time
- Random assignment at the same time makes the groups alike, so the only systematic difference is the thing you changed.
- Decide the sample size and the metric before you look.