courses › Business Stats & Decisions

A/B Tests

level 41 course

Compare two versions fairly, read the result honestly, and ask if it's worth it.

Pen and paper is fine · no calculator needed why?

Opens at level 41.

Builds on: Base Rates & Conditional Rates (not open yet) · Expected Value (not open yet)

the lesson

The idea, the techniques and a tip for each skill, right here. The Learn page adds worked examples for every skill and untimed practice.

Read the lesson · about 3 minutes

The idea

An A/B test shows two versions to comparable groups and compares their rates. Say the difference two ways. Percentage points subtract the rates: 4% to 5% is 1 percentage point. Relative lift divides that by the first rate: 1 ÷ 4 = 25%.

A difference in a test can be luck. A 95% interval shows the range of effects the data allow. If it includes zero, "no effect" hasn't been ruled out. And "significant" means the result would be unlikely with no real effect, not that the effect is big or worth its cost.

Techniques

Percentage points, then lift

Comparing two conversion rates.

Turn each count into a rate: conversions ÷ recipients.
Percentage points: B's rate minus A's rate.
Relative lift: that difference divided by A's rate.

worked example

Example: A gets 30 purchases from 500 recipients; B gets 45 from 500. What is B's relative lift over A?

A = 30 ÷ 500 = 6%, B = 45 ÷ 500 = 9%.
Difference: 3 percentage points.
Lift: 3 ÷ 6 = 0.5 = 50%.

Answer: 50%

Find both ends

Reading an estimate with a 95% interval.

Work out the low and high ends of the interval.
Both ends above zero: the data point to a real lift. Then check it is worth the cost.
Zero inside: unclear. It proves neither a win nor no effect.

worked example

Example: An estimated lift is 3 percentage points, give or take 4. What is the low end of the interval, in percentage points? (Use a minus sign if negative.)

3 − 4 = −1, 3 + 4 = 7, so zero is inside.

Answer: −1

Significant, then worth it

Deciding whether a significant result is worth keeping.

Significant means unlikely if there were no real effect.
Net result: what it adds minus what it costs to run.
A real effect can still lose money.

worked example

Example: A significant change adds $400 a month of contribution (revenue minus variable costs) and costs $550 a month to run. What is its net monthly result? (Use a minus sign if negative.)

$400 − $550 = −$150 a month.
Likely real, but it loses money.

Answer: −$150

Tips by skill

TipDifference in points: Turn both counts into percents, then subtract A from B. The answer is in percentage points.
TipRelative lift: Find both rates, subtract, then divide the difference by A's rate.
TipRead the interval: Is zero inside the interval? Inside: nothing is proven either way. Entirely above zero: the data point to a real lift.
TipSignificant, but worth it?: Significant means unlikely if there were no real effect, not big or worth it. Subtract the running cost from what it adds.
TipDesign a fair test: Pick the plan that splits comparable people at random, at the same time, with the plan fixed before you look.

Watch out for

Giving the relative lift when asked for percentage points, or the reverse. 30% to 36% is 6 percentage points but a 20% lift.
Treating an interval that includes zero as proof of no effect, or a positive estimate as proof of a win.
Assuming a significant result is worth doing. Subtract what it costs to run.
Comparing different audiences or weeks and crediting every difference to the change.

skills · practice stats

From rounds of this course only: box, review and test-out answers are left out. Once a skill has 40 tries, it compares your first 20 tries with your last 20.

Difference in points par 17 s not tried yet
worked example
Variant A converts 48 of 200 recipients; B converts 350 of 1,000. What is the difference in percentage points (B minus A)? (Use a minus sign if negative.)

Answer: 11 points
1. A = 48 ÷ 200 = 24%; B = 350 ÷ 1,000 = 35%.
2. The difference is 11 percentage points. On its own this doesn't prove the difference is real.
Relative lift par 18 s not tried yet
worked example
A gets 4 purchases from 100 recipients; B gets 5 from 100. What is B's relative lift over A?

Answer: 25%
1. Rates: A = 4%, B = 5%.
2. (B − A) ÷ A = 1 ÷ 4 = 25%.
3. That is a 25% relative lift; in absolute terms the rates differ by 1 percentage points.
Read the interval par 19 s not tried yet
worked example
An estimated conversion lift is 1.5 percentage points, with a 95% interval from −2.5 to 5.5 points. What can you conclude?
1. B is proven better
2. The effect is proven to be zero
3. There's a 95% chance the lift is exactly 1.5 points
4. It includes zero, so it doesn't rule out no effect
Answer: It includes zero, so it doesn't rule out no effect
1. The interval runs from −2.5 to 5.5, so zero is inside it: "no effect" hasn't been ruled out, but it hasn't been proven either.
2. An interval shows the range of effects the data are consistent with.
Significant, but worth it? par 18 s not tried yet
worked example
A statistically significant change adds $3,270 of expected contribution a month and costs $3,920 a month to run. What is its net monthly result? (Use a minus sign if negative.)

Answer: -$650.00
1. $3,270 − $3,920 = −$650.
2. "Significant" means the result would be unlikely if there were no real effect, not that the effect is big or worth paying for.
Design a fair test par 13 s not tried yet
worked example
You want to compare two pricing pages. Which plan is fair?
1. Run A this week and B next week
2. Randomly split comparable site visitors between A and B at the same time
3. Give A to longtime customers and B to new prospects
4. Show B only to people who responded last time
Answer: Randomly split comparable site visitors between A and B at the same time
1. Random assignment at the same time makes the groups alike, so the only systematic difference is the thing you changed.
2. Decide the sample size and the metric before you look.

rest ladder

Win 3 of your last 4 rounds and the course rests. A win is 90% right, within 2× the round's par. Pass the review when it comes back and the next rest is longer.

1 day
3 days
7 days
14 days
30 days
60 days
mastered · every 90 days

your rounds

No rounds yet.