courses › Business Stats & Decisions

Compare two versions fairly, read the result honestly, and ask if it's worth it.

Pen and paper is fine · no calculator needed why?

Learn first (about 3 minutes)

Opens at level 41.

Sign in to start

Builds on: Base Rates & Conditional Rates (not open yet) · Expected Value (not open yet)

skills · practice stats

From rounds of this course only: box, review and trial answers are left out. Once a skill has 40 tries, it compares your first 20 tries with your last 20.

• Difference in points par 17 s not tried yet

  worked example

  Variant A converts 48 of 200 recipients; B converts 350 of 1,000. What is the difference in percentage points (B minus A)? (Use a minus sign if negative.)

  Answer: 11 points

  1. A = 48 ÷ 200 = 24%; B = 350 ÷ 1,000 = 35%.
  2. The difference is 11 percentage points. On its own this doesn't prove the difference is real.

• Relative lift par 18 s not tried yet

  worked example

  A gets 4 purchases from 100 recipients; B gets 5 from 100. What is B's relative lift over A?

  Answer: 25%

  1. Rates: A = 4%, B = 5%.
  2. (B − A) ÷ A = 1 ÷ 4 = 25%.
  3. That is a 25% relative lift; in absolute terms the rates differ by 1 percentage points.

• Read the interval par 19 s not tried yet

  worked example

  An estimated conversion lift is 1.5 percentage points, with a 95% interval from −2.5 to 5.5 points. What can you conclude?

  1. B is proven better
  2. The effect is proven to be zero
  3. There's a 95% chance the lift is exactly 1.5 points
  4. It includes zero, so it doesn't rule out no effect

  Answer: It includes zero, so it doesn't rule out no effect

  1. The interval runs from −2.5 to 5.5, so zero is inside it: "no effect" hasn't been ruled out, but it hasn't been proven either.
  2. An interval shows the range of effects the data are consistent with.

• Significant, but worth it? par 18 s not tried yet

  worked example

  A statistically significant change adds $3,270 of expected contribution a month and costs $3,920 a month to run. What is its net monthly result? (Use a minus sign if negative.)

  Answer: -$650.00

  1. $3,270 − $3,920 = −$650.
  2. "Significant" means the result would be unlikely if there were no real effect, not that the effect is big or worth paying for.

• Design a fair test par 13 s not tried yet

  worked example

  You want to compare two pricing pages. Which plan is fair?

  1. Run A this week and B next week
  2. Randomly split comparable site visitors between A and B at the same time
  3. Give A to longtime customers and B to new prospects
  4. Show B only to people who responded last time

  Answer: Randomly split comparable site visitors between A and B at the same time

  1. Random assignment at the same time makes the groups alike, so the only systematic difference is the thing you changed.
  2. Decide the sample size and the metric before you look.

rest ladder

Win 3 of your last 4 rounds and the course rests. A win is 90% right, within 2× the round's par. Pass the review when it comes back and the next rest is longer.

1. 1 day
2. 3 days
3. 7 days
4. 14 days
5. 30 days
6. 60 days
7. mastered · every 90 days

your rounds

No rounds yet.