courses › Business Stats & Decisions

Data Traps

level 39 course

Biased samples, false causes, mismatched definitions, and misleading averages.

In your head, jot if needed · no calculator why?

Opens at level 39.

Builds on: Averages & Spread (not open yet)

the lesson

The idea, the techniques and a tip for each skill, right here. The Learn page adds worked examples for every skill and untimed practice.

Read the lesson · about 3 minutes

The idea

Numbers can be exact and still mislead. Four questions catch most traps.

Who is missing? A big sample of the wrong people is still wrong. What else changed? Two things moving together can come from a third cause, a reversed cause or chance. Do the definitions match? Two numbers called "sales" can count different things, like contracts and revenue. Is the average typical? One huge job drags the mean far above a normal one.

Techniques

Find who's missing

Judging whether a sample stands for the whole group.

Ask who the question is about: all customers, all hires, all orders.
Ask who could get into the sample.
Anyone left out stays out, however big the sample.

worked example

Example: A cleaning service had 400 customers: 250 renewed and 150 left. To learn why customers leave, it surveys the 250 who renewed. What percent of its customers can the survey never hear from?

The 150 who left are missing, and they are the people the question is about.
150 ÷ 400 = 0.375 = 37.5%.

Answer: 37.5%

Compare like with like

Deciding whether a change caused a result.

List what else changed at the same time, or differs between the groups.
Compare groups that differ only in the one thing.
If you cannot, the data cannot single out a cause.

worked example

Example: Stores with a café sell $12,000 a week and stores without $8,000, but the café stores are bigger. Among big stores only, it is $12,000 versus $11,000. How big is the weekly gap once size is matched?

Raw gap: $12,000 − $8,000 = $4,000.
Matched gap: $12,000 − $11,000 = $1,000.
Once size is matched, most of the gap disappears.

Answer: $1,000

Mean against median

Deciding which number describes a typical value.

Find the median: sort, then take the middle value, or average the middle two.
Find the mean, the total divided by the count.
Far apart: big values drag the mean, so use the median. Close: either works.

worked example

Example: Job values: $100, $120, $140, $160, $980. What is the mean job value?

Total $1,500 ÷ 5 = $300.
The median is $140.
The mean is over double the median, so $140 is typical.

Answer: $300

Tips by skill

TipWho's missing from the sample?: Ask who the question is about, then who could get into the sample. The group left out is the problem.
TipCause or coincidence?: Look for anything else that changed or differs between the groups. If there is one, the effect is not isolated.
TipSame name, same number?: Check what each number counts, over what period and from what source. Different definitions are not one number.
TipWhat's a typical value?: Compare the mean with the median. Far apart: the median is typical. About the same: either works.

Watch out for

Trusting a large, convenient sample. More of the same people doesn't bring back the ones left out.
Crediting the most noticeable recent change when several things changed at once.
Comparing numbers that share a name. Check what each counts, the period and the source.
Using the mean as the typical value when an outlier distorts it.

skills · practice stats

From rounds of this course only: box, review and test-out answers are left out. Once a skill has 40 tries, it compares your first 20 tries with your last 20.

Who's missing from the sample? par 14 s not tried yet
worked example
To judge product quality, you read only the 500 five-star reviews. Is that sample representative?
1. Yes — a big enough sample fixes it
2. No — samples can never be trusted
3. Yes — five-star reviewers know the product best
4. No — unhappy reviewers are missing
Answer: No — unhappy reviewers are missing
1. Unhappy reviewers are missing, so the sample leans one way.
2. Who gets into the sample matters more than how many: a huge sample that leaves a group out is still wrong.
Cause or coincidence? par 15 s not tried yet
worked example
Staff with more training make more errors, but they are given the hardest jobs. Have we isolated the training's effect?
1. No — training must cause the errors
2. Yes — the difference is big enough to prove it
3. Yes — the two moved together, so one caused the other
4. No — they get harder jobs, so we can't tell what the training did
Answer: No — they get harder jobs, so we can't tell what the training did
1. They get harder jobs, so we can't tell what the training did.
2. Two things moving together can come from a third cause, a reversed cause, or chance. You need a comparison that changes only one thing.
Same name, same number? par 15 s not tried yet
worked example
One report counts orders placed; another counts orders delivered. Can you compare them as the same "orders" number?
1. Yes — just average the two reports
2. No — always use whichever is bigger
3. No — they measure different things; reconcile the definitions first
4. Yes — both are called orders
Answer: No — they measure different things; reconcile the definitions first
1. They measure different things: cancelled and pending orders separate the two.
2. Before comparing, check what's counted, the period, the source, and the accounting basis.
What's a typical value? par 12 s not tried yet
worked example
Job values: $80, $200, $280, $1,690, $1,790. Which number best describes a typical job?
1. Either — the mean and median are about the same here
2. The largest value
3. The median — one or two big jobs pull the mean up
4. The mean — it always describes the typical job best
Answer: The median — one or two big jobs pull the mean up
1. Median $280 versus mean $808: the big jobs drag the mean up, so the median better describes a typical job.

rest ladder

Win 3 of your last 4 rounds and the course rests. A win is 90% right, within 2× the round's par. Pass the review when it comes back and the next rest is longer.

1 day
3 days
7 days
14 days
30 days
60 days
mastered · every 90 days

your rounds

No rounds yet.