Data Traps
Biased samples, false causes, mismatched definitions, and misleading averages.
In your head, jot if needed · no calculator why?
Opens at level 39. You're level 1. You can read and practice here now.
the idea
Numbers can be exact and still mislead. Four questions catch most traps.
Who is missing? A big sample of the wrong people is still wrong. What else changed? Two things moving together can come from a third cause, a reversed cause or chance. Do the definitions match? Two numbers called "sales" can count different things, like contracts and revenue. Is the average typical? One huge job drags the mean far above a normal one.
techniques
Find who's missing
- Ask who the question is about: all customers, all hires, all orders.
- Ask who could get into the sample.
- Anyone left out stays out, however big the sample.
worked example
A cleaning service had 400 customers: 250 renewed and 150 left. To learn why customers leave, it surveys the 250 who renewed. What percent of its customers can the survey never hear from?
- The 150 who left are missing, and they are the people the question is about.
- 150 ÷ 400 = 0.375 = 37.5%.
Answer: 37.5%
Compare like with like
- List what else changed at the same time, or differs between the groups.
- Compare groups that differ only in the one thing.
- If you cannot, the data cannot single out a cause.
worked example
Stores with a café sell $12,000 a week and stores without $8,000, but the café stores are bigger. Among big stores only, it is $12,000 versus $11,000. How big is the weekly gap once size is matched?
- Raw gap: $12,000 − $8,000 = $4,000.
- Matched gap: $12,000 − $11,000 = $1,000.
- Once size is matched, most of the gap disappears.
Answer: $1,000
Mean against median
- Find the median: sort, then take the middle value, or average the middle two.
- Find the mean, the total divided by the count.
- Far apart: big values drag the mean, so use the median. Close: either works.
worked example
Job values: $100, $120, $140, $160, $980. What is the mean job value?
- Total $1,500 ÷ 5 = $300.
- The median is $140.
- The mean is over double the median, so $140 is typical.
Answer: $300
watch out for
- Trusting a large, convenient sample. More of the same people doesn't bring back the ones left out.
- Crediting the most noticeable recent change when several things changed at once.
- Comparing numbers that share a name. Check what each counts, the period and the source.
- Using the mean as the typical value when an outlier distorts it.
practice
Who's missing from the sample?
worked example
To judge product quality, you read only the 500 five-star reviews. Is that sample representative?
- Yes — a big enough sample fixes it
- No — samples can never be trusted
- Yes — five-star reviewers know the product best
- No — unhappy reviewers are missing
Answer: No — unhappy reviewers are missing
- Unhappy reviewers are missing, so the sample leans one way.
- Who gets into the sample matters more than how many: a huge sample that leaves a group out is still wrong.
Cause or coincidence?
worked example
Staff with more training make more errors, but they are given the hardest jobs. Have we isolated the training's effect?
- No — training must cause the errors
- Yes — the difference is big enough to prove it
- Yes — the two moved together, so one caused the other
- No — they get harder jobs, so we can't tell what the training did
Answer: No — they get harder jobs, so we can't tell what the training did
- They get harder jobs, so we can't tell what the training did.
- Two things moving together can come from a third cause, a reversed cause, or chance. You need a comparison that changes only one thing.
Same name, same number?
worked example
One report counts orders placed; another counts orders delivered. Can you compare them as the same "orders" number?
- Yes — just average the two reports
- No — always use whichever is bigger
- No — they measure different things; reconcile the definitions first
- Yes — both are called orders
Answer: No — they measure different things; reconcile the definitions first
- They measure different things: cancelled and pending orders separate the two.
- Before comparing, check what's counted, the period, the source, and the accounting basis.
What's a typical value?
worked example
Job values: $80, $200, $280, $1,690, $1,790. Which number best describes a typical job?
- Either — the mean and median are about the same here
- The largest value
- The median — one or two big jobs pull the mean up
- The mean — it always describes the typical job best
Answer: The median — one or two big jobs pull the mean up
- Median $280 versus mean $808: the big jobs drag the mean up, so the median better describes a typical job.