Hypothesis Testing
Prerequisites: Linear regression, Probability, sampling, and simulation
A hypothesis test asks whether a pattern in your sample is strong enough to suggest it's real, rather than a fluke of random sampling.
The logic: assume there's no effect (the null hypothesis), then ask how surprising your data would be under that assumption. That surprise is the p-value — the probability of seeing results at least this extreme if the null hypothesis were true.
Comparing two groups: t-test
Do automatic and manual cars differ in fuel economy?
result <- t.test(mpg ~ am, data = mtcars)
round(result$estimate, 2)
round(result$p.value, 4)
round(result$conf.int, 2)
mean in group 0 mean in group 1
17.15 24.39
[1] 0.0014
[1] -11.28 -3.21
attr(,"conf.level")
[1] 0.95
- Automatic cars (
am = 0) average 17.2 mpg; manual cars 24.4. - The p-value (0.0014) says a difference this large would be very unlikely if the true averages were equal.
- The 95% confidence interval for the difference is about -11.3 to -3.2 mpg — more useful than the p-value alone, because it shows the likely size of the effect.
(By default R uses Welch's t-test, which doesn't assume the two groups have equal variance.)
Comparing categories: chi-squared test
Is the number of cylinders related to transmission type?
counts <- table(mtcars$cyl, mtcars$am)
counts
round(chisq.test(counts)$p.value, 4)
0 1
4 3 8
6 4 3
8 12 2
[1] 0.0126
Warning message:
In chisq.test(counts) : Chi-squared approximation may be incorrect
The warning appears because some expected counts are small (fewer than
5); with small samples, fisher.test(counts) gives an exact p-value
instead.
Other common tests
| Question | Test |
|---|---|
| Two group means (paired measurements) | t.test(x, y, paired = TRUE) |
| Two groups, non-normal or ordinal data | wilcox.test() |
| Three or more group means | aov() (ANOVA) |
| Relationship between two numeric variables | cor.test() |
| Proportions | prop.test() |
Using p-values responsibly
- A p-value isn't the probability that the null hypothesis is true, and "p < 0.05" isn't a magic line between real and fake effects.
- Statistical significance isn't practical importance: with enough data, trivially small differences become "significant". Report effect sizes and confidence intervals.
- Running many tests and reporting only the significant ones produces
false discoveries. Decide your analysis in advance, or correct for
multiple comparisons (
p.adjust()).