Mathematics
Hypothesis Testing with p-Values and Significance Levels
Quick fact
A p-value does not tell you the probability that your hypothesis is true; instead, it tells you the probability of observing data at least as extreme as yours, assuming the null hypothesis is true. Even a p-value of 0.049 does not mean the result is 95% likely to be true.
Why this is interesting
Have you ever wondered how scientists decide if a new drug really works or if the results are just due to chance? Hypothesis testing with p-values is the statistical yardstick that helps answer that question.
Read the full explanation
Understanding Hypothesis Testing with p-Values and Significance Levels
Hypothesis testing starts by assuming the status quo, called the null hypothesis (H₀). For example, 'the new drug has no effect.' The opposite claim, 'the drug does have an effect,' is the alternative hypothesis (H₁). We then collect sample data and compute a test statistic that summarizes how far the sample result is from what H₀ predicts. The p-value is the probability of getting a test statistic as extreme as the one observed (or more extreme) if H₀ were true. A small p-value means the observed data would be unusual under H₀, casting doubt on it. To make a decision, we set a significance level (α), often 0.05. If p ≤ α, we reject H₀ and declare the result statistically significant; if p α, we fail to reject H₀ (which is not the same as saying H₀ is true).
A deeper explanation
The mechanism behind hypothesis testing lies in the sampling distribution. Because sample statistics vary from sample to sample, we need to know how they behave under the null hypothesis. This distribution, often normal via the Central Limit Theorem, tells us what values are expected if H₀ is true. The test statistic (like a z-score) locates our observed sample result within that null distribution. The p-value is the area in the tail(s) beyond that test statistic, representing the probability of a result at least as extreme under H₀. By comparing p to α, we control the long-run rate of false positives—the chance of rejecting H₀ when it is actually true (Type I error). This procedure only tells us whether the evidence is strong enough to reject the null, not the size of the effect or whether the result is practically important.