Mathematics
Sufficient Statistics and the Rao-Blackwell Theorem
Quick fact
The Rao-Blackwell theorem guarantees that you can always improve an unbiased estimator—without adding bias—by averaging it over all datasets that produce the same sufficient statistic. This sometimes transforms a crude guess into the best possible unbiased estimator, with the minimal variance.
Why this is interesting
Imagine you have a raw dataset with thousands of numbers, but you only need a simple summary to estimate a population parameter. Could that summary retain all the statistical information? And could it even help you build a better estimator?
Read the full explanation
Understanding Sufficient Statistics and the Rao-Blackwell Theorem
A sufficient statistic is a function of the data that captures everything the data tells us about a parameter of interest. Think of it as a compressed file that retains all the essential information: any inference you could make from the full data, you can make from the statistic alone. For example, to estimate the mean of a normal population, the sample average is sufficient—the individual data points beyond their average add nothing new for estimating the mean. Now, suppose you have an estimator, a rule for guessing the parameter from data, that is unbiased (on average, it hits the true value) but has high variance. The Rao-Blackwell theorem offers a systematic way to improve it: take your raw estimator and replace it by its conditional expectation given the sufficient statistic. This new estimator is still unbiased (because of the law of total expectation), and its variance is never larger than that of the original estimator. In practice, it 'averages out' the noise that is irrelevant to the parameter, because the sufficient statistic already contains all relevant information. So the theorem provides a two-step recipe for better estimation: first, find a sufficient statistic; second, condition any unbiased estimator on it. The result is an improved estimator that is also a function of the sufficient statistic, meaning it uses the data efficiently.
A deeper explanation
The power of the Rao-Blackwell theorem lies in the variance decomposition. For an estimator T and a sufficient statistic S, the law of total variance states that Var(T) = Var(E[T|S]) + E[Var(T|S)]. Since the second term is non-negative, the variance of the conditioned estimator E[T|S] is always less than or equal to the original. Because S is sufficient, the conditional distribution of the data given S does not depend on the parameter, and hence the conditional expectation E[T|S] is a well-defined estimator—it does not secretly use the unknown parameter. Why does sufficiency matter here? If S were not sufficient, the conditional expectation might depend on the parameter, making the estimator unusable. Sufficiency ensures that conditioning on S removes all parameter-related randomness from T, leaving only the part that is irrelevant to the parameter, which is averaged out to zero. The theorem also guarantees unbiasedness: by the law of total expectation, E[E[T|S]] = E[T] = θ. This is a remarkable improvement because we get a better (or equal) estimator without sacrificing unbiasedness. This theorem is a stepping stone to the Lehmann-Scheffé theorem, which adds completeness to sufficiency to ensure that the Rao-Blackwellized estimator is the unique minimum variance unbiased estimator. The concepts underpin modern statistical theory, connecting to exponential families, decision theory, and the Cramér-Rao lower bound, which gives a universal lower bound on variance that any unbiased estimator must satisfy.