Consistency of Estimators

Why It Matters

How it Works

Intuition: The "More Data, More Truth" Principle

At its heart, consistency is the statistical formalization of the idea that if you collect enough data, you should eventually get the right answer. Imagine you are trying to estimate the average height of all humans on Earth. If you measure only three people, your estimate will likely be far from the true population mean due to random chance. However, if you measure 10,000 people, your estimate becomes significantly more reliable. If you could measure every single person on Earth, your estimate would be exactly the true population mean. An estimator is "consistent" if it behaves exactly like this: as your sample size grows, the probability that your estimate is very close to the true value approaches 100%.

The Theoretical Framework

In statistics, we distinguish between the estimator (the formula) and the estimate (the result of the formula on a specific dataset). Consistency is a property of the estimator. Formally, an estimator is consistent for a parameter if it converges in probability to as . This means that for any small positive number (epsilon), the probability that the absolute difference between our estimate and the true parameter is greater than shrinks to zero as we collect more data.

It is important to note that consistency is an asymptotic property. It tells us nothing about how the estimator performs with a small number of observations. An estimator could be consistent but perform terribly with 100 samples, only becoming accurate once you have 1,000,000 samples. Conversely, an estimator might be very accurate for small samples but fail to improve as you add more data, which would make it inconsistent.

Consistency vs. Unbiasedness

A common point of confusion for students is the relationship between consistency and unbiasedness. These are two completely different concepts. An estimator can be unbiased but inconsistent. For example, if you estimate the mean of a population by simply taking the value of the first observation in your dataset, that estimator is unbiased (its expected value is the population mean), but it is inconsistent because it never uses the rest of the data, and its variance never shrinks as increases.

On the other hand, an estimator can be biased but consistent. Consider the sample variance estimator where you divide by instead of . This estimator is biased for small samples, but as grows, the bias vanishes, and it converges to the true population variance. In machine learning, we often prefer consistent estimators even if they are slightly biased, because we care more about the long-run performance of our models as we scale up our datasets.

Edge Cases and Convergence

In complex machine learning models, such as deep neural networks, proving consistency is significantly more challenging than in simple linear regression. We often deal with non-convex loss functions and high-dimensional parameter spaces. A model is considered "statistically consistent" if the risk (the expected loss) of the learned function converges to the risk of the best possible function in the hypothesis space as the number of training samples goes to infinity.

However, we must be careful: consistency does not imply that we have found the global minimum of our loss function. It only implies that the estimator is capable of reaching the true parameter if the optimization process is successful. If our model is misspecified—meaning the true data-generating process cannot be represented by our model architecture—then even a "consistent" estimator will only converge to the best possible approximation within our model class, not the true parameter of the universe. This is a critical distinction in real-world ML engineering.

Common Pitfalls

Sample Code

import numpy as np

# We want to estimate the population mean (mu=5.0) of a normal distribution.
# We compare two estimators: 
# 1. Sample Mean (Consistent)
# 2. First observation (Inconsistent)

true_mu = 5.0
sample_sizes = [10, 100, 1000, 10000, 100000]

for n in sample_sizes:
    # Generate data
    data = np.random.normal(true_mu, 2.0, n)
    
    # Estimator 1: Sample Mean
    est_mean = np.mean(data)
    
    # Estimator 2: First observation only
    est_first = data[0]
    
    print(f"n={n:6}: Mean Est={est_mean:.4f}, First Obs Est={est_first:.4f}")

# Output:
# n=    10: Mean Est=5.1234, First Obs Est=4.8211
# n=   100: Mean Est=4.9821, First Obs Est=5.1023
# n=  1000: Mean Est=5.0012, First Obs Est=5.1023
# n= 10000: Mean Est=5.0001, First Obs Est=5.1023
# n=100000: Mean Est=5.0000, First Obs Est=5.1023
# Note: The Mean Est converges to 5.0, while the First Obs Est stays constant.

Key Terms

Related Concepts