Statistics Paper I: Core Topics
Learning Objectives
- Understand the fundamental concepts of Statistics Paper I: Core Topics
- Apply key formulas and techniques to solve problems
- Practice with exam-level questions to build speed and accuracy
1. Foundations of Probability Theory
1.1 The Axiomatic Framework of Kolmogorov
The modern theory of probability rests on the axiomatic system introduced by Andrey Kolmogorov in 1933, which placed the subject on the same rigorous measure-theoretic footing as analysis. A probability space is defined as a triple consisting of a sample space Ω containing all elementary outcomes, a σ-algebra ℱ of subsets of Ω representing events, and a probability measure P mapping ℱ to the interval [0, 1]. The three axioms require that the probability of any event be non-negative, that the probability of the entire sample space equal unity, and that the probability of a countable union of mutually exclusive events equal the sum of their individual probabilities. This last axiom, countable additivity, distinguishes the modern theory from earlier finite-additivity approaches and enables the treatment of continuous sample spaces and limiting arguments essential for statistical inference. The axioms imply immediate consequences such as the probability of the empty set being zero, the complement rule P(Aᶜ) = 1 − P(A), and the monotonicity property that A ⊆ B implies P(A) ≤ P(B). These foundational results allow the construction of more complex probabilistic models while guaranteeing internal consistency.
1.2 Conditional Probability and the Multiplication Rule
Conditional probability quantifies how the occurrence of one event alters the likelihood of another. For events A and B with P(B) > 0, the conditional probability of A given B is defined as P(A|B) = P(A ∩ B) / P(B). This definition captures the intuitive idea of restricting the sample space to those outcomes where B has occurred and then measuring the relative frequency of A within that reduced space. The multiplication rule P(A ∩ B) = P(A|B)P(B) follows directly and extends to chains of events, yielding P(A₁ ∩ A₂ ∩ … ∩ Aₙ) = P(A₁)P(A₂|A₁)⋯P(Aₙ|A₁∩…∩Aₙ₋₁). This factorization is indispensable in sequential experiments and in the derivation of likelihood functions for dependent data. The law of total probability partitions the sample space using a countable collection of mutually exclusive and exhaustive events {Bᵢ}, giving P(A) = Σᵢ P(A|Bᵢ)P(Bᵢ). This decomposition underpins Bayesian updating, where prior probabilities P(Bᵢ) are revised to posterior probabilities P(Bᵢ|A) upon observing evidence A.
1.3 Bayes' Theorem and Its Inferential Role
Bayes' theorem, P(Bᵢ|A) = P(A|Bᵢ)P(Bᵢ) / Σⱼ P(A|Bⱼ)P(Bⱼ), provides the mathematical mechanism for inverting conditional probabilities. The numerator combines the likelihood of the observed data under hypothesis Bᵢ with the prior probability of that hypothesis; the denominator acts as a normalizing constant ensuring the posterior probabilities sum to one. In statistical inference, the theorem formalizes the process of learning from data: prior beliefs about parameters are updated to posterior beliefs through the likelihood function. The theorem also reveals the base-rate fallacy, where neglect of the prior P(Bᵢ) leads to overestimation of rare-event probabilities. Applications range from diagnostic testing, where sensitivity and specificity must be combined with disease prevalence, to spam filtering, where word frequencies update the probability that a message is unwanted. The continuous analogue replaces sums with integrals and probability mass functions with densities, yielding the posterior density π(θ|x) ∝ f(x|θ)π(θ), which is the cornerstone of Bayesian statistics.
1.4 Independence and Its Implications
Two events A and B are independent when P(A ∩ B) = P(A)P(B), which is equivalent to P(A|B) = P(A) whenever P(B) > 0. Independence means that knowledge of B provides no information about A, and vice versa. The concept extends to collections of events: a finite family {A₁, …, Aₙ} is mutually independent if the probability of any intersection equals the product of the individual probabilities. Pairwise independence does not imply mutual independence, a subtle point illustrated by the classic example of two fair coin tosses where A = {first head}, B = {second head}, C = {same outcome} are pairwise independent but not mutually independent. For random variables, independence means the joint distribution factorizes into the product of marginals. This factorization simplifies the computation of expectations of products, since E[XY] = E[X]E[Y] for independent variables, and it underlies the additivity of variance for independent sums: Var(ΣXᵢ) = ΣVar(Xᵢ). Independence is a modeling assumption that must be justified by the physical mechanism of the experiment; its violation leads to biased standard errors and invalid hypothesis tests.
2. Random Variables and Mathematical Expectation
2.1 Definition and Classification of Random Variables
A random variable is a measurable function from the sample space Ω to the real line ℝ, assigning a numerical value to each elementary outcome. Measurability ensures that events of the form {X ≤ x} belong to the σ-algebra, so probabilities can be assigned to them. Random variables are classified by the nature of their range: discrete random variables take at most countably many values and are described by a probability mass function p(x) = P(X = x); continuous random variables possess a probability density function f(x) such that P(a ≤ X ≤ b) = ∫ₐᵇ f(x) dx; mixed random variables combine both behaviors, having a probability mass at some points and a density elsewhere. The cumulative distribution function F(x) = P(X ≤ x) unifies these cases, being non-decreasing, right-continuous, with limits 0 at −∞ and 1 at +∞. For discrete variables F is a step function; for continuous variables it is absolutely continuous with derivative f(x) almost everywhere. The quantile function F⁻¹(p) = inf{x : F(x) ≥ p} provides the inverse mapping used in simulation and in defining measures of spread such as the interquartile range.
2.2 Expectation, Variance, and Moment Generating Functions
The expectation of a random variable represents its long-run average value under repeated independent sampling. For a discrete variable, E[X] = Σ x p(x); for a continuous variable, E[X] = ∫ x f(x) dx, provided the sum or integral converges absolutely. Expectation is linear: E[aX + bY + c] = aE[X] + bE[Y] + c for any constants a, b, c and random variables X, Y, regardless of dependence. The variance Var(X) = E[(X − μ)²] = E[X²] − μ² measures dispersion about the mean μ = E[X]. The standard deviation σ = √Var(X) restores the original units. Higher central moments characterize shape: the third standardized moment γ₁ = E[(X−μ)³]/σ³ measures skewness, indicating asymmetry; the fourth standardized moment γ₂ = E[(X−μ)⁴]/σ³ − 3 measures excess kurtosis, indicating tail heaviness relative to the normal distribution. The moment generating function M_X(t) = E[etX], when it exists in a neighborhood of zero, uniquely determines the distribution and generates moments via differentiation: E[Xᵏ] = M_X⁽ᵏ⁾(0). The characteristic function φ_X(t) = E[eitX] always exists and provides an alternative transform useful for proving limit theorems. The cumulant generating function K_X(t) = log M_X(t) yields cumulants, which simplify under addition of independent variables: the nth cumulant of a sum equals the sum of nth cumulants.
2.3 Conditional Expectation and Regression
Conditional expectation E[Y|X = x] generalizes the idea of an average to subpopulations defined by the value of another variable. For discrete X, E[Y|X = x] = Σ y p(y|x); for continuous X, it is ∫ y f(y|x) dy. The law of iterated expectations, E[Y] = E[E[Y|X]], states that the overall mean equals the average of the conditional means, a result used repeatedly in variance decomposition: Var(Y) = E[Var(Y|X)] + Var(E[Y|X]). The conditional expectation function m(x) = E[Y|X = x] is the best predictor of Y given X in the sense of minimizing mean squared error among all functions of X. The best linear predictor, obtained by restricting to functions of the form a + bX, yields the regression line Y = α + βX with β = Cov(X,Y)/Var(X) and α = E[Y] − βE[X]. The correlation coefficient ρ = Cov(X,Y)/(σ_X σ_Y) measures the strength of linear association, bounded between −1 and 1 by the Cauchy–Schwarz inequality. When (X,Y) is bivariate normal, the conditional expectation is exactly linear and the conditional variance is constant, making the regression line coincide with the conditional mean function.
3. Standard Probability Distributions
3.1 Discrete Distributions and Their Interrelationships
The Bernoulli distribution with parameter p models a single trial with two outcomes, success (1) and failure (0). The binomial distribution Bin(n, p) arises as the sum of n independent Bernoulli(p) variables, counting the number of successes in n trials. Its probability mass function is P(X = k) = C(n, k) pᵏ (1−p)n−k for k = 0, 1, …, n, with mean np and variance np(1−p). The Poisson distribution Pois(λ) models the number of rare events occurring in a fixed interval, with pmf P(X = k) = e−λ λᵏ / k! and equal mean and variance λ. It emerges as the limiting distribution of Bin(n, p) when n → ∞, p → 0 with np = λ fixed, providing a practical approximation when n is large and p is small. The geometric distribution Geo(p) counts the number of trials until the first success, with pmf P(X = k) = (1−p)k−1 p for k = 1, 2, … and mean 1/p. The negative binomial NB(r, p) generalizes this to the number of trials until the r-th success, with mean r/p. The hypergeometric distribution Hyp(N, K, n) describes sampling without replacement from a finite population of size N containing K successes, with pmf P(X = k) = C(K, k) C(N−K, n−k) / C(N, n). When N is large relative to n, the hypergeometric is well approximated by the binomial, reflecting the diminishing effect of sampling without replacement.
3.2 Continuous Distributions and Their Properties
The uniform distribution U(a, b) assigns constant density 1/(b−a) on the interval [a, b], serving as the baseline for generating other distributions via transformation methods. The exponential distribution Exp(λ) with density λe−λx for x ≥ 0 is the only continuous distribution possessing the memoryless property P(X > s+t | X > s) = P(X > t), making it the natural model for waiting times in a Poisson process. The gamma distribution Gamma(α, β) with density (β^α/Γ(α)) xα−1 e−βx for x > 0 generalizes the exponential (α = 1) and the chi-square (α = k/2, β = 1/2). Its mean is α/β and variance α/β². The beta distribution Beta(α, β) with density xα−1(1−x)β−1/B(α, β) on (0, 1) is the conjugate prior for the binomial proportion and models random probabilities. The normal distribution N(μ, σ²) with density (1/√(2πσ²)) e−(x−μ)²/(2σ²) occupies a central role due to the central limit theorem; its symmetry, bell shape, and the 68–95–99.7 rule make it the default approximation for sums of independent random variables. The chi-square distribution with k degrees of freedom is the distribution of the sum of k independent standard normal squares; it underlies goodness-of-fit tests and variance inference. The t-distribution with ν degrees of freedom arises as the ratio of a standard normal to the square root of an independent chi-square divided by its degrees of freedom; its heavier tails account for the uncertainty in estimating σ. The F-distribution with (d₁, d₂) degrees of freedom is the ratio of two independent chi-squares divided by their respective degrees of freedom; it is the reference distribution for analysis of variance and regression F-tests.
3.3 Approximations and Transformations Between Distributions
Relationships among distributions are not merely theoretical curiosities; they provide computational shortcuts and justify asymptotic procedures. The normal approximation to the binomial, with continuity correction, is accurate when np(1−p) > 5. The Poisson approximation to the binomial works when n is large and p is small. The gamma distribution with integer shape parameter α = r is the distribution of the sum of r independent exponential(β) variables, linking waiting times for the r-th event in a Poisson process to the gamma. The beta distribution with parameters (α, β) is the distribution of the α-th order statistic from a uniform sample of size α+β−1. The square of a standard normal is chi-square with one degree of freedom; the sum of independent chi-squares is chi-square with degrees of freedom added. The ratio of two independent chi-squares divided by their degrees of freedom follows the F-distribution. The t-statistic t = (X̄ − μ)/(S/√n) follows a t-distribution with n−1 degrees of freedom when sampling from a normal population, providing the basis for small-sample inference on the mean. These connections form a web that allows the statistician to move between exact and approximate distributions depending on sample size and modeling assumptions.
| Distribution | Support | PMF | Mean | Variance | Primary Use |
|---|---|---|---|---|---|
| Bernoulli(p) | {0, 1} | pˣ(1−p)1−x | p | p(1−p) | Single binary trial |
| Binomial(n, p) | {0, 1, …, n} | C(n, x) pˣ(1−p)n−x | np | np(1−p) | Number of successes in n independent trials |
| Poisson(λ) | {0, 1, 2, …} | e−λ λˣ / x! | λ | λ | Count of rare events in fixed interval |
| Geometric(p) | {1, 2, …} | (1−p)x−1 p | 1/p | (1−p)/p² | Trials until first success |
| Negative Binomial(r, p) | {r, r+1, …} | C(x−1, r−1) pʳ(1−p)x−r | r/p | r(1−p)/p² | Trials until r-th success |
| Hypergeometric(N, K, n) | {max(0, n−N+K), …, min(n, K)} | C(K, x)C(N−K, n−x)/C(N, n) | nK/N | n(K/N)(1−K/N)(N−n)/(N−1) | Sampling without replacement |
4. Joint Distributions and Correlation
4.1 Joint, Marginal, and Conditional Distributions
When two or more random variables are observed simultaneously, their joint behavior is described by a joint probability mass function p(x, y) for discrete variables or a joint probability density function f(x, y) for continuous variables. The marginal distribution of X is obtained by summing or integrating over the other variable: p_X(x) = Σ_y p(x, y) or f_X(x) = ∫ f(x, y) dy. The conditional distribution of Y given X = x is defined as p(y|x) = p(x, y)/p_X(x) for discrete variables and f(y|x) = f(x, y)/f_X(x) for continuous variables, provided the denominator is positive. Independence is equivalent to the factorization p(x, y) = p_X(x)p_Y(y) or f(x, y) = f_X(x)f_Y(y) for all x, y. This factorization implies that conditional distributions equal marginal distributions, and that the joint cumulative distribution function factorizes as F(x, y) = F_X(x)F_Y(y). The support of the joint distribution reveals dependence: if the set of possible (x, y) pairs is not a Cartesian product, the variables cannot be independent. For example, the uniform distribution on the unit disc has circular support, so X and Y are dependent despite having symmetric marginals.
4.2 Covariance, Correlation, and Linear Relationships
The covariance Cov(X, Y) = E[(X − μ_X)(Y − μ_Y)] = E[XY] − μ_Xμ_Y measures the average product of deviations from the means. Positive covariance indicates that large values of X tend to occur with large values of Y; negative covariance indicates the opposite. The correlation coefficient ρ = Cov(X, Y)/(σ_X σ_Y) standardizes covariance to the interval [−1, 1], removing the effect of scale. A correlation of ±1 implies a perfect linear relationship Y = a + bX almost surely, with b having the same sign as ρ. Zero correlation (uncorrelatedness) does not imply independence except in the special case of jointly normal variables. For bivariate normal (X, Y), zero correlation is equivalent to independence because the joint density factorizes when ρ = 0. The Cauchy–Schwarz inequality guarantees |ρ| ≤ 1, with equality only when Y is a linear function of X. The coefficient of determination ρ² represents the proportion of variance in Y explained by the linear regression on X. In multivariate settings, the covariance matrix Σ = E[(X − μ)(X − μ)ᵀ] and the correlation matrix R with entries ρij summarize all pairwise linear associations.
4.3 Bivariate Normal Distribution and Its Properties
The bivariate normal distribution is the most important joint distribution in statistical practice. Its density is f(x, y) = 1/(2πσ_Xσ_Y√(1−ρ²)) exp{−1/(2(1−ρ²)) [ (x−μ_X)²/σ_X² − 2ρ(x−μ_X)(y−μ_Y)/(σ_Xσ_Y) + (y−μ_Y)²/σ_Y² ] }. Every linear combination aX + bY is normally distributed, and any subset of components is multivariate normal. The conditional distribution of Y given X = x is normal with mean μ_Y + ρ(σ_Y/σ_X)(x − μ_X) and variance σ_Y²(1 − ρ²). This linearity of conditional expectation and constancy of conditional variance are characteristic properties of the bivariate normal. The regression of Y on X is exactly linear, and the regression of X on Y is also linear, with slopes βY|X = ρσ_Y/σ_X and βX|Y = ρσ_X/σ_Y. The product of the two slopes equals ρ². The bivariate normal arises naturally as the distribution of the sample mean and sample variance in normal sampling, and as the limiting distribution of sample moments under mild conditions.
| Distribution | Support | Mean | Variance | Key Property | |
|---|---|---|---|---|---|
| Uniform(a, b) | [a, b] | 1/(b−a) | (a+b)/2 | (b−a)²/12 | Constant density; basis for inverse transform sampling |
| Exponential(λ) | [0, ∞) | λe−λx | 1/λ | 1/λ² | Memoryless; inter-arrival times in Poisson process |
| Gamma(α, β) | (0, ∞) | β^α xα−1e−βx/Γ(α) | α/β | α/β² | Sum of α independent Exp(β); conjugate prior for Poisson rate |
| Beta(α, β) | (0, 1) | xα−1(1−x)β−1/B(α, β) | α/(α+β) | αβ/((α+β)²(α+β+1)) | Conjugate prior for binomial proportion; models random probabilities |
| Normal(μ, σ²) | (−∞, ∞) | e−(x−μ)²/(2σ²)/√(2πσ²) | μ | σ² | CLT limit; maximum entropy for given variance |
| Chi-square(k) | [0, ∞) | xk/2−1e−x/2/(2k/2Γ(k/2)) | k | 2k | Sum of k independent Z²; variance inference |
| t(ν) | (−∞, ∞) | Γ((ν+1)/2)/(√(νπ)Γ(ν/2)) (1+x²/ν)−(ν+1)/2 | 0 (ν>1) | ν/(ν−2) (ν>2) | Heavier tails than normal; small-sample mean inference |
| F(d₁, d₂) | [0, ∞) | Complex ratio of gamma functions | d₂/(d₂−2) (d₂>2) | Complex | Ratio of independent chi-squares/df; ANOVA and regression |
5. Convergence Concepts and Limit Theorems
5.1 Modes of Convergence for Random Variables
Convergence of sequences of random variables is more nuanced than convergence of real numbers because random variables are functions on a probability space. Convergence in probability, denoted Xₙ →ₚ X, means that for every ε > 0, P(|Xₙ − X| > ε) → 0 as n → ∞. This mode captures the idea that the probability of a large deviation vanishes. Almost sure convergence, Xₙ → X a.s., requires P(limn→∞ Xₙ = X) = 1, meaning the set of sample paths where convergence fails has probability zero. Almost sure convergence implies convergence in probability, but the converse is false; a sequence can converge in probability without converging almost surely, as illustrated by the "typewriter" sequence of indicators on shrinking intervals. Convergence in distribution, Xₙ →_d X, means the cumulative distribution functions converge at all continuity points of the limit distribution: FXₙ(x) → F_X(x). This is the weakest mode, concerned only with the limiting distribution, not with the joint behavior of the sequence. Convergence in probability implies convergence in distribution, but not vice versa. Convergence in L^p (mean of order p) means E[|Xₙ − X|^p] → 0; L^p convergence implies convergence in probability by Markov's inequality. These hierarchies are essential for understanding the strength of various limit theorems and for justifying asymptotic approximations in statistical inference.
5.2 Laws of Large Numbers
The weak law of large numbers (WLLN) states that for a sequence of independent and identically distributed random variables with finite mean μ, the sample mean X̄ₙ converges in probability to μ. Chebyshev's inequality provides a simple proof when the variance is finite: P(|X̄ₙ − μ| ≥ ε) ≤ Var(X)/(nε²) → 0. The strong law of large numbers (SLLN) asserts almost sure convergence of X̄ₙ to μ under the same i.i.d. assumption with finite mean. The SLLN is a deeper result, requiring measure-theoretic tools such as the Borel–Cantelli lemmas or martingale convergence. Both laws justify the frequentist interpretation of probability as long-run relative frequency. The WLLN suffices for most statistical applications because convergence in probability is preserved under continuous transformations (Slutsky's theorem), allowing the construction of consistent estimators. The SLLN guarantees that the sample mean eventually stays arbitrarily close to the population mean for almost every realization, a stronger assurance for simulation and ergodic theory.
5.3 Central Limit Theorem and Its Extensions
The central limit theorem (CLT) is the cornerstone of asymptotic statistics. For i.i.d. random variables with finite mean μ and finite variance σ², the standardized sum (Σi=1^n X_i − nμ)/(σ√n) = √n (X̄ₙ − μ)/σ converges in distribution to a standard normal N(0, 1). This remarkable result holds regardless of the shape of the underlying distribution, explaining the ubiquity of the normal approximation. The Lindeberg–Lévy CLT is the classical version; the Lindeberg–Feller CLT extends it to independent but not identically distributed variables under a Lindeberg condition that prevents any single variable from dominating the sum. The Lyapunov CLT provides a simpler sufficient condition involving moments of order 2+δ. The CLT justifies the use of normal critical values for t-statistics and z-statistics in large samples, the construction of asymptotic confidence intervals, and the derivation of the asymptotic distribution of maximum likelihood estimators. The delta method extends the CLT to smooth functions of the sample mean: if √n (X̄ₙ − μ) →_d N(0, σ²), then √n (g(X̄ₙ) − g(μ)) →_d N(0, [g'(μ)]²σ²) for differentiable g. This enables inference on transformed parameters such as log-odds or variance-stabilizing transformations.
6. Statistical Inference and Sampling Theory
6.1 Point Estimation: Principles and Criteria
Point estimation seeks a single value, derived from the sample, that approximates an unknown population parameter. An estimator is a statistic, a function of the sample, used to estimate the parameter. Unbiasedness requires E[θ̂] = θ for all θ in the parameter space; it ensures that the estimator is correct on average over repeated sampling. The mean squared error MSE(θ̂) = E[(θ̂ − θ)²] = Var(θ̂) + Bias(θ̂)² decomposes estimation error into variance and squared bias. Among unbiased estimators, the one with minimum variance is the minimum variance unbiased estimator (MVUE). The Cramér–Rao lower bound states that for any unbiased estimator of a differentiable parameter, Var(θ̂) ≥ 1/I(θ), where I(θ) = E[(∂/∂θ log f(X;θ))²] is the Fisher information. An unbiased estimator achieving this bound is efficient. The Rao–Blackwell theorem improves an unbiased estimator by conditioning on a sufficient statistic: if T is sufficient and U is unbiased, then E[U|T] is unbiased with variance no larger than U. The Lehmann–Scheffé theorem completes the picture: if T is complete and sufficient, then any unbiased function of T is the unique MVUE. These results provide a systematic pathway to optimal estimation: find a sufficient statistic, check completeness, and construct an unbiased function of it.
6.2 Maximum Likelihood Estimation and Its Properties
The method of maximum likelihood chooses the parameter value that maximizes the likelihood function L(θ) = f(x₁, …, xₙ; θ), or equivalently the log-likelihood ℓ(θ) = log L(θ). The MLE θ̂ satisfies the likelihood equation ∂ℓ/∂θ = 0, provided regularity conditions hold. Under standard regularity, the MLE is consistent, asymptotically normal with variance equal to the inverse Fisher information, and asymptotically efficient, achieving the Cramér–Rao bound in the limit. The invariance property states that if θ̂ is the MLE of θ, then g(θ̂) is the MLE of g(θ) for any function g. This simplifies estimation of transformed parameters. The observed Fisher information −∂²ℓ/∂θ² evaluated at θ̂ provides an estimate of the asymptotic variance, used to construct Wald confidence intervals. The likelihood ratio statistic λ = 2[ℓ(θ̂) − ℓ(θ₀)] converges in distribution to chi-square with degrees of freedom equal to the number of restrictions, providing the basis for likelihood ratio tests. When the likelihood is intractable, numerical optimization such as Newton–Raphson or EM algorithms is employed.
Interval Estimation: Confidence Intervals and Pivot Methods
Interval estimation replaces a single point estimate with a random interval that covers the unknown parameter with a prescribed probability. A confidence interval (CI) for a parameter θ at confidence level 1−α is a pair of statistics (L, U) such that Pθ(L ≤ θ ≤ U) = 1−α for all θ in the parameter space. The construction typically relies on a pivot — a function of the sample and the parameter whose distribution does not depend on θ. For a normal sample with known variance σ², the pivot Z = (X̄ − μ)/(σ/√n) follows a standard normal distribution, yielding the familiar symmetric interval X̄ ± zα/2·σ/√n. When σ² is unknown, the pivot T = (X̄ − μ)/(S/√n) follows a Student t distribution with n−1 degrees of freedom, giving the t‑interval X̄ ± tα/2,n−1·S/√n.
For non‑normal populations, large‑sample theory provides approximate pivots via the central limit theorem. The Wald interval uses the asymptotic normality of the MLE: θ̂ ± zα/2·√(I(θ̂)⁻¹). The score interval inverts the score test and often has better coverage for small samples, especially for binomial proportions. The likelihood‑ratio interval inverts the likelihood‑ratio test and is invariant under reparameterisation. Exact intervals exist for discrete distributions (Clopper–Pearson for binomial, Garwood for Poisson) but are conservative. Bootstrap percentile and bias‑corrected accelerated (BCa) intervals provide simulation‑based alternatives when analytic pivots are unavailable.
Interpretation is frequentist: in repeated sampling, a proportion 1−α of such intervals will contain the true parameter. The interval itself is random; the parameter is fixed. A common mistake is to state “there is a 95 % probability that μ lies in this interval”. The correct phrasing is “we are 95 % confident that the procedure that generated this interval produces intervals covering μ in 95 % of repeated samples”. Coverage probability can differ from nominal level for small samples or discrete parameters; checking actual coverage via simulation is good practice.
Hypothesis Testing: Neyman–Pearson Lemma, Likelihood Ratio Tests, and Power
Hypothesis testing formalises the decision between a null hypothesis H₀ and an alternative H₁. A test is a function φ(x) ∈ [0,1] giving the probability of rejecting H₀ when the data are x. The size (significance level) is supθ∈Θ₀ Eθ[φ(X)] ≤ α. The power function β(θ) = Eθ[φ(X)] for θ ∈ Θ₁ measures the probability of correctly rejecting H₀. The Neyman–Pearson lemma states that for simple hypotheses H₀: θ = θ₀ versus H₁: θ = θ₁, the most powerful test of size α rejects when the likelihood ratio L(θ₁)/L(θ₀) exceeds a constant k, with k chosen to achieve size α. This likelihood‑ratio test (LRT) is uniformly most powerful (UMP) among all tests of size α.
For composite hypotheses, a UMP test rarely exists. The generalised likelihood‑ratio test (GLRT) rejects for large values of λ = 2[ℓ(θ̂) − ℓ(θ̂₀)], where θ̂ is the unrestricted MLE and θ̂₀ is the MLE under H₀. Under regularity conditions, λ converges in distribution to χ² with degrees of freedom equal to the difference in parameter dimensions (Wilks’ theorem). This provides an asymptotic χ² test for nested models. The Wald test uses (θ̂ − θ₀)ᵀ I(θ̂)(θ̂ − θ₀) and the score test uses U(θ₀)ᵀ I(θ₀)⁻¹ U(θ₀); all three are asymptotically equivalent.
Power analysis guides sample‑size determination. For a one‑sided normal mean test with known variance, power = 1 − Φ(zα − δ√n/σ) where δ = μ₁ − μ₀. The non‑centrality parameter δ√n/σ quantifies the distance between hypotheses. In practice, one specifies α, desired power (often 0.8 or 0.9), and a minimally important effect size to solve for n. Multiple testing adjustments (Bonferroni, Holm, Benjamini–Hochberg) control family‑wise error rate or false discovery rate when many hypotheses are examined simultaneously.
Sufficiency, Completeness, and Ancillarity: Factorization Theorem and Basu’s Theorem
A statistic T(X) is sufficient for θ if the conditional distribution of X given T does not depend on θ. The Fisher–Neyman factorisation theorem gives a practical criterion: f(x;θ) = g(T(x);θ) h(x) for some functions g and h. Minimal sufficiency means T is a function of every other sufficient statistic; it can be found by examining the ratio f(x;θ)/f(y;θ) and identifying the equivalence classes where the ratio is constant in θ. For exponential families in canonical form, the natural statistic is minimal sufficient.
Completeness strengthens sufficiency: a statistic T is complete if Eθ[g(T)] = 0 for all θ implies g(T) = 0 almost surely. In a full‑rank exponential family, the natural statistic is complete. The Lehmann–Scheffé theorem then guarantees that any unbiased function of a complete sufficient statistic is the unique MVUE. Ancillarity is the opposite notion: a statistic A is ancillary if its distribution does not depend on θ. Basu’s theorem states that any boundedly complete sufficient statistic is independent of every ancillary statistic. This independence underpins the separation of inference about θ from inference about nuisance parameters and justifies conditioning on ancillary statistics to obtain exact tests (e.g., Fisher’s exact test for 2×2 tables conditions on the marginal totals, which are ancillary for the odds ratio).
Practical identification: for a normal sample with unknown mean μ and known variance, X̄ is complete sufficient; the sample variance S² is ancillary for μ. For a uniform distribution on (0,θ), the maximum X_(n) is sufficient but not complete; the range is ancillary. Recognising these structures allows rapid construction of MVUEs via Rao–Blackwellisation and Lehmann–Scheffé, and guides the choice of conditioning statistics for exact inference.
Bayesian Inference: Prior, Posterior, and Decision‑Theoretic Criteria
Bayesian inference treats the parameter θ as a random variable with a prior distribution π(θ) reflecting pre‑sample uncertainty. After observing data x, Bayes’ theorem updates the prior to the posterior π(θ|x) ∝ L(θ)π(θ). Conjugate priors yield posteriors in the same family: Beta–Binomial, Gamma–Poisson, Normal–Normal (known variance), Inverse‑Gamma–Normal (unknown variance). The posterior mean, median, or mode serve as point estimates; the posterior mean minimises posterior expected squared error loss, the median minimises absolute error loss, and the mode (MAP) minimises 0‑1 loss.
Credible intervals are direct probability statements: a 1−α credible set C satisfies ∫_C π(θ|x) dθ = 1−α. Highest posterior density (HPD) intervals have the shortest length for a given coverage. Unlike confidence intervals, credible intervals allow the interpretation “there is probability 1−α that θ lies in C”. Prior choice influences results, especially with small samples. Objective priors (Jeffreys prior π(θ) ∝ √I(θ), reference priors) aim to minimise prior information. Sensitivity analysis checks robustness to prior specification.
Decision theory formalises estimation and testing as minimising expected loss. For a loss function L(θ,δ), the Bayes rule δ*(x) minimises the posterior expected loss ∫ L(θ,δ) π(θ|x) dθ. Under squared error loss, δ* is the posterior mean; under absolute error loss, the posterior median. In hypothesis testing, the Bayes factor BF₁₀ = m₁(x)/m₀(x) compares marginal likelihoods under H₁ and H₀; posterior odds = prior odds × BF₁₀. Model selection uses posterior model probabilities or information criteria such as DIC, WAIC, or LOO‑CV that approximate out‑of‑sample predictive performance.
Sampling Theory: Design‑Based Estimation, Horvitz–Thompson Estimator, and Variance Estimation
Design‑based inference treats the population values as fixed and randomness arises solely from the sampling design. Let U = {1,…,N} be the finite population, y_i the study variable, and π_i = P(i ∈ s) the first‑order inclusion probability under a sampling design p(s). The Horvitz–Thompson (HT) estimator of the population total Y = ∑i∈U y_i is Ŷ_HT = ∑i∈s y_i/π_i. It is design‑unbiased: E_p[Ŷ_HT] = Y for any design with π_i > 0. The variance is Var_p(Ŷ_HT) = ∑i∈U∑j∈U (πij − π_iπ_j) (y_i/π_i)(y_j/π_j), where πij are second‑order inclusion probabilities.
For simple random sampling without replacement (SRSWOR) of size n, π_i = n/N, πij = n(n−1)/[N(N−1)], and the HT estimator reduces to the familiar expansion estimator Ŷ = (N/n)∑i∈s y_i with variance N²(1−f)S²/n, where f = n/N and S² is the population variance. For stratified sampling, the HT estimator applies within each stratum and the overall variance is the sum of stratum variances. Ratio and regression estimators improve efficiency by exploiting auxiliary variables x_i with known population totals X; the ratio estimator Ŷ_R = (∑i∈s y_i)/(∑i∈s x_i) · X is approximately unbiased and has smaller variance when y and x are highly correlated.
Variance estimation in complex designs uses linearisation (Taylor series), replication methods (balanced repeated replication, jackknife, bootstrap), or the generalised variance estimator (GVF). The Sen–Yates–Grundy form Var_p(Ŷ_HT) = −½ ∑i≠j (πij − π_iπ_j) (y_i/π_i − y_j/π_j)² is often preferred because it yields non‑negative estimates for many designs. Software packages (R survey, Stata svy) implement these estimators. Understanding the distinction between design‑based and model‑based (superpopulation) approaches is essential: design‑based properties hold for the realised sample under the known design, while model‑based properties rely on a stochastic model for the population values.
- Confusing confidence level with posterior probability — confidence intervals are frequentist, credible intervals are Bayesian.
- Assuming a UMP test exists for every composite alternative; often only GLRT or locally most powerful tests are available.
- Forgetting to verify the regularity conditions (differentiability, support independence) before applying the Cramér–Rao bound or Wilks’ theorem.
- Treating the Horvitz–Thompson estimator as model‑unbiased; it is design‑unbiased only under the sampling design.
- Using the normal approximation for a binomial proportion without checking np and n(1−p) ≥ 5, leading to poor coverage.
- Neglecting to condition on an ancillary statistic when an exact test is required, resulting in conservative or liberal size.
- Misapplying the delta method by omitting the derivative g'(μ) or using it when g is not differentiable at the parameter value.
These advanced topics complete the core inferential toolkit required for rigorous statistical analysis and examination success.
Asymptotic Theory and the Delta Method
Asymptotic theory provides the bridge between finite‑sample distributions and the limiting normal approximations that underlie most large‑sample inference. The three fundamental modes of convergence — almost sure, in probability, and in distribution — are ordered by strength: almost sure convergence implies convergence in probability, which in turn implies convergence in distribution. Slutsky’s theorem states that if a sequence of random vectors converges in distribution to a limit and another sequence converges in probability to a constant, then their sum, product, and quotient (where defined) converge in distribution to the corresponding operation on the limits. The Cramér–Wold device reduces multivariate convergence in distribution to univariate convergence of all linear combinations, a tool essential for proving multivariate central limit theorems.
The delta method extends the central limit theorem to smooth nonlinear functions of sample moments. Let √n (Tₙ − θ) converge in distribution to N(0, Σ) and let g be a function differentiable at θ with gradient ∇g(θ). Then √n (g(Tₙ) − g(θ)) converges in distribution to N(0, ∇g(θ)ᵀ Σ ∇g(θ)). In the multivariate case the same formula holds with the Jacobian matrix. A classic application is the variance‑stabilizing transformation for the sample correlation coefficient r: Fisher’s z = ½ log((1+r)/(1−r)) has asymptotic variance 1/(n−3), independent of the true correlation ρ. Another is the square‑root transformation for a Poisson mean, where √X̄ has approximate variance ¼. The delta method also yields the asymptotic distribution of the sample variance S², the sample coefficient of variation, and the odds ratio in 2×2 tables. Care is required when the derivative vanishes at the parameter value; then a second‑order delta method (involving the Hessian) gives a non‑normal chi‑square type limit.
Key Conditions and Extensions
- Differentiability of g at the true parameter; if g is not differentiable, the limit may be non‑normal (e.g., sample maximum).
- Lindeberg–Feller conditions for the underlying CLT; the delta method inherits these requirements.
- Multivariate delta method: Jacobian replaces gradient; covariance propagation uses matrix multiplication.
- Second‑order delta method: when ∇g(θ)=0, the limit is a quadratic form in normals, leading to chi‑square distributions.
Large Sample Properties of Maximum Likelihood Estimators
Under standard regularity conditions — identifiable parameter, interior true value, differentiable log‑likelihood, finite Fisher information, and dominance conditions for differentiation under the integral — the maximum likelihood estimator (MLE) θ̂ₙ enjoys three cornerstone properties. Consistency: θ̂ₙ converges in probability to the true θ₀. Asymptotic normality: √n (θ̂ₙ − θ₀) converges in distribution to N(0, I(θ₀)⁻¹), where I(θ₀) is the Fisher information matrix. Efficiency: the asymptotic covariance attains the Cramér–Rao lower bound, making the MLE asymptotically minimum variance among regular estimators. The proof uses a Taylor expansion of the score function around θ₀, the law of large numbers for the observed information, and the central limit theorem for the score.
The invariance principle states that if θ̂ₙ is the MLE of θ, then g(θ̂ₙ) is the MLE of g(θ) for any function g. This simplifies estimation of transformed parameters but does not guarantee optimal finite‑sample properties. When the model is misspecified, the MLE converges to the pseudo‑true value that minimizes Kullback–Leibler divergence; its asymptotic covariance becomes the sandwich estimator I(θ*)⁻¹ J(θ*) I(θ*)⁻¹, where J is the variance of the score. Quasi‑MLE (e.g., Poisson regression for over‑dispersed counts) exploits this robustness. The observed Fisher information (negative Hessian at θ̂ₙ) and expected Fisher information are both consistent estimators of I(θ₀); the former is often preferred in practice because it requires no integration. Wilks’ theorem extends the likelihood ratio statistic 2(log L(θ̂ₙ) − log L(θ₀)) to a χ² distribution with degrees of freedom equal to the number of restrictions, provided the null hypothesis lies in the interior of the parameter space.
Bootstrap Resampling and Its Variants
The bootstrap approximates the sampling distribution of a statistic by resampling from the empirical distribution (nonparametric bootstrap) or from a fitted parametric model (parametric bootstrap). For a statistic Tₙ = t(X₁,…,Xₙ), the nonparametric bootstrap draws B samples of size n with replacement from the original data, computes Tₙ* for each, and uses the empirical distribution of Tₙ* to estimate bias, variance, confidence intervals, and p‑values. The bootstrap is consistent for smooth functionals of the empirical distribution (e.g., sample mean, sample variance, M‑estimators) provided the underlying distribution has finite moments and the statistic is Hadamard differentiable. It fails for non‑smooth statistics such as the sample maximum, median in discrete distributions, or parameters on the boundary of the parameter space.
Several refinements improve coverage accuracy. The bootstrap‑t (percentile‑t) interval uses the bootstrap distribution of the studentized statistic (Tₙ*−Tₙ)/se* to construct equal‑tailed intervals with second‑order correctness. The bias‑corrected and accelerated (BCa) interval adjusts for both bias and skewness using the jackknife influence values; it achieves second‑order accuracy for a wide class of statistics. The parametric bootstrap is essential when the model structure is known (e.g., regression errors) but the error distribution is unspecified; one resamples residuals or generates new responses from the fitted model. The wild bootstrap handles heteroscedasticity by multiplying residuals by random weights with mean zero and variance one. For time series, the block bootstrap preserves dependence by resampling blocks of consecutive observations. Theoretical guarantees require the block length to grow with n but slower than n. In high‑dimensional settings, the bootstrap may be inconsistent for the maximum of many correlated statistics; multiplier bootstrap or Gaussian approximation techniques are then preferred.
Practical Guidelines
- Use at least B = 2000 bootstrap replications for variance estimation; B = 5000–10000 for percentile or BCa intervals.
- Check bootstrap histograms for multimodality or extreme skewness; these signal potential failure.
- For regression, the pairs bootstrap (resampling (X,Y) pairs) is design‑free but less efficient than the residual bootstrap when the model is correctly specified.
- Always report the bootstrap method (nonparametric, parametric, wild, block) and the number of replications.
Decision‑Theoretic Foundations: Minimax and Admissibility
Statistical decision theory formalizes inference as a game between nature (choosing a parameter θ ∈ Θ) and the statistician (choosing a decision rule δ). A loss function L(θ, a) quantifies the penalty for taking action a when the true state is θ. The risk function R(θ, δ) = E_θ[L(θ, δ(X))] is the expected loss. A rule δ₁ dominates δ₂ if R(θ, δ₁) ≤ R(θ, δ₂) for all θ with strict inequality for some θ; a dominated rule is inadmissible. An admissible rule is not dominated by any other rule. The Bayes risk of a rule with respect to a prior π is r(π, δ) = ∫ R(θ, δ) π(θ) dθ. A Bayes rule minimizes Bayes risk for a given prior; under mild conditions, every Bayes rule with a proper prior is admissible. The minimax rule minimizes the maximum risk sup_θ R(θ, δ). If a Bayes rule has constant risk (i.e., its risk does not depend on θ), it is minimax. The least favorable prior is a prior for which the Bayes rule is minimax; its existence is guaranteed in many settings by the von Neumann minimax theorem.
Stein’s paradox illustrates that the usual estimator X̄ for a multivariate normal mean (dimension ≥ 3) is inadmissible under squared error loss. The James‑Stein estimator δ_JS = (1 − (p−2)/‖X‖²) X dominates X̄, shrinking toward the origin. This result extends to empirical Bayes and hierarchical models. Complete class theorems state that the set of admissible rules is essentially the set of Bayes rules and limits of Bayes rules. In hypothesis testing, a test is admissible if no other test has uniformly smaller type‑I and type‑II error probabilities. The Karlin–Rubin theorem provides UMP tests for one‑parameter exponential families; for multiparameter problems, minimax tests often coincide with likelihood ratio tests calibrated to equalize maximum risk. Decision‑theoretic thinking also underpins model selection criteria: AIC estimates expected Kullback–Leibler loss, BIC approximates Bayes factors under unit‑information priors, and cross‑validation directly estimates predictive loss.
Advanced Survey Sampling: Calibration, Raking, and Nonresponse Adjustment
Calibration estimators improve design‑based efficiency by forcing weighted sample totals to match known population totals for auxiliary variables. Let w_i = 1/π_i be the base design weights. The calibrated weights w_i* = w_i (1 + q_i x_iᵀ λ) are chosen to satisfy Σi∈s w_i* x_i = X, where X is the known population total of the auxiliary vector x_i, q_i are known positive constants, and λ is a vector of Lagrange multipliers. The resulting generalized regression (GREG) estimator Ŷ_GREG = Σ w_i* y_i is approximately design‑unbiased and has variance close to that of the optimal regression estimator. When the auxiliary variables are categorical, calibration reduces to post‑stratification; when they are continuous, it yields ratio or regression estimators. Raking (iterative proportional fitting) calibrates simultaneously to several marginal totals by sequentially adjusting weights to match each margin; it converges to the maximum entropy weights satisfying all constraints. Raking is widely used in large‑scale surveys (e.g., the American Community Survey) to align sample demographics with census benchmarks.
Nonresponse introduces a missing‑data mechanism that can bias design‑based estimators if the response probability depends on the study variable. Weighting class adjustment partitions the sample into homogeneous response groups and inflates weights of respondents by the inverse of the estimated response rate within each class. Propensity‑score weighting models the response probability as a function of auxiliary variables (e.g., logistic regression) and uses the inverse predicted probability as an adjustment factor. Both methods assume missing at random (MAR) given the auxiliary variables. Doubly robust estimators combine outcome regression with propensity weighting; they remain consistent if either model is correct. Variance estimation for calibrated and nonresponse‑adjusted weights must account for the additional randomness from estimating λ and response propensities. Linearization (Taylor series) and replication methods (jackknife, bootstrap) extended to the calibrated weights are standard. The generalized variance function (GVF) approach models the variance of key estimates as a smooth function of the estimate itself, providing quick approximate standard errors for tabular output.
- Confusing the delta method’s first‑order approximation with exact finite‑sample distributions; the method only guarantees asymptotic normality.
- Assuming the MLE is always asymptotically normal without verifying regularity conditions such as parameter interiority and differentiability of the likelihood.
- Applying the nonparametric bootstrap to non‑smooth statistics (e.g., sample maximum, quantiles of discrete distributions) and expecting valid confidence intervals.
- Treating a Bayes rule with an improper prior as automatically admissible; impropriety can lead to inadmissible generalized Bayes rules.
- Using calibration weights without checking the positivity of calibrated weights; negative weights can produce erratic estimates and variance underestimation.
- Ignoring the MAR assumption in nonresponse adjustment and applying weighting class corrections when response depends on unobserved outcomes.
- Review these traps before the exam to avoid common pitfalls.