In standard linear regression, we assume the response variable $Y$ can range from $−\infty$ to $+\infty$. However, real-world data is often restricted, for example:

• Probabilities (Binary data) must be between 0 and 1.
• Counts (Poisson data) must be positive integers.

If we tried to predict these directly with a linear equation $X \beta$ (which is unbounded), we might predict a probability of -0.5 or a count of -10, which is physically impossible.

Components of GLM

A GLM consists of three key components:

1. Random Component: The probability distribution of the response variable, $Y$, must belong to the Exponential Family of Distributions.

2. Systematic Component (Linear Predictor): A linear combination of the predictor variables, $\eta=x^T \beta$.

3. Link Function: A function, $g(⋅)$, that maps the bounded expected value of the response, $\mu=E[Y]$, to the linear predictor: $g(\mu)=\eta$.

It is important to realize that each observation is drawn from a separate distribution which is determined by the predictor variables $X$. Even though the aim of the regression is to estimate the common parameters $\beta$, each observation has its individual $X$ variables which in turn define a separate distribution (sharing the same family, but with unique mean and variance parameters).

During model fitting, we view the observed data $Y$ as a random realization drawn from these distributions, while the model attempts to estimate the underlying vector of means $\mu$ that generated them.

The Exponential Family of Distributions

The Foundation

The Random Component of a GLM relies entirely on the Exponential Family of Distributions. This is a class of probability distributions (including Normal, Poisson, Binomial, Gamma, etc.) whose density/mass function can be written in the standardized, canonical form:

• $\theta$ (Canonical Parameter) relates to the mean $\mu$ of the distribution and represents the “location” of the distribution.
• $y \theta$ (Interaction) “rewards” the likelihood function. If both the observation $y$ and the parameter $\theta$ are large and positive, the exponent increases, maximizing the probability.
• $b(\theta)$ (Cumulant) is a convex function which balances $y \theta$ by not letting it increase to infinity during the optimization $\theta$.
• $a(\phi)$ (Dispersion) is a function of a dispersion parameter $\phi$ (often the variance, $\sigma^2$). It scales the distribution.
• $c(y, \phi)$ is a normalization term for the data structure itself (e.g., factorials or constants). It encodes properties which do not depend on $\theta$, and thus on the mean.

The Utility of the Exponential Family

The canonical form of Exponential Family yields a unified set of mathematical properties essential for efficient model fitting.

Mean-Variance Relationship

The mean ($\mu$) and the variance ($V$) of the distribution are directly defined by the cumulant function $b(\theta)$ and the dispersion parameter $\phi$.

• Mean: $E[Y]=b’(\theta)$
• Variance: $Var(Y)=b’’(\theta) \cdot a(\phi)$

Deriving the Mean

If $f(y; \theta)$ is a probability density, its integral over all possible values of $y$ is 1.

We can put both sides under the derivative with respect to $\theta$ and land with this equation:

By putting the derivative under the integral and by applying the chain rule we get this:

And then by distributing and rearranging:

The expression on the left is the expected value of $Y$. In the second term, $b’(\theta)$ is a constant (it doesn’t depend on $y$), so it pulls out. What’s left is simply the integral over the density function which equals 1.

And therefore,

Deriving the Variance

We start from our previous result: $\int f(y; \theta) \cdot \left(\frac{y}{a(\phi)} - \frac{‘b(\theta)}{a(\phi)}\right) dy = 0$. This expression is differentiated from both sides with respect to $\theta$ using the product rule ($uv’ + u’v$).

Then again, considering that the integral over the density function which equals 1 and by rearranging we get this:

The expression on the left is the variance of $Y$. Therefore,

Sufficient Statistics (Data Compression)

This is a critical property for Big Data. Because the data $y$ enters the probability formula only through the linear product $y \theta$, the likelihood can be factored.

Below you can find details on the maximum likelihood estimation but here it suffices to say that if the Canonical Link is used (where $\theta=X\beta$), the term becomes $y(X\beta)$. This means the sum $\sum y_{i} ​x_{ij}$​ is a sufficient statistic for $\beta$.

This means that the algorithm does not need to store millions of raw rows in memory; it can compress the entire dataset into a simple vector of sums to perform the optimization.

Canonical Link Function

The canonical link function $g_{\text{canonical}}​(\mu)$ is the specific link that maps the mean directly to the canonical parameter:

In this case $\theta$ is linearly modeled as $x^{T} \beta$. So essentially $\theta = \eta$.

Since $\mu = b’(\theta)$, the canonical link function $g_{\text{canonical}}​(\mu)$ is the inverse function of $b’(\theta)$.

Here is an overview of the common canonical link functions:

Other Important (Non-Canonical) Models

Sometimes, the distribution’s theoretical variance structure is deemed inappropriate for the data, but the mean structure is fine. This leads to commonly used extensions:

Maximum Likelihood Estimation (MLE) Simplification

Using the canonical form results in a concave log-likelihood function, which guarantees a unique global maximum. This makes the MLE process simpler and more robust. More to it the section below.

Maximul Likelihood Estimation

In GLMs, the model parameters $\beta$ (the coefficients) are estimated using Maximum Likelihood Estimation (MLE). The goal of MLE is to find the values of $\beta$ that make the observed data most probable.

The Likelihood and Log-Likelihood Functions

For a set of $n$ independent observations $Y=(y_1​,y_2​,…,y_n​)$, the Likelihood Function, $\mathcal{L}(\beta∣Y)$, is the product of the individual density/mass functions at each observation, $f(y_i​)$, where each $f(y_i​)$ is parameterized by a unique $\theta_i$​ that depends on $\beta$ via the link function:

For computational simplicity (turning a product into a sum) and numerical stability, we maximize the Log-Likelihood Function:

Expanding the density function with the canonical form of Exponential Family:

• The first term, $\sum_{i=1}^n \frac{y_i \theta_i - b(\theta_i)}{a(\phi_i)}$​, is the kernel of the log-likelihood and contains all the terms dependent on the parameters $\beta$ (since $\theta$​ is a function of $\beta$).
• The second term, $\sum_{i=1}^n c(y_, \phi_i)$, does not depend on the regression coefficients $\beta$, so it can be ignored when solving for the MLEs.

Therefore, the likelihood is proportionate to this construct:

The Maximization Process (Finding the Score)

To find the values of $\beta$ that maximize the log-likelihood, we take the partial derivative of $l(\beta)$ with respect to each $\beta_j$​ and set the resulting expression (the Score Function) equal to zero:

Using the chain rule, the score equation becomes:

From deriving the mean and the variance of $Y$ in the section above we know that $\mu = b’(\theta)$, therefore $\frac{\partial \mu}{\partial \theta} = b’’(\theta)$. We also know that $Var(Y) = b’’(\theta) \cdot a(\phi)$, thus $\frac{\partial \mu}{\partial \theta} = \frac{Var(Y)}{a(\phi)}$. Inverting this gives us the term we need: $\frac{\partial \theta}{\partial \mu} = \frac{a(\phi)}{Var(Y)}$.

By applying it to the equation above we arrive at the general score equation:

Intuitively, the variance appearing in the denominator acts as a weighting mechanism. The term $(y−\mu)$ is the error (residual), while the term $1/Var(Y)$ down-weights observations that are naturally noisy. In this way the model “tries harder” to fit observations with low variance and “forgets” about observations with high variance.

Simplification with the Canonical Link

We can expand a part of the chain rule described above as this:

Then if we use the Canonical Link ($g_c​(\mu)=\theta$), we have $\mu=b’(\theta)$ and $\theta=\eta$. Therefore the expression above simplifies as follows:

When putting this into the original score equation the terms $Var(Y)$ cancels out. Considering that $a(\phi)$ is constant we have this simpliefied form:

This removal of the complex non-linear variance term from the denominator is why canonical link models are so computationally robust and guaranteed to converge, and why we can utilize the sufficient statistic by aggregating the data.

Solving the Equation

The score equation has no closed-form solution (except for a few special cases like OLS and Poisson regression with a canonical link). Since the Score Equation is non-linear, we use an iterative algorithm called Fisher Scoring (or Newton-Raphson), which can be expressed as Iteratively Reweighted Least Squares (IRLS).

Recall the multivariate Newton’s update rule for a parameter vector $\beta$:

In the context of GLMs, we are maximizing the log-likelihood $l(\beta)$. Therefore the gradient $\Delta f$ is the Score Function $U(\beta)$, and the the hessian $Hf$ is the matrix of second derivatives. In practice, we often use the Expected Hessian (the negative Fisher Information Matrix), a variation known as Fisher Scoring.

Substituting these into the Newton formula, the update becomes:

Elaboration on Fisher Information

For a GLM, the Information Matrix is derived as the variance of the Score Function (the gradient of the log-likelihood). It takes the form of a “weighted sandwich” of your data:

where

• $N$ is the total number of observations.
• $x_i$ is the vector of predictors for observation $i$ (the $i$-th row of the design matrix $X$).
• $W_i$ is the Working Weight for observation i, defined as:

The weight $W_i$ is the mathematical result of the Fisher Information of a single observation. It accounts for two critical factors:

1. Precision $\frac{1}{Var(Y_i)}$: This term accounts for the mean-variance relationship of the chosen distribution. Observations with higher predicted variance are considered “noisier” and are down-weighted.

2. Sensitivity $\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2$: This derivative reflects the link function’s curvature. Observations where a small change in the linear predictor $\eta$ leads to a large change in the mean $\mu$ are more informative and receive higher weight.

Linearization for the Newton Update

Due to the non-linear link function, the variable $y$ which we want to predict lives in another dimension as the linear predictor $\eta = X \beta$. Therefore we set $z$ as a First-order [Taylor expansion(/calculus/taylor-series/)] of the link function $g(y)$ around the current mean $\mu$:

This produces the Working Dependent Variable. Intuitively it’s a current linear prediction $\eta$, adjusted by the current error $(y−\mu)$, which has been “stretched” or “shrunk” by the slope of the link function. By making approximation $y \approx z$ we transition from the outcome space of $y$ to a linear dimension, and the error $(y−\mu)$ is translated into the same space.

Now let’s look again at the score equation. We can force $W$ into it by multiplying and dividing by $\frac{\partial \mu_i}{\partial \eta_i}$:

where $e$ is the vector of scaled errors. From the expression of the Working Dependent Variable $e$ is equivalent to $(z - \eta)$.

Putting it all together

By plugging the information matrix, and the linearized score function into the newton update we get this equation:

Then when we redistribute $\left(X^T W X \right)^{-1} X^T W$ across $(z - \eta)$, the part involving $\eta$ (which is $X \beta_{t-1}$) becomes negative $\beta_{t-1}$. Which simplifies to this:

The Algorithm Steps

1. Initialize: Start with an initial guess for $\beta$.

2. Calculate $\mu$ and $W$: Use the current $\beta$ to calculate the predicted means and the weights based on the chosen distribution.

3. Construct $z$: Form the working dependent variable using the residuals and the link function derivative.

4. Solve WLS: Calculate the new $\beta$ using the WLS formula.

5. Repeat: Iterate until the change in $\beta$ is smaller than a pre-defined threshold (convergence).

Standard Error of Coefficients

In statistics, the variance of an estimator (like a regression coefficient) is inversely related to the amount of information the data contains about that parameter. This relationship is tied to the shape of the likelihood function: the more sharply peaked the likelihood function is at its maximum, the more “certain” we are of the estimate. A sharper peak corresponds to a smaller variance and, consequently, a smaller standard error.

The matrix that mathematically quantifies this information—and specifically the curvature of the log-likelihood—is called the Fisher Information Matrix, $I(\beta)$. The estimated variance-covariance matrix for the coefficients, $V(\hat \beta)$, is obtained by inverting the Information Matrix at the final Maximum Likelihood estimates:

The diagonal elements of $V(\hat \beta)$, provide the variances of individual coefficients, and their square roots are the Standard Errors you see in model output.

Generalized Estimating Equations (GEE)

Standard Generalized Linear Models (GLMs) have a crucial assumption:

• Independence: All observations ($Y_i$​) are independent of each other, conditional on the covariates.

This assumption is violated when data is clustered (grouped) or longitudinal (repeated measures over time on the same subject). An example of clustered data is student test scores clustered within a classroom. Students in the same classroom are likely more similar due to the same teacher, curriculum, and environment.

In these cases, fitting a standard GLM (like logistic or Poisson regression) is problematic because the standard errors of coefficients (and thus p-values and confidence intervals) will be incorrectly small, leading to an inflated sense of precision and potentially false conclusions of statistical significance.

Generalized Estimating Equations (GEE) provide a robust, non-likelihood-based approach to analyzing clustered or longitudinal data where observations within groups are correlated. It is an extension of the GLM framework, but with a critical focus on getting the standard errors correct.

How GEE works

Suppose you have 10 different classrooms, each with 30 students. You want to determine if spending more time on homework (your covariate) leads to better test scores.

1. Initial Estimate: GEE begins exactly like a standard GLM by calculating the initial coefficients $\beta^{(0)}$. For example, it might estimate that for every extra hour of homework, scores increase by 5 points. This estimate is usually unbiased even if the internal correlation is initially ignored.

2. Define Correlation Structure: You must define a “working correlation structure” ($R$) that describes how students within the same classroom are related.

A common choice is Exchangeable (Compound Symmetry), which assumes every pair of students in a class shares the same correlation level, $\alpha$. At this stage, you only define the shape of the matrix; the actual numerical value of $\alpha$ is estimated later from the data.

The following steps are then repeated until the coefficients converge:

Note, each cluster is processed separately.

1. Calculate Means and Variances ($A_i$​): Using the current coefficient estimate $\beta^{(t−1)}$, we calculate the cluster mean vector $\mu_i$​ and the variance matrix $A_i$​.

$A_i$ is a diagonal matrix where each entry is the variance of a single observation based on the distribution’s mean-variance relationship (e.g., for Poisson, the variance is simply $μ_{ij}$​).

1. Calculate and Standardize Residuals: We determine the raw residuals $S_i$​ and then “standardize” them by dividing the error by the predicted standard deviation (from matrix $A$):

1. Update Global Parameters ($\alpha$ and $\phi$): These are global parameters shared across all clusters. While each classroom has its own data, the “rules of the game”—the noise level ($\phi$) and the correlation strength ($\alpha$)—are estimated as single values for the entire dataset.

$\phi$ accounts for overdispersion (when data is noisier than theoretically expected). It is calculated from the average of the squared Pearson residuals:

here

• $N$ is the total number of observations.
• $p$ is number of regression coefficients ($\beta$).

If $\phi$ is 1, the data matches the theoretical variance perfectly. If it is equal 2, the data is twice as noisy as a standard Poisson/Binomial distribution would expect. GEE uses this to “inflate” the standard errors so you don’t overstate your confidence.

$\alpha$ is estimated by looking at the “co-movement” of residuals within the same cluster. For the Exchangeable structure (where every pair has the same correlation), we sum the products of residuals for all possible pairs within each cluster:

If student $j$ and student $k$ consistently both have “higher than expected” scores (positive $r$ $\times$ positive $r$) or “lower than expected” scores (negative $r$ $\times$ negative $r$), the product is positive, and $$alpha$ goes up. If their residuals are totally unrelated, the positive and negative products cancel out, and $\alpha$ stays near zero.

1. Construct Cluster Covariance ($V_i$​): We combine the theoretical variance ($A_i$​), the correlation structure ($R_i$​), and the dispersion ($\phi$) into a single covariance matrix for the cluster:

• $A_i^{1/2}$ is diagonal matrix where the j-th diagonal entry is the square root of the GLM variance

$V_i^{t}$​ represents the total estimated variance of the response vector $Y_i$​, accounting for both the distribution type and the clustering effect.

1. Update Coefficients ($\beta$): Finally, we solve the Generalized Estimating Equation to find the new estimate $β^{(t)}$:

Here $D_i$ is the derivative matrix $\frac{\partial \mu_i}{\partial \beta}$. It acts like the design matrix X but is “adjusted” for the link function. (In the standard GLM section, this derivative was included within the weight matrix $W$, allowing for the use of the “clean” $X$ matrix instead).

This non-linear equation is solved iteratively (via Newton-Raphson or Fisher Scoring), approximating the solution as:

Robust Estimation of Parameters Covariance

Once the $\hat \beta$​ has converged, the final step is to calculate the robust standard errors, which are necessary for hypothesis testing and confidence intervals.

The Robust (Sandwich) Covariance Matrix for $\hat \beta$​ is calculated as:

The standard errors for the coefficients are the square roots of the diagonal elements of this matrix.

$B$ (The Bread, or Model-Based Part)

This is the term calculated from the final iteration of the fitting process. It represents the Model-Based (or Naive) covariance estimate.

This is exactly the same “Information Matrix” we discussed for standard GLMs ($X^{T}WX$). In GEE, it represents what the variance of your coefficients should be if your “guess” about the correlation ($V_i$​) was 100% correct. It’s called the “Bread” because it surrounds the center. In a perfect world where your model is perfect, the variance would just be $B^{−1}$.

$C$ (The Meat, or Empirical Part)

This matrix captures the observed variability between the clusters, making no assumptions about the internal correlation structure. It is the sum of the outer products of the weighted residuals from each cluster.

$S_i S_i^T$: This is the “outer product” of the residuals. While $V_i$​ is what your model thinks the covariance is, $S_i S_i^T$ is the actual, raw covariance observed in the data for that cluster.

For example, if students in Classroom 5 are much more similar to each other than your model predicted, $S_i S_i^T$ will be much larger than $V_i$​.

If S_i S_i^T$ is equal to $V_i$ then the “meat” simplifies and becomes equal to $B$. Consequently, the robust estimator is simply $B^{-1}$, same as the standard GLM variance.

The beauty of the Sandwich Estimator is its consistency. Even if the “working” correlation structure $V_i$​ is specified incorrectly, the Meat ($C$) uses the actual observed residuals to correct the Model-Based Bread ($B$). As long as your sample size of clusters ($K$) is sufficiently large, the resulting standard errors will be asymptotically correct.

Essentially, Fisher Information measures how sensitive the likelihood function is to small changes in the parameter $\theta$. A high Fisher Information value indicates that small changes in the parameter result in large changes in the likelihood (the likelihood function is sharply peaked), meaning the data provides a strong signal about the true value of $\theta$, leading to more precise estimators.

The Derivation of Fisher Information

The definition of Fisher Information arises directly from the Maximum Likelihood Estimation (MLE) process. We consider a random variable $X$ drawn from a distribution $f(X;\theta)$, where $\theta$ is the parameter of interest.

The Maximum Likelihood Estimator ($\theta_{MLE}$​) is found by maximizing the log-likelihood function $l(\theta)$, which means setting the first derivative to zero.

The first derivative of the log-likelihood function with respect to $\theta$ is called the score function, $S(\theta)$:

The expected value of the score function is a key property: for a well-behaved density function, the expected score is zero.

This means that on average, the score function is centered around zero when evaluated at the true parameter value $\theta$.

Fisher Information is defined as the variance of the score function. Since the expected value of the score is zero, its variance is simply the expected value of the score squared (from the expression of variance via the mean as $E[Y^2]−(E[Y])^2$):

Intuitively, high variance of the score function indicates high certainty with regards to the direction in which $\theta$ should be nudged in order to reach 0 (and the maximum of the likelihood function).

Alternative Definition via the Second Derivative

Under regularity conditions (specifically, that the order of integration and differentiation can be interchanged), a more computationally convenient definition of Fisher Information involves the second derivative of the log-likelihood function.

This second definition is often used in practice because the second derivative measures the curvature of the log-likelihood function. A high (negative) curvature means a sharp peak, which corresponds to high information.

While these two definitions are mathematically equivalent in expectation, the second derivative version highlights why the Information Matrix represents the curvature of our knowledge, while the squared score version highlights why it represents the precision (inverse variance) of our estimate.

Key Applications of Fisher Information

Because a high Fisher Information value indicates high certainty in the parameter estimate, it implies that the estimator has low variance. In this sense, Fisher Information and variance are inversely related: high information corresponds to low uncertainty. In practice, we invert the Fisher Information Matrix to obtain the Covariance Matrix, which provides a direct measure of the estimator’s uncertainty.

Asymptotic Variance of the MLE

This relationship is formalized by the Asymptotic Normality of the MLE. According to the Central Limit Theorem, as the sample size $n$ grows large, the Maximum Likelihood Estimator converges in distribution to a normal distribution centered at the true parameter $\theta$, with a variance equal to the inverse Fisher Information:

Since we can calculate the Fisher Information analytically, we can use its inverse as a readily available estimate of the variance of the MLE, which is crucial for constructing confidence intervals and performing hypothesis tests.

The Cramér-Rao Lower Bound (CRLB)

The most important application of Fisher Information is defining the Cramér-Rao Lower Bound. The CRLB states that for any unbiased estimator $\hat \theta$ of $\theta$, the variance of that estimator must be greater than or equal to the inverse of the Fisher Information:

The CRLB sets a theoretical minimum variance. No matter how clever an unbiased estimator you design, its precision can never exceed this limit. Estimators that achieve this minimum bound are called efficient estimators.

Optimal Experimental Design

Fisher Information is used to design experiments or observational studies that maximize the precision of parameter estimates.

In a design problem, the Fisher Information often depends on the design variables (e.g., sample sizes, spacing of observation points). By choosing the design that maximizes $I(\theta)$, researchers ensure they collect data that is maximally informative about the parameters of interest, thereby minimizing the cost and effort required to achieve a target level of statistical precision.

Statistical Tests

The Fisher Information matrix (the multidimensional generalization for multiple parameters) is integral to three major hypothesis tests:

• Wald Test: Uses the estimated inverse Fisher Information matrix to calculate the standard error of the parameter estimates.

• Score Test: Uses the score function and the Fisher Information evaluated under the null hypothesis to test the significance of a parameter.

• Likelihood Ratio Test: Although it doesn’t explicitly use the Fisher Information in its calculation, its asymptotic properties and derivation are closely related to the concepts that lead to Fisher Information.

The Delta Method is a powerful analytical tool in statistics used to find the approximate variance (or standard error) of a complicated, non-linear function of an estimator.

Imagine you have a reliable statistical estimate, $\hat \theta$, of a true population parameter, $\theta$. You also know the variance of this estimate, $Var(\hat \theta)$.

The Problem: Now, suppose your real interest lies in a transformed version of $\hat \theta$, represented by a non-linear function $g(\hat \theta)$ (e.g., $g(\hat \theta)=\log(\hat \theta)$ or $g(\hat \theta)=\theta^2$). Calculating the exact variance, $Var(g(\hat \theta))$, is often analytically impossible because non-linear functions warp distributions.

The Solution: The Delta Method’s solution is simple: It replaces the complicated non-linear function with a simple linear one by applying first-order Taylor Approximation.

Step 1: The Intuitive Approach (Taylor Approximation)

Recall that a Taylor Series approximates a complex function near a known point with a tangent line. In our case, we approximate our transformed estimator $g(\hat \theta)$ near the true value $\theta$:

In this approximation:

• $g(\theta)$ is the value of the function at the true (but unknown) parameter $\theta$.
• $g’(\theta)$ is the slope (the first derivative) of the function $g$ evaluated at the true value.
• $(\hat \theta − \theta)$ is the error of our estimate.

If we subtract $g(\theta)$ from both sides, we get an equation that links the error of the output directly to the error of the input:

This equation reveals the core mechanism: the error in our new metric is simply the original error multiplied by the slope $g’(\theta)$.

Since variance is just the average squared error, we can easily apply the standard rules for the variance of a linear transformation: $Var(aX+b)=a^{2}Var(X)$.

• $g(\theta)$ is a constant, so its variance is 0.
• $g’(\theta)$ is a fixed slope (a constant multiplier).
• $\hat \theta−\theta$ has the same variance as $\hat \theta$ (since subtracting a constant doesn’t change variance).

Applying the variance rules to the approximation yields the Delta Method formula for the approximate variance:

The approximate variance of the transformed statistic is the original variance, scaled by the square of the function’s derivative (slope) evaluated at the true value.

• If the absolute value of the slope is small $(∣g′(\theta)∣<1)$, the transformation compresses the variability.
• If the slope is large $(∣g′(\theta)∣>1)$, the transformation stretches the variability.

Step 2: The Formal Approach (Asymptotic Theory)

The derivation above is intuitive, but to use this for Hypothesis Tests or Confidence Intervals, we need to prove that the resulting distribution is actually Normal. This requires Asymptotic Theory (what happens as sample size $n$ approaches infinity).

The Problem of Collapsing Variance

By the Law of Large Numbers, our estimator $\hat \theta_n$​ gets closer to $\theta$ as $n$ increases. Consequently, the error $(\hat \theta_{n}−\theta)$ shrinks to zero. We can see this clearly with the sample mean $\bar X$. Its variance is:

As $n \rightarrow \infty$, the distribution collapses into a single point. A single point has no “shape,” so we cannot calculate probabilities or confidence intervals.

The Fix: Magnifying the Error

To study the shape of the distribution before it vanishes, we put it under a “microscope.” We scale the error up by $\sqrt{n}$​.

• The variance shrinks by $\frac{1}{n}$​.
• Multiplying the error by $\sqrt{n}$​ scales the variance by $(\sqrt{n})^2 = n$.
• This perfectly cancels the shrinkage, stabilizing the distribution so it converges to a non-degenerate Normal distribution.

The Formal Statement

We assume our original estimator satisfies the Central Limit Theorem. This theorem describes the distribution of the magnified error:

The Delta Method Theorem simply applies our linear approximation from Step 1 to this magnified error:

This confirms that the transformed error also follows a Normal distribution, with the variance scaled by the slope squared.

In practice we don’t know the true value of $\theta$ but we do have a consistent estimator $\hat \theta$, and as the sample grows it coverges to $\theta$. Therefore, we can plug $\hat \theta$ in place of $\theta$ when we calculate the derivative.

Why is It Useful

Once we have established the asymptotic normality and found the approximate variance of the transformed statistic, we can move from theoretical approximation to practical inference:

• Calculate the Standard Error (SE): The SE is simply the square root of the final approximate variance.
• Construct Confidence Intervals: Using the property of asymptotic normality, we can build large-sample confidence intervals for the transformed parameter $g(θ)$.
• Perform Hypothesis Tests: The approximate normal distribution allows us to calculate z-scores and p-values for transformed quantities.

By identifying the distribution that best fits our data, we unlock powerful insights, make informed predictions, and choose the right statistical tests for our analysis.

Discrete vs. Continuous Distributions

Probability distributions are broadly categorized based on the type of data they model.

Discrete Distributions: These deal with variables that have a finite or countable number of outcomes. The distribution is described by a Probability Mass Function (PMF), which assigns a specific probability value to each individual outcome. For example, the number of heads in a series of coin tosses.

Continuous Distributions: These are for variables that can take any value within a given range. The distribution is described by a Probability Density Function (PDF), which gives the relative likelihood of a variable taking a certain value. While the probability of a continuous variable assuming an exact value is zero, the PDF allows us to calculate the probability of the variable falling within a specific range using integral calculus.

Common Discrete Distributions

These distributions are used to model count data and other discrete random variables.

Bernoulli Distribution

This is the simplest of all distributions, representing a single trial with two outcomes (e.g., 0 or 1). It’s essentially a binomial distribution with a single trial ($n$=1). It models events like a single coin flip.