7.5 Negative Binomial Regression | A Guide on Data Analysis

Var ( Y i ) = E ( Y i ) = μ i Var ( 𝑌 𝑖 ) = 𝐸 ( 𝑌 𝑖 ) = 𝜇 𝑖 However, in many real-world datasets, the variance exceeds the mean—a phenomenon known as overdispersion. When overdispersion is present, the Poisson model underestimates the variance, leading to: Inflated test statistics (small p-values). Overconfident predictions. Poor model fit. To address overdispersion, Negative Binomial regression introduces an extra dispersion parameter θ 𝜃 to allow variance to be greater than the mean: Var ( Y i ) = μ i + θ μ 2 i Var ( 𝑌 𝑖 ) = 𝜇 𝑖 + 𝜃 𝜇 𝑖 2 where: μ i = exp ( x ′ i θ ) 𝜇 𝑖 = exp ⁡ ( 𝑥 𝑖 ′ 𝜃 ) is the expected count. θ 𝜃 is the dispersion parameter. When θ → 0 𝜃 → 0 , the NB model reduces to the Poisson model. Thus, Negative Binomial regression is a generalization of Poisson regression that accounts for overdispersion. We apply Negative Binomial regression to the bioChemists dataset to model the number of research articles (Num_Article) as a function of several

Var ( Y i ) = E ( Y i ) = μ i Var ( 𝑌 𝑖 ) = 𝐸 ( 𝑌 𝑖 ) = 𝜇 𝑖 However, in many real-world datasets, the variance exceeds the mean—a phenomenon known as overdispersion. When overdispersion is present, the Poisson model underestimates the variance, leading to: Inflated test statistics (small p-values). Overconfident predictions. Poor model fit. To address overdispersion, Negative Binomial regression introduces an extra dispersion parameter θ 𝜃 to allow variance to be greater than the mean: Var ( Y i ) = μ i + θ μ 2 i Var ( 𝑌 𝑖 ) = 𝜇 𝑖 + 𝜃 𝜇 𝑖 2 where: μ i = exp ( x ′ i θ ) 𝜇 𝑖 = e