What is the design effect?

Individuals in the same cluster tend to respond alike, so according to the Cochrane Handbook their observations can no longer be treated as independent; it describes the intracluster correlation coefficient (ICC) as the ‘similarity’ of individuals within a cluster. Kerry and Bland (1998) give it as the variance of true cluster means divided by that variance plus the within-cluster variance: ρ = σc² / (σc² + σw²).

The United Nations Statistics Division (2005) credits Kish (1965) with defining the design effect as the ratio of an estimate’s variance under a complex design to its variance under simple random sampling of the same size. According to Kerry and Bland, when m individuals are taken from each cluster it equals 1 + (m − 1)ρ, and with one observation per cluster (m = 1) it is 1. Even a small ICC matters when clusters are large: for m = 50 and ρ = 0.019, 1 + 49 × 0.019 = 1.931, almost twice the sample.

The United Nations Statistics Division gives the effective sample size as n / deff. According to the Cochrane Handbook, analysing clustered data as if participants were independent makes confidence intervals too narrow and P values too small, which can produce false-positive conclusions. Naing et al. (2006) say the design effect must be applied as a multiplier to a sample size calculated for simple random sampling.