Coverage Test (Optional)

Published

Sunday Jun 8, 2025

Disclaimer: Taken from response by ChatGPT


1 βœ… What is Coverage Probability?

The coverage probability of a confidence interval method is:

The probability that the interval contains the true parameter value, over repeated samples.

For a \(95%\) confidence interval for a proportion \(p\), this means:

\[ \text{Coverage at } p = \mathbb{P}_p\left( \text{CI contains } p \right) \]

It answers: β€œIf the true proportion is p, how often does the method capture it?”

2 🎯 How Coverage is Computed (for Binomial CI)

We assume:

  • \(Y \sim \text{Binomial}(n, p)\)
  • Each possible outcome \(y = 0, 1, \dots, n\) has probability \(\binom{n}{y} p^y (1 - p)^{n - y}\)

We calculate coverage for each \(p\) as:

\[ \text{Coverage}(p) = \sum_{y=0}^{n} \mathbb{P}(Y = y) \cdot \mathbf{1}\left[ p \in \text{CI}(y) \right] \]

Where:

  • \(\mathbf{1}[p \in \text{CI}(y)] = 1\) if the interval contains \(p\), otherwise \(0\)
  • \(\text{CI}(y)\) is the confidence interval computed from \(y\)

3 In Code Terms

For each value of \(p\) in the grid:

  1. Loop through all \(y = 0, 1, \ldots, n\)
  2. For each \(y\):
    • Compute \(\text{CI}(y)\) using the chosen method (Wald, Wilson, etc.)
    • Check if \(p \in \text{CI}(y)\)
    • Weight that check by the binomial probability \(P(Y = y)\)
    1. Sum those weighted checks β†’ coverage probability at that \(p\)

4 Example (manually, say n = 3, p = 0.2)

Let’s say: - You compute CIs for each \(y \in \{0, 1, 2, 3\}\) - You get: - CI(0): \([0.00, 0.40]\) - CI(1): \([0.05, 0.60]\) - CI(2): \([0.30, 0.85]\) - CI(3): \([0.60, 1.00]\)

Then check:

  • Is 0.2 in CI(0)? βœ…
  • Is 0.2 in CI(1)? βœ…
  • Is 0.2 in CI(2)? ❌
  • Is 0.2 in CI(3)? ❌

Get binomial probabilities:

  • \(P(Y = 0) = (1 - 0.2)^3 = 0.512\)
  • \(P(Y = 1) = 3 \cdot 0.2 \cdot (0.8)^2 = 0.3844\)
  • \(P(Y = 2) = 0.096\)
  • \(P(Y = 3) = 0.008\)

Then: \[ \text{Coverage at } p = 0.2 = 0.512 + 0.384 = 0.896 \]

βΈ»

βœ… Why This Is Accurate

This is effectively a theoretical simulation: you compute the expected proportion of times the CI captures the true p using the binomial model β€” without sampling.

It reflects exact coverage (not approximate), assuming:

  • The distribution of data is exactly binomial
  • The CI method is applied correctly for every possible \(y\)