Coverage Test (Optional)

Disclaimer: Taken from response by ChatGPT

---

1 ✅ What is Coverage Probability?

The coverage probability of a confidence interval method is:

The probability that the interval contains the true parameter value, over repeated samples.

For a \(95%\) confidence interval for a proportion \(p\), this means:

\[ \text{Coverage at } p = \mathbb{P}_p\left( \text{CI contains } p \right) \]

It answers: “If the true proportion is p, how often does the method capture it?”

2 🎯 How Coverage is Computed (for Binomial CI)

We assume:

• \(Y \sim \text{Binomial}(n, p)\)
• Each possible outcome \(y = 0, 1, \dots, n\) has probability \(\binom{n}{y} p^y (1 - p)^{n - y}\)

We calculate coverage for each \(p\) as:

\[ \text{Coverage}(p) = \sum_{y=0}^{n} \mathbb{P}(Y = y) \cdot \mathbf{1}\left[ p \in \text{CI}(y) \right] \]

Where:

• \(\mathbf{1}[p \in \text{CI}(y)] = 1\) if the interval contains \(p\), otherwise \(0\)
• \(\text{CI}(y)\) is the confidence interval computed from \(y\)

3 In Code Terms

For each value of \(p\) in the grid:

1. Loop through all \(y = 0, 1, \ldots, n\)
2. For each \(y\):
   • Compute \(\text{CI}(y)\) using the chosen method (Wald, Wilson, etc.)
   • Check if \(p \in \text{CI}(y)\)
   • Weight that check by the binomial probability \(P(Y = y)\)
   3. Sum those weighted checks → coverage probability at that \(p\)

4 Example (manually, say n = 3, p = 0.2)

Let’s say: - You compute CIs for each \(y \in \{0, 1, 2, 3\}\) - You get: - CI(0): \([0.00, 0.40]\) - CI(1): \([0.05, 0.60]\) - CI(2): \([0.30, 0.85]\) - CI(3): \([0.60, 1.00]\)

Then check:

• Is 0.2 in CI(0)? ✅
• Is 0.2 in CI(1)? ✅
• Is 0.2 in CI(2)? ❌
• Is 0.2 in CI(3)? ❌

Get binomial probabilities:

• \(P(Y = 0) = (1 - 0.2)^3 = 0.512\)
• \(P(Y = 1) = 3 \cdot 0.2 \cdot (0.8)^2 = 0.3844\)
• \(P(Y = 2) = 0.096\)
• \(P(Y = 3) = 0.008\)

Then: \[ \text{Coverage at } p = 0.2 = 0.512 + 0.384 = 0.896 \]

⸻

✅ Why This Is Accurate

This is effectively a theoretical simulation: you compute the expected proportion of times the CI captures the true p using the binomial model — without sampling.

It reflects exact coverage (not approximate), assuming:

• The distribution of data is exactly binomial
• The CI method is applied correctly for every possible \(y\)