Coverage Test (Optional)
Disclaimer: Taken from response by ChatGPT
1 β What is Coverage Probability?
The coverage probability of a confidence interval method is:
The probability that the interval contains the true parameter value, over repeated samples.
For a \(95%\) confidence interval for a proportion \(p\), this means:
\[ \text{Coverage at } p = \mathbb{P}_p\left( \text{CI contains } p \right) \]
It answers: βIf the true proportion is p, how often does the method capture it?β
2 π― How Coverage is Computed (for Binomial CI)
We assume:
- \(Y \sim \text{Binomial}(n, p)\)
- Each possible outcome \(y = 0, 1, \dots, n\) has probability \(\binom{n}{y} p^y (1 - p)^{n - y}\)
We calculate coverage for each \(p\) as:
\[ \text{Coverage}(p) = \sum_{y=0}^{n} \mathbb{P}(Y = y) \cdot \mathbf{1}\left[ p \in \text{CI}(y) \right] \]
Where:
- \(\mathbf{1}[p \in \text{CI}(y)] = 1\) if the interval contains \(p\), otherwise \(0\)
- \(\text{CI}(y)\) is the confidence interval computed from \(y\)
3 In Code Terms
For each value of \(p\) in the grid:
- Loop through all \(y = 0, 1, \ldots, n\)
- For each \(y\):
- Compute \(\text{CI}(y)\) using the chosen method (Wald, Wilson, etc.)
- Check if \(p \in \text{CI}(y)\)
- Weight that check by the binomial probability \(P(Y = y)\)
- Sum those weighted checks β coverage probability at that \(p\)
4 Example (manually, say n = 3, p = 0.2)
Letβs say: - You compute CIs for each \(y \in \{0, 1, 2, 3\}\) - You get: - CI(0): \([0.00, 0.40]\) - CI(1): \([0.05, 0.60]\) - CI(2): \([0.30, 0.85]\) - CI(3): \([0.60, 1.00]\)
Then check:
- Is 0.2 in CI(0)? β
- Is 0.2 in CI(1)? β
- Is 0.2 in CI(2)? β
- Is 0.2 in CI(3)? β
Get binomial probabilities:
- \(P(Y = 0) = (1 - 0.2)^3 = 0.512\)
- \(P(Y = 1) = 3 \cdot 0.2 \cdot (0.8)^2 = 0.3844\)
- \(P(Y = 2) = 0.096\)
- \(P(Y = 3) = 0.008\)
Then: \[ \text{Coverage at } p = 0.2 = 0.512 + 0.384 = 0.896 \]
βΈ»
β Why This Is Accurate
This is effectively a theoretical simulation: you compute the expected proportion of times the CI captures the true p using the binomial model β without sampling.
It reflects exact coverage (not approximate), assuming:
- The distribution of data is exactly binomial
- The CI method is applied correctly for every possible \(y\)