Audit sample size calculator
To estimate a share (resolved, held, matched) within ±5 points at 95% confidence, you need to label at most 385 records, and fewer when the share is far from 50% or the population is small. With the defaults below (synthetic): an expected share of 50% and a ±5-point margin need 385 labels, or 282 if only 1,050 claims are left to audit.
How many to label?
Label 282 records to estimate the share within ±5.0 points at 95% confidence.
How wide is my interval?
80.0% with a 95% interval of 73.9% to 85.0%.
How it is calculated
records to label, for an expected share p, a margin E and confidence z: n0 = z² × p × (1 − p) ÷ E² n = n0 ÷ (1 + (n0 − 1) ÷ N) when N claims are left to audit answer = the next whole number, at most N the interval for k found of n labelled (Wilson): centre = (p + z²/2n) ÷ (1 + z²/n) half-width = z × √(p(1 − p)/n + z²/4n²) ÷ (1 + z²/n)
Worked examples
- Label 385 records to estimate the share within ±5.0 points at 95% confidence. (Expected share 50.0%.)
- Label 282 records to estimate the share within ±5.0 points at 95% confidence. (Expected share 50.0%, 1,050 claims left.)
- Label 246 records to estimate the share within ±5.0 points at 95% confidence. (Expected share 80.0%.)
- Label 1,068 records to estimate the share within ±3.0 points at 95% confidence. (Expected share 50.0%.)
- 160 found of 200 labelled: 80.0% with a 95% interval of 73.9% to 85.0%.
- 100 found of 200 labelled: 50.0% with a 95% interval of 43.1% to 56.9%.
- 44 found of 50 labelled: 88.0% with a 95% interval of 76.2% to 94.4%.
Questions
Why does Attify's judge read 200 per company?
- 200 gives a 95% interval of about ±7 points in the worst case (a share near 50%), and narrower nearer 0% or 100%. It balances the cost of judging against the precision of the rate.
What do 50 human labels show about the judge?
- Attify validates the judge for a standard version once at least 50 human labels agree with it 85% of the time or more on each judged decision, and each published score has its own human audit of 50 judged records (all of them when fewer). 50 labels measure agreement only roughly: 44 agreeing of 50 (88.0%) has a 95% interval of 76.2% to 94.4%, so passing the bar does not show that the judge's true agreement is 85% or more.