Statistical inference and resource allocation in expert data production

Project 1: Scalable Expert Quality Control and Adaptive Review
Author: Theodore Ouyang

Joint design of selective verification, residual-corrected estimation, and reviewer capacity

Review policies change the distribution of errors we observe. Directing more expert time toward difficult submissions can raise the error rate in the reviewed sample and assign harder work to stronger experts. Uncorrected quality statistics may therefore penalize an effective allocation policy. The technical question is how to improve delivered work while preserving quality evidence that can support the next round of decisions.

In this project, I co-developed operating procedures for assessing contributor reliability, assigning tiered review, conducting random expert audits, and updating evaluation criteria. The analysis below formalizes those production practices as a testable research design. The derivations specify the conditions under which the design is valid, and reproducible numerical experiments quantify the gains under those conditions.

1 The observation mechanism creates its own failure modes

Let $X$ denote task features and $Y\in\{0,1\}$ indicate an error in the original submission. Under fixed acceptance criteria, the conditional error probability is $q(x)=P(Y=1\mid X=x)$. Let $S=1$ indicate selection for an independent reference audit. Conditional on $X$, selection is randomized with probability $\pi(x)>0$, independently of the reference label that has yet to be observed.

Using the error fraction among verified submissions to assess the full population targets:

When review priorities identify high-risk tasks, the covariance is typically positive. A higher error detection rate in the reviewed sample cannot then be read directly as a deterioration in overall quality. This identity describes the distribution of a selected item. In a finite batch, the ratio of observed errors to selected items also has a random denominator, so the two quantities should not be conflated.

Expert rankings are affected by the same mechanism. Let $a_{jh}$ be the probability that expert $j$ is correct on task stratum $h$, and let $p_{jht}$ describe the mix of tasks assigned to that expert in period $t$. Even with unchanged ability, observed accuracy can change:

Comparisons therefore require a fixed target task mix $w_h$, reporting $A_j^{\mathrm{std}}=\sum_h w_ha_{jh}$, with overlapping assignments or shared anchor cases to support estimation within each stratum. A beta-binomial posterior can account for unequal sample sizes in comparable audits. Pooling different difficulty levels, criteria versions, and selection mechanisms into a single binomial sample leaves the preceding bias unresolved.

These distinctions also guide method selection. Dawid-Skene models represent differences in reviewer error patterns. DeCCaF already incorporates task conditions, error costs, and reviewer capacity into deferral, and discusses how assigning difficult cases can distort evaluations of expert performance. The design here addresses the interface between these methods and population quality inference in ongoing production. Dawid-Skene, DeCCaF

2 Correct predictions with audits to recover the population target

Fix a batch of $N$ original submissions, its acceptance criteria, and its reference labels $Y_1,\ldots,Y_N$. The target is the finite-batch error rate $\mu=N^{-1}\sum_iY_i$. Let $f_i\in[0,1]$ be a fixed, inexpensive risk prediction and $\pi_i\in[\pi_{\mathrm{min}},1]$ the audit probability, with independent draws $S_i\sim\mathrm{Bernoulli}(\pi_i)$. Both predictions and probabilities are set before the current batch's reference labels are revealed.

Use a prediction-assisted, residual-corrected estimator:

Its validity follows directly. Write $e_i=Y_i-f_i$ and condition on the fixed batch, predictions, and sampling probabilities:

Inverse-probability weighting restores representation altered by selective verification. Predictions reduce the residual that expert audits must resolve. This design-based unbiasedness does not require $f_i=q(X_i)$: inaccurate predictions can reduce efficiency, but they do not introduce the same selection bias under the specified sampling design. The estimator follows established active statistical inference methods. The project design specifies the object being audited, the criteria version, and the records needed for subsequent decisions. Active Statistical Inference: mean estimation and sampling design

Original and revised submissions must be retained separately. Estimating $\mu^{\mathrm{raw}}$ on originals and $\mu^{\mathrm{final}}$ on final submissions distinguishes defects generated in production from defects corrected by review. When criteria change, the same anchor cases can be adjudicated under both versions; submissions from different periods can then be compared under a common version. This prevents stricter standards from being mistaken for declining ability.

3 Allocate audit effort according to residuals and cost

Independent Bernoulli sampling eliminates cross-covariance terms, giving:

This expression identifies the allocation objective: audit effort should address what the predictions have not resolved. For a binary error label with conditional error probability $q_i$,

The two terms are conditional variance and squared prediction error. A high error probability does not necessarily imply high inferential value. If a category is almost certain to contain an error and the predictor correctly recognizes that fact, another audit may provide less information than one on a moderately risky task whose outcome is uncertain. The high-risk task may still merit correction, so the value of measurement and the value of remediation require separate models.

Let $c_i>0$ be the cost of an independent reference audit and $B$ the expected audit-cost budget. First consider the optimal design when the true residual second moments $m_i>0$ are known:

The omitted constant $\sum_i m_i$ does not depend on the allocation. For probabilities strictly between their lower and upper bounds, the Lagrangian first-order condition yields:

Here $\lambda>0$ is determined by the budget, and feasibility requires $B\ge\pi_{\mathrm{min}}\sum_i c_i$. If the budget covers a full audit, every item is audited. The square-root rule increases sampling for larger residuals while reducing the relative frequency of expensive audits. The positive probability floor preserves observation across the task space.

Implementation substitutes regularized estimates $\widehat m_i>0$ obtained from earlier independent audits or a pilot batch. True $m_i$ defines the optimality analysis; estimated values determine the allocation. Residuals can be estimated by task stratum, criteria version, and cost stratum, without fitting unstable parameters for individual submissions. A schedule that requires a fixed number of audits in every batch instead needs a fixed-size sampling design and its inclusion probabilities. The budget above is an expectation, not a hard capacity guarantee for each batch.

4 Derive the gain and identify when it disappears

Compare uniform audit probabilities with cost-aware allocation using the same predictor. Let $C=\sum_i c_i$, so that the uniform probability is $p=B/C$. If the true residual second moments are available and no optimal probability reaches a clipping boundary,

Here $V$ is design variance averaged over the conditional label distribution. If a common sampling probability is used within each fixed task stratum, substituting that stratum's true mean squared residual for $m_i$ also gives an exact finite-batch result. The Cauchy-Schwarz inequality implies:

Equality holds if and only if $m_i/c_i$ is constant across items. Gains therefore depend on identifiable differences in residual second moment per unit audit cost. With homogeneous residuals and costs, the optimal allocation reduces to uniform sampling. When clipping is active, constrained optimization can still be compared with a feasible uniform policy, but the gain must be calculated from the clipped solution's objective value.

The same analysis exposes the cost of misspecification. If the allocation uses inaccurate estimates $\widehat m_i$ and neither allocation is clipped, the excess variance relative to the true optimum is:

Severely underestimating $\widehat m_i$ in part of the task space increases the penalty through the denominator. A positive sampling floor limits the consequences of misspecification; independent calibration batches determine whether active allocation is worthwhile. Mixing active and uniform probabilities as $\pi_i^{(\rho)}=(1-\rho)\widehat\pi_i+\rho p$ preserves the same cost budget. The mixing coefficient should be selected through independent audit comparisons. Adding a fixed amount of random sampling does not itself guarantee an improvement over uniform sampling. Robust active statistical inference studies budget-preserving paths and optimization under residual misspecification. Robust Sampling for Active Statistical Inference

5 Allocate review by expected loss reduction

For task $i$ under action $a$, let $r_{ia}$ be the expected error loss after the action, $d_{ia}$ its cost, and $t_{iaj}$ the time required from expert $j$. Actions may include additional review, expert adjudication, obtaining further evidence, or revision. Binary choices $z_{ia}\in\{0,1\}$ satisfy:

Here $H_j$ is available reviewer time, and $\eta\ge0$ converts resource cost to the scale of error loss. DeCCaF provides a basis for cost- and capacity-constrained deferral. This design defines an action as the complete handling procedure and uses independent reference audits to evaluate residual risk after that procedure. DeCCaF: capacity constraints and global assignment

When review costs are equal and each task uses one slot, the appropriate ranking is by gain, $\Delta_i=r_{i0}-r_{i1}$. Missing evidence may make a high-risk task difficult to repair through one more review, while a clear calculation error in a lower-risk task may be readily corrected. For tasks $i,k$, moving the review slot from $i$ to $k$ changes total loss by exactly:

Whenever $\Delta_k>\Delta_i$, the exchange strictly reduces loss, regardless of the tasks' initial risk ranking. Under more complex capacity constraints, a batch optimizer can apply the same gain-based objective while excluding infeasible assignments, such as sending every difficult case to one expert.

The precision required of risk estimates can also be stated explicitly. Suppose $\lvert\widehat r_{ia}-r_{ia}\rvert\le\delta$ for every feasible action, costs and capacities are known, and $\widehat z$ exactly minimizes the predicted objective. Relative to the true optimal feasible assignment $z^*$,

This bound identifies post-action loss as the calibration target. An approximate optimizer adds its objective-error bound. If action losses cannot be estimated on comparable tasks, optimization cannot supply the missing evidence. Production validation should therefore prioritize randomized pilots among permitted actions, calibrating risk against task conditions, criteria versions, and outcomes of the complete procedure.

The two budgets can also enter a single batch-level planning problem. Let $\tau^2$ be the target variance of the quality estimate, $B_{\mathrm{total}}$ the total cost budget, and $V_{\widehat m}(\boldsymbol\pi)=N^{-2}\sum_i(1-\pi_i)\widehat m_i/\pi_i$ the planning variance computed from the fixed residual model. The joint objective is:

Here $H_j^{\mathrm{review}}$ is hard capacity allocated specifically to review actions; statistical audits are scheduled separately. This mixed-integer program with a convex audit constraint imposes a model-based precision target for subsequent decisions while seeking to reduce current defects. The audit target is the frozen original submission, so $m_i$ does not depend on the batch's revision actions. Estimating final-submission quality instead requires an action-dependent residual model. Audit cost remains an expectation, and the formulation makes no hard guarantee about realized audit hours in each batch. Fixed quotas require the corresponding sampling design. Independent audits must check residual calibration to establish whether the variance target is met. An infeasible program makes the conflict between quality requirements and available resources explicit.

6 Reproducible experiments at equal cost

The following synthetic experiments test the mechanisms; they do not report client outcomes. A fixed batch contains 10,000 submissions in strata of 6,000, 3,000, and 1,000 items, with 60, 300, and 350 errors, respectively. The overall error rate is 7.10%. Audit costs are 1, 1.5, and 3 units per item. Every policy has an expected audit-cost budget of 2,000 units and a minimum sampling probability of 3%.

Risk predictions for the three strata are 2%, 8%, and 30%. Their batch average is 6.60%, differing from the true error rate by 0.50 percentage points. Cost-aware allocation uses the true within-stratum mean squared residuals, representing ideal residual calibration. The misspecified policy reverses the residual ordering across strata. With predictions, budget, and original submissions held fixed, 200,000 independent sampling repetitions check the analytical variances.

Under ideal residual calibration, standard deviation falls by 9.33%. This measures greater precision in quality estimation, not a reduction in submission errors. For the high-risk-first policy, the uncorrected detection rate averages approximately 19.60%, while the corrected estimate averages approximately 7.1015%. For all five policies, Monte Carlo variances differ from their analytical values by less than 0.5%, confirming agreement between the sampling implementation and the estimator.

A separate, exactly calculable allocation experiment isolates remediation gains. Two groups contain 100 items each, with initial error risks of 0.40 and 0.15. Additional review reduces those risks to 0.30 and 0.01, respectively. Review costs are equal and capacity is 100 items. Ranking by initial risk leaves $100(0.30)+100(0.15)=45$ expected errors; ranking by expected loss reduction leaves $100(0.40)+100(0.01)=41$. Expected error loss falls by 8.89% at the same review volume. This example treats action risks as known to test the allocation mechanism; production gains also depend on the accuracy of estimated action risks.

With homogeneous residuals and costs, optimal audit probabilities become uniform. With equal remediation gains, exchanging review slots yields no improvement. These two zero-gain conditions, together with the misspecification experiment, determine when more complex policies merit adoption.

7 Establish whether the design improves production

Production trials should test quality inference and control of delivered work separately. Detecting more errors is not, by itself, evidence of better delivered quality.

First, use frozen batches with complete independent adjudication to compare uniform sampling, the existing high-risk review policy, cost-aware residual allocation, and calibrated mixtures. Keep the predictor fixed and include pilot audits in the budget. Compare bias, mean squared error, and interval coverage computed for the actual sampling design. Examine task strata and criteria versions separately to check that aggregate improvements do not conceal missed defects in particular groups.

Next, compare existing tiered rules with gain-based action allocation at equal total expert hours. Measure serious defects in final submissions and total cost per accepted submission. Cost includes routine review, random audits, adjudication, revision, and resources occupied while work waits. If shared reviewer capacity creates interference between tasks, randomize policies by batch or matched time window and estimate uncertainty at that level.

Adoption can then follow an operational criterion: reduce cost per accepted submission while remaining within a prespecified tolerance for serious defects, or exceed a prespecified minimum meaningful reduction in residual serious errors at equal total cost. Audit precision must remain within its required range. Ablation studies should distinguish gains from the inference method, gains from review actions, and gains from their combination.

The contributor records, tiered review procedures, and random audits I established in production provide an operational basis for this design. The further research contribution is to distinguish three quantities: a task's initial risk, the information value of an independent reference audit, and the loss that additional handling can remove. Each budget then has a defined optimization objective, and quality evidence remains interpretable after the review policy changes.