Large-Scale Domain Expert Feedback Governance and Adaptive Review

I. How selective review changes quality measurement

In a production RLHF pipeline, whatever policy routes submissions to review also changes the statistical properties of the observed sample. To see how, let $X$ denote observable task features, including the domain, prompt complexity and contributor history. Let $Y \in \{0, 1\}$ indicate whether the original submission contains an error under fixed acceptance criteria, so that the true conditional error probability is $q(x) = P(Y = 1 \mid X = x)$. Write $S = 1$ when a submission is selected for an independent reference audit. Conditional on $X$, selection is randomized with probability $\pi(x) > 0$, and it is independent of the reference label, which has not yet been observed.

Suppose an operational dashboard estimates population quality from the error fraction among verified submissions. What it actually targets is:

When review does succeed in targeting higher true error risk, the covariance between audit probability $\pi(X)$ and true defect probability $q(X)$ is strictly positive, and the dashboard reads high. A higher error detection rate in the reviewed sample can therefore reflect more effective risk targeting. By itself it does not establish that production quality has deteriorated. These identities describe the distribution of a selected item. In a finite batch the observed error fraction also has a random denominator, so it has to be kept distinct from the population expression above.

Expert rankings suffer from the same selection mechanism. Let $a_{jh}$ be the probability that expert $j$ answers a task in stratum $h$ correctly, and let $p_{jht}$ describe that expert's assigned task mix in period $t$. Even if underlying ability stays constant, observed accuracy moves with the task mix:

Comparisons across periods therefore require a fixed target task mix $w_h$ and standardized accuracy:

Overlapping assignments or shared anchor cases support estimation within each stratum. Pooling tasks of different difficulty, rubric versions and selection mechanisms into a single unstratified binomial sample leaves the selection bias unresolved. In the project's production routing engine I used a conjugate Beta-Binomial model for each contributor's reliability within a task stratum. Given a prior $\operatorname{Beta}(\alpha_{0h}, \beta_{0h})$ and $n_{jht}$ audited tasks with $k_{jht}$ verified successes, the posterior $\operatorname{Beta}(\alpha_{0h} + k_{jht}, \beta_{0h} + n_{jht} - k_{jht})$ gives shrinkage-regularized point estimates and credible intervals that account for unequal sample sizes across comparable audits.

The same statistical requirements guide the choice of reviewer models and allocation methods. Dawid-Skene models describe how reviewer confusion matrices differ relative to latent ground truth. DeCCaF brings task conditions, error costs and reviewer capacity into human-AI deferral, including what difficult assignments do to naive evaluations of expert performance. The production architecture ties selective review assignment to unbiased population quality inference in ongoing post-training work (Dawid-Skene; DeCCaF).

II. Correcting predictions with audits to estimate population quality

Fix a production batch of $N$ original submissions, its acceptance criteria and its latent reference labels $Y_1, \dots, Y_N$. The inferential target is the finite-batch error rate $\mu = N^{-1} \sum_{i=1}^N Y_i$. Let $f_i \in [0, 1]$ be a fixed, inexpensive risk prediction that combines contributor reliability posteriors, task complexity and automated heuristic checks. Assign audit probability $\pi_i \in [\pi_{\min}, 1]$ and draw $S_i \sim \operatorname{Bernoulli}(\pi_i)$ independently across submissions, with the predictions and the sampling probabilities locked before the current batch's reference labels are revealed.

I used a prediction-assisted estimator with inverse-probability-weighted (IPW) residual correction:

Its validity is design-based, and the conditional expectation below shows why. Write the prediction residual as $e_i = Y_i - f_i$ and condition on the fixed batch, the predictions and the sampling probabilities:

The inverse-probability weights restore the population representation that selective verification altered, and the predictions reduce the residual variation that human expert audits have to resolve. Design-based unbiasedness does not require $f_i = q(X_i)$: inaccurate predictions may reduce efficiency, but the estimator remains unbiased under the specified Bernoulli sampling design. The estimator follows active statistical inference principles (Active Statistical Inference: mean estimation and sampling design). To use it in production, I defined procedures for freezing the audited objects, recording rubric version histories and retaining the telemetry that optimization needs.

Original and revised submissions have to be retained as separate database entities. Telling defects introduced during initial production apart from those corrected through review takes two estimates, $\hat{\mu}_{\text{raw}}$ on frozen originals and $\hat{\mu}_{\text{final}}$ on post-review deliverables. A change in post-training criteria causes a different confusion, since stricter acceptance standards can be misread as declining contributor ability. To prevent that, a shared set of anchor cases is adjudicated under both rubric versions whenever the criteria change, and submissions from different periods can then be compared under a common reference version.

III. Allocating audit effort by residual uncertainty and cost

Because the sampling is independent Bernoulli, the cross-item covariance terms drop out and the design variance is exactly:

This expression sets the objective for audit allocation: audit effort should go to the uncertainty that automated risk predictions leave unresolved. For a binary error label with conditional error probability $q_i$, the expected squared residual has two components:

The first is the conditional Bernoulli variance $q_i(1 - q_i)$ and the second is the squared calibration error $(q_i - f_i)^2$. It follows that a high error probability $q_i$ need not imply high inferential value. If a task category is almost certain to contain an error ($q_i \approx 1$) and the predictor recognizes this ($f_i \approx 1$), then $m_i \approx 0$. An independent reference audit of that item contributes less information about population quality than an audit of a moderately risky item whose outcome remains uncertain ($q_i \approx 0.5$). So a remediation model and a measurement model look at the same task differently, the first measuring the benefit of correcting high-risk tasks and the second evaluating how audits improve population estimates.

Let $c_i > 0$ be the empirical cost of an independent reference audit, calibrated by reconciling platform review logs with active-work records from the team's KPI monitoring system, and let $B$ be the expected audit-cost budget. When the true residual second moments $m_i > 0$ are known, the optimal design solves:

The objective omits the constant $-\sum_i m_i$ from the variance numerator because that constant does not depend on the allocation $\pi$. For probabilities strictly between their bounds, the Lagrangian first-order condition gives the clipped square-root allocation:

For a budget below the cost of a full audit, $\lambda > 0$ is chosen to satisfy the budget constraint. The multiplier need not be unique when clipping is active, and feasibility requires $B \ge \pi_{\min} \sum_i c_i$. If the budget does cover a full audit, every item is audited ($\pi_i^* = 1$). The square-root rule samples harder where expected residuals are larger and more lightly where audits cost more, while the positive probability floor $\pi_{\min}$ maintains coverage across the task space and bounds the IPW weights.

Production allocation uses regularized stratum estimates $\hat{m}_i > 0$ from earlier independent audits or a pilot calibration batch. The true $m_i$ defines the optimality analysis, and the estimates $\hat{m}_i$ determine operational allocation. Residuals are estimated at the level of (task stratum × criteria version × cost stratum) to avoid fitting unstable parameters to individual submissions. The budget $B$ constrains expected audit cost per batch. A workflow requiring an exact integer audit count needs a feasible fixed-size sampling design whose first-order inclusion probabilities sum to that count. The variance of such a design depends on joint inclusion probabilities, and its inclusion probabilities need not equal the Bernoulli optimum $\pi_i^*$.

IV. Allocation gains and the conditions they require

Holding the predictor fixed, I derive what allocation based on residuals and costs gains over uniform sampling. The same derivation shows when that gain vanishes or the allocation fails. Let $C = \sum_{i=1}^N c_i$, so the uniform sampling probability at the same expected cost is $p = B / C$. If the true residual second moments $m_i$ are available and no optimal probability reaches a clipping boundary, averaging the design variance over the conditional label distribution gives:

If all items within a fixed task stratum share a sampling probability, substituting that stratum's finite-batch mean squared residual for $m_i$ turns these expressions into exact finite-batch identities. By the Cauchy-Schwarz inequality:

Equality holds if and only if $m_i / c_i$ is constant across items. The efficiency gain therefore depends on identifiable variation in residual second moment per unit audit cost, and with homogeneous residuals and costs the optimal allocation is uniform. If clipping is active at $\pi_{\min}$ or 1$1$, constrained optimization remains no worse than a feasible uniform policy, and the gain can be evaluated directly from the clipped primal objective.

The analysis also puts a number on the variance penalty from model misspecification. If allocation uses inaccurate residual estimates $\hat{m}_i$ and neither allocation is clipped, the excess variance relative to the true optimum is:

The dangerous term is the denominator $\sqrt{\hat{m}_i}$, through which a severe underestimate of $\hat{m}_i$ in any part of the task space increases the penalty. A positive sampling floor $\pi_{\min}$ limits the worst-case variance increase. Active and uniform sampling probabilities can also be combined without changing the budget:

The shrinkage weight $\rho \in [0, 1]$ can be tuned using independent calibration batches (Robust Sampling for Active Statistical Inference).

V. Allocating review by expected loss reduction

Statistical auditing measures population quality with minimum variance. Operational review has a different job, removing defects from the current batch before it is delivered for training, so one objective is estimation precision and the other is delivered quality. For task $i$ under handling action $a$, such as direct acceptance, peer review, expert adjudication or contributor revision, let $r_{ia}$ be the expected error loss remaining after the action, $d_{ia}$ its operational cost, and $t_{iaj}$ the time required from specialist $j$, measured through platform records and the team's KPI monitoring system. Binary assignment decisions $z_{ia} \in \{0, 1\}$ solve:

Here $H_j$ is the specialist's available time, and $\eta \ge 0$ converts resource cost to the scale of error loss (DeCCaF: capacity constraints and global assignment).

When review costs are equal and each reviewed task uses one slot, the marginal remediation gain $\Delta_i = r_{i0} - r_{i1}$ determines the optimal priority order. Initial risk $r_{i0}$ measures the risk before review. A high-risk task may be difficult to repair through one additional review because external evidence is missing, which gives a small $\Delta_i$. A self-contained calculation error in a moderate-risk task may be fully correctable, which gives a large $\Delta_k$. Moving one review slot from task $i$ to task $k$ changes total expected loss by:

Whenever $\Delta_k > \Delta_i$, the exchange strictly reduces delivered defect loss, regardless of the initial risk ordering.

Error in the action-risk estimates also determines a performance bound for review allocation. Suppose $|\hat{r}_{ia} - r_{ia}| \le \delta$ for every feasible action, costs and capacities are known, and $\hat{z}$ minimizes the predicted objective $\hat{F}(z)$. Relative to the true optimal feasible assignment $z^*$:

The bound identifies post-action residual loss $r_{ia}$ as the empirical quantity that must be calibrated, which provides a basis for randomized pilot trials across permitted review actions.

The remediation and measurement budgets can be combined in a batch-level mixed-integer convex program (MICP). Let $\tau^2$ be the target variance ceiling for the quality estimate, $B_{\text{total}}$ the total cost budget, and $V_{\hat{m}}(\pi) = N^{-2}\sum_i (1 - \pi_i)\hat{m}_i / \pi_i$ the planning variance under the calibrated residual model:

The hard capacity $H_j^{\text{review}}$ goes to remediation actions, while reference audits are scheduled independently on frozen original submissions. The precision ceiling is model-based, so it requires independent audits to check residual calibration. An expected audit-cost constraint imposes no hard limit on realized audit hours. If the program is infeasible, the precision target, the feasible review assignments and the available engineering budget are in conflict.

VI. Reproducible experiments at equal cost

I tested the mechanisms above in synthetic experiments with publicly specified parameters. A fixed batch contains $N = 10,000$ submissions in three strata of 6,000$6,000$, 3,000$3,000$ and 1,000$1,000$ items, with 60$60$, 300$300$ and 350$350$ errors, respectively, so the true batch defect rate is $\mu = 7.10\%$. Unit audit costs across the three strata are 1$1$, 1.5$1.5$ and 3$3$. Every policy gets an expected audit budget of $B = 2,000$ units and a minimum sampling probability of $\pi_{\min} = 3\%$.

Risk predictions for the three strata are 2%$2\%$, 8%$8\%$ and 30%$30\%$. Their batch average of 6.60%$6.60\%$ differs from the true defect rate by 0.50$0.50$ percentage points. The cost-aware allocation uses the true within-stratum mean squared residuals, which represents ideal calibration, and the misspecified policy reverses the residual ordering across strata. With predictions, budget and submissions held fixed, 200,000$200,000$ independent Monte Carlo repetitions verify the analytical variance formulas:

Under ideal residual calibration, the standard deviation of the population quality estimate falls by 9.33%, a 17.79% reduction in variance. The high-risk-first policy fares worse. Its uncorrected error detection rate averages 19.60%, overstating the population defect rate. IPW residual correction recovers 7.1015%, but the policy oversamples the more expensive third stratum, which increases estimation variance by 33.77%. Across all five policies, Monte Carlo variances differ from the analytical formulas by no more than 0.5%$0.5\%$.

A separate exact allocation experiment isolates the effect of remediation. Two task groups of 100$100$ items each have initial error risks of 0.40$0.40$ and 0.15$0.15$, and additional review reduces these risks to 0.30$0.30$ ($\Delta_1 = 0.10$) and 0.01$0.01$ ($\Delta_2 = 0.14$), respectively. Review costs are equal, and capacity is 100$100$ items. Ranking by initial risk leaves $100(0.30) + 100(0.15) = 45$ expected errors, while ranking by expected loss reduction ($\Delta_i$) leaves $100(0.40) + 100(0.01) = 41$, which reduces expected defect loss by 8.89% at the same review volume.

VII. Production validation and evaluation criteria

I defined separate production evaluations for statistical quality inference and operational defect control. Errors detected in a targeted queue indicate how well review locates defects, but an improvement in delivered quality has to be assessed through the defects that remain and the associated costs.

1. Inference on frozen, adjudicated batches. Compare uniform sampling, legacy high-risk routing, cost-aware residual allocation and defensive mixtures against complete independent adjudication, assessing bias, mean squared error (MSE) and interval coverage within task strata and rubric versions.

2. Remediation trials at matched capacity. Compare legacy tiered rules with marginal-gain ($\Delta_i$) action allocation at equal total specialist hours, measuring the severe defects remaining in the final post-training corpus and the fully loaded cost per accepted submission.

From September 2023 I led the development of the contributor reliability records, tiered review workflows and random reference audits that established the project's operating standards for expert collaboration, quality measurement and resource allocation. I specified the sampling, calibration and capacity assumptions under which quality estimation is unbiased and, when residual second moments are correctly specified, audit allocation is optimal. The synthetic benchmarks test these mechanisms under their stated conditions.