Large-Scale Domain Expert Feedback Governance and Adaptive Review
System Architecture
Intake
Allocate
Decide
Output
Returns to 02 Allocate: Adjudications update reliability posteriors and review intensity
Problem and Approach
In late 2023 I started from a question about expert capacity: how should a post-training program spend a limited pool of reviewers when each review does two jobs at once? A review reveals something about the quality of the whole batch, and a correction improves the data that ships. The two values call for different allocations of the same hours. High-risk routing shows the tension. When review succeeds in targeting the tasks with higher true error probabilities, selection is correlated with the defect it looks for (), and on a reference batch at the program's scale the error rate in the review queue reads two to four times the batch's rate.
I separated the two jobs and then joined them under one budget. Quality is measured by independent reference audits drawn with known probabilities, so the estimate is correct whatever the risk model gets wrong. Audit effort goes where residual uncertainty per unit of expert cost is largest, and repair goes where a review removes the most defect loss per unit cost. A single rule then sets both. An audit that finds a defect has also repaired it, so risk enters the audit decision as a discount on the audit's price, and risk-first routing turns out to be this rule with the value of measurement set to zero. Costs, latencies and capacities came from production records. The design coordinated hundreds of distributed specialists and external vendor queues, and I kept refining its specifications as later programs took it up. Sections 1 through 7 give the derivations, the results on the reference batch and the operating standards.
Key Contributions
- I formulated expert review as a statistical decision problem in which each review has two separate values: what it reveals about the quality of the batch, and what it repairs in the data that ships. The formulation took shape in late 2023, and I kept refining it as the program grew. For each task I defined an initial error risk (), the statistical information value of an independent reference audit (), and the expected defect loss removable through additional human handling (). A high-risk task can be worth little to audit and little to repair, so the three quantities call for different uses of the same expert capacity.
- I derived the allocation each value calls for. Audit probabilities follow a clipped square-root rule in residual second moment and cost, which is Neyman's allocation carried to a separate expert cost for every item, with a sampling floor. Repair is ranked by removable defect loss per unit cost under specialist-capacity limits.
- I joined the two in one rule and located risk-first routing inside it. A reference audit that finds a defect also corrects it, so the optimal audit probability is , in which the item's residual risk is a rebate on the audit's price . With the price of precision at zero the rule becomes a threshold on risk per unit cost, which is risk-first routing. The batch program separates by price, so ten thousand tasks are solved by a search over a handful of multipliers.
- I derived what the allocation is worth in expert hours and how accurate its inputs must be. Every precision target costs the same fixed fraction of the uniform audit budget. Errors in the residual estimates cost a quarter of their weighted variance, so a 20% relative error raises the variance by about 1%.
- The framework supported a program of more than 300 domain experts across several specialties and an external vendor network, and later post-training programs reused its design. I led its development and defined the mathematical specifications and operating standards for task organization, reliability calibration, acceptance adjudication and review allocation. Those procedures made distributed expert judgments traceable and open to calibration as post-training feedback.
- I designed tiered review rules from Beta-Binomial reliability posteriors, credible intervals, verified sample counts and task-level risk priors. In one workflow, a flagged submission was dynamically assigned to three additional specialists, and an initially accepted one went to one or two. Expert adjudications fed back into the system, updating contributor reliability posteriors, informing targeted coaching and adjusting review intensity.
- I paired independent random reference audits with escalation of high-risk tasks, so quality could still be measured while review concentrated on the hardest work. Recurring annotator disagreements became explicit acceptance criteria. I separated missing context from ambiguous instructions and from contributor errors, then revised the task instructions, boundary cases and shared calibration anchors accordingly. Later comparisons against reviewer-error and cost-sensitive deferral models informed the escalation thresholds.
- I built the SQL and Python pipelines behind the allocation solvers. They brought together task metadata, review histories, acceptance outcomes, turnaround times and behavioral logs, and reconciled platform activity with the team's KPI monitoring system. From these records the solvers took empirical estimates of unit verification costs (), task-stratum latencies () and hard specialist capacity limits (). I also translated client-reported model defects into reproducible engineering investigations and standard operating procedures.
- On a reference batch at the program's scale, the joint rule ships 38.6% fewer defects than risk-first routing at the same budget and the same precision, and 19% to 67% fewer across a 20% envelope around that batch. The uncorrected review queue reads 19.60% where the batch's defect rate is 7.10%. The residual- and cost-aware allocation reaches any precision target with 15.2% less audit budget than uniform sampling (7.5% to 27.3% across the envelope), and risk-first routing carries 62.7% more variance than that allocation (14% to 137%).
1. What a review queue measures
In a production RLHF pipeline, whatever policy routes submissions to review also changes the statistical properties of the observed sample. To see how, let denote observable task features, including the domain, prompt complexity and contributor history. Let indicate whether the original submission contains an error under fixed acceptance criteria, so that the true conditional error probability is . Write when a submission is selected for an independent reference audit. Conditional on , selection is randomized with probability , and it is independent of the reference label, which has not yet been observed.
Suppose an operational dashboard estimates population quality from the error fraction among verified submissions. What it actually targets is:
When review succeeds in targeting higher true error risk, the covariance between audit probability and true defect probability is strictly positive, and the dashboard reads high. A rising detection rate in the reviewed sample is what better targeting looks like, and the batch may be unchanged or improving underneath it. The identities describe a selected item; in a finite batch the observed fraction has a random denominator and converges to this value.
This is Neyman's argument against purposive selection, applied to a review queue: a sample chosen to serve one purpose fails on the quantities it was not chosen for (Neyman 1934). Financial auditors already act on it. Under the GAO Financial Audit Manual the auditor examines large and unusual items in full, samples the remainder at random, projects only the random sample to the population and records which selection is which. The PRM800K report describes the bias and leaves it there: its authors chose convincing wrong-answer solutions for labeling and noted that the choice skewed the data set toward wrong answers (Let's Verify Step by Step).
Expert rankings suffer from the same selection mechanism. Let be the probability that expert answers a task in stratum correctly, and let describe that expert's assigned task mix in period . Even if underlying ability stays constant, observed accuracy moves with the task mix:
Comparisons across periods therefore require a fixed target task mix and standardized accuracy:
Overlapping assignments or shared anchor cases support estimation within each stratum. In the project's production routing engine I used a conjugate Beta-Binomial model for each contributor's reliability within a task stratum. Given a prior and audited tasks with verified successes, the posterior gives shrinkage-regularized point estimates and credible intervals that account for unequal sample sizes across comparable audits. Section 7 sets out how these records drive contributor decisions.
Each piece of this problem has a literature, and each literature stops at one of the two jobs. Dawid-Skene models estimate how reviewers' confusion matrices differ and carry no sampling design. PPI++ gives valid inference from a small labeled sample and a large set of predictions when the labels are drawn uniformly. Active Statistical Inference chooses which labels to collect and prices every label the same; cost-optimal active evaluation prices two kinds of rater, at one price per kind for all items. ActiveLab decides whether the next annotation should go to a new example or to one already labeled, with a better classifier as the goal. DeCCaF assigns cases to experts under error costs and capacity limits, and the quality of the batch is outside its objective. A production program has to price both jobs in one unit, expert hours, and the sections below do that.
2. Measuring a batch with audits and predictions
Fix a production batch of original submissions, its acceptance criteria and its latent reference labels . The inferential target is the finite-batch error rate . Let be a fixed, inexpensive risk prediction that combines contributor reliability posteriors, task complexity and automated heuristic checks. Assign audit probability and draw independently across submissions, with the predictions and the sampling probabilities locked before the current batch's reference labels are revealed.
I used a prediction-assisted estimator with inverse-probability-weighted (IPW) residual correction:
Write the prediction residual as and condition on the fixed batch, the predictions and the sampling probabilities:
Validity comes from the design and efficiency from the model. The inverse-probability weights restore the population representation that selective verification altered, and the estimate is unbiased for any predictions at all, because the only randomness it relies on is the audit lottery the program itself runs. Better predictions shrink the residuals and with them the number of audits a given precision needs. That division of labor suits production: the estimate stays correct while the risk model is still being improved. The form is the difference estimator of survey sampling, Horvitz-Thompson weighting applied to residuals, and the augmented IPW estimator of Robins, Rotnitzky and Zhao; PPI++ studies it with black-box predictions.
How much the predictions can save has a known ceiling under uniform sampling: a predictor no more accurate than the work it scores can at most double the effective audit sample (Dorner et al.). The gains in Sections 3 and 4 come from where the audits are sent, which is a separate source of efficiency.
To use the estimator in production, I defined procedures for freezing the audited objects, recording rubric version histories and retaining the telemetry that optimization needs. Original and revised submissions are retained as separate database entities. Telling defects introduced during initial production apart from those corrected through review takes two estimates, on frozen originals and on post-review deliverables. A change in post-training criteria causes a different confusion, since stricter acceptance standards can be misread as declining contributor ability. A shared set of anchor cases is therefore adjudicated under both rubric versions whenever the criteria change, and submissions from different periods are compared under a common reference version.
3. Where audits buy the most information
Because the sampling is independent Bernoulli, the cross-item covariance terms drop out and the design variance is exactly:
This expression sets the objective for audit allocation: audit effort should go to the uncertainty that automated risk predictions leave unresolved. For a binary error label with conditional error probability , the expected squared residual has two components:
The first is the conditional Bernoulli variance and the second is the squared calibration error . A high error probability therefore need not carry high inferential value. If a task category is almost certain to contain an error () and the predictor recognizes this (), then , and an independent reference audit of that item says less about population quality than an audit of a moderately risky item whose outcome is still open (). A repair model and a measurement model look at the same task and want different things from it.
Where an audit is informative
Let be the empirical cost of an independent reference audit, calibrated by reconciling platform review logs with active-work records from the team's KPI monitoring system, and let be the expected audit-cost budget. With residual second moments , the optimal design solves:
The objective omits the constant , which does not depend on the allocation. For probabilities strictly between their bounds, the Lagrangian first-order condition gives the clipped square-root allocation:
The multiplier is set by the budget, feasibility requires , and a budget that covers a full audit audits everything. The rule samples harder where expected residuals are larger and more lightly where audits cost more, while the floor keeps every part of the task space covered and bounds the IPW weights.
The square-root form has a long ancestry. With equal costs and strata it is Neyman's allocation; with stratum costs it is the optimum allocation of Cochran's text; inclusion probabilities proportional to a model's error standard deviation are the anticipated-variance optimum of Isaki and Fuller; and Chen and Lumley set such designs wave by wave from earlier waves. Machine-learning work reached the same rule for labels of equal cost. Expert audits do not cost the same: on the reference batch of Section 6, an independent reference solution for the hardest tasks costs three times a routine one. My rule carries a cost for every item.
Production allocation uses regularized stratum estimates from earlier independent audits or a pilot calibration batch, at the level of (task stratum × criteria version × cost stratum), so that no parameter is fitted to an individual submission. A weekly program is a multiwave design in which each batch's audits set the next batch's allocation. The budget constrains expected audit cost per batch; a workflow that needs an exact audit count uses a fixed-size design with the same first-order inclusion probabilities.
4. What the allocation is worth, and how accurate its inputs must be
Let , and . Uniform sampling at the same expected cost uses probability . With no probability at a clipping boundary, averaging the design variance over the conditional label distribution gives:
If all items within a task stratum share a sampling probability, substituting that stratum's finite-batch mean squared residual for makes these exact finite-batch identities. By the Cauchy-Schwarz inequality , with equality if and only if is constant across items. The gain exists exactly where residual uncertainty per unit of audit cost varies across the batch.
The same precision for a fixed fraction of the budget. A program manager fixes the precision the dashboard needs and asks what it costs. Solving each variance for the budget that reaches a target :
The ratio does not depend on the target. Whatever precision the program asks for, the residual- and cost-aware allocation delivers it for the same fraction of the uniform audit budget, and that fraction is a property of the batch: how unevenly residual uncertainty and audit cost are spread across it. The hours it frees, , are available for repair, which is the first link between the two jobs.
How accurate the residual estimates must be. The allocation is computed from estimates . Write and weight each item by its share of the optimal budget. With neither allocation clipped, the variance of the allocation built on the estimates exceeds the optimum by:
The first-order terms cancel. Three consequences follow for the specification of the estimation pipeline. Overestimating or underestimating every residual by a common factor costs nothing, since only the dispersion of the relative errors enters. A relative error of 20% across strata raises the variance by about one percent of , so stratum-level estimates carried over from earlier batches need to be accurate only to that order. And the exact expression shows where the rule can be hurt: through , by a gross underestimate of a residual somewhere in the task space.
Two guards bound that last case. The floor bounds every weight. Mixing the computed probabilities with the uniform design at the same budget,
bounds the variance for any estimates whatever, because every probability is then at least :
This is the defensive mixture of importance sampling (Hesterberg 1995) written for a Bernoulli audit design. The weight is tuned on earlier batches; robust active inference later gave an asymptotic choice of the same weight.
5. One budget, two products
Statistical auditing measures population quality. Operational review has a different job, removing defects from the current batch before it is delivered for training. For task under handling action , such as direct acceptance, peer review, expert adjudication or contributor revision, let be the expected error loss remaining after the action, its operational cost, and the time required from specialist , measured through platform records and the team's KPI monitoring system. Binary assignment decisions solve:
Here is the specialist's available time, and converts resource cost to the scale of error loss.
Repair follows removable loss. When review costs are equal and each reviewed task uses one slot, the marginal remediation gain determines the optimal priority order. A high-risk task may be hard to repair through one additional review because external evidence is missing, which gives a small . A self-contained calculation error in a moderate-risk task may be fully correctable, which gives a large . Moving one review slot from task to task changes total expected loss by:
Whenever , the exchange strictly reduces delivered defect loss, regardless of the initial risk ordering. Optimal triage under a budget has the same threshold structure, with the human's case-level loss in place of the model's risk (Okati et al.). A worked example fixes the size of the effect. Two task groups of items each have initial error risks of and , and additional review reduces these risks to () and (). Review costs are equal and capacity is items. Ranking by initial risk leaves expected errors; ranking by leaves , a reduction of 8.89% at the same review volume.
Where 100 reviews remove the most loss
The quantity that has to be calibrated is the post-action risk. Suppose for every feasible action, costs and capacities are known, and minimizes the predicted objective . Relative to the true optimal feasible assignment :
That bound is the reason the program ran randomized pilot trials across the permitted review actions: they estimate directly.
The batch program. Audits and repair draw on the same money. Let be the variance ceiling for the quality estimate, the total cost budget, and the planning variance under the calibrated residual model. A reference audit produces an independent reference solution, so an audited item found defective ships corrected. Writing for the action assigned to item , the item ships with expected loss , and the batch is planned by a mixed-integer convex program:
The hard capacity goes to remediation actions, and reference audits are scheduled independently on frozen original submissions.
The joint rule. Attach a price to the budget, to the precision ceiling (in units of ) and to each specialist's hours. At those prices the program separates into one small problem per task, and each has a closed form:
with when . Three readings of the audit rule carry the argument of this article.
1. Risk is a rebate. The item's residual risk is subtracted from the audit's price . Risk does not drive the audit; it makes the audit cheaper, by the repair the audit performs on the way.
2. Measurement alone. If audits repaired nothing, the rebate would vanish and the rule would be the allocation of Section 3.
3. Repair alone. If nobody valued the estimate, , the rule audits exactly the items with , a threshold on risk per unit cost. That is risk-first routing. The legacy policy is the joint rule of a program that does not care what its batch quality is.
Initial risk, residual uncertainty per unit cost and removable loss now sit in one pair of formulas, and design-unbiasedness is untouched: the probabilities depend only on quantities fixed before the labels are revealed, and is still computed on frozen originals.
The prices also make the program cheap to solve. For fixed every task's audit probability and action follow from the two formulas above in one pass, so a weekly batch of ten thousand tasks is a search over , and one price per specialist, with a linear-time step. The linear relaxation of the assignment splits at most one task per binding budget or capacity constraint, so rounding to whole assignments moves at most that many tasks, and the gap between the priced bound and the rounded plan certifies the solution. When the program is infeasible, the price that diverges names what is short: the precision target, the specialist hours or the budget.
The ratio is the exchange rate between the two products: the number of delivered defects the program gives up to tighten the estimate by one unit of variance. Section 6 reads it off the reference batch.
6. Results on a reference batch
Reference batch. Production records and operating figures are withheld under the nondisclosure agreement. The results below are stated on a reference batch at the program's scale: one week of work, submissions in three strata by difficulty, most of them routine and a tenth of them hard, with defect rates and audit costs that rise with difficulty. Each headline result is given twice: at the reference batch, and as its range over an envelope in which each entry of the batch moves independently by up to 20% in either direction.
Table 1. The reference batch in three strata by difficulty.
| Routine | Intermediate | Hardest | |
|---|---|---|---|
| Share of the batch | 60% | 30% | 10% |
| Defect rate of original submissions () | 1% | 10% | 35% |
| Frozen risk prediction () | 2% | 8% | 30% |
| Cost of an independent reference audit (, routine = 1) | 1 | 1.5 | 3 |
| Share of defects removed by one additional review | 80% | 80% | 30% |
The batch defect rate is . One review costs half of a reference audit in the same stratum. The audit budget is 2,000 cost units, 14.8% of the cost of auditing everything, with a sampling floor of .
Measurement at equal audit budget. Every policy below uses the estimator of Section 2 and spends the same expected audit budget.
Table 2. Audit policies at an equal expected audit budget.
| Audit policy | Audit probability by stratum | Standard deviation of the estimate (percentage points) | Variance relative to uniform | Range over the envelope |
|---|---|---|---|---|
| Uniform sampling | 14.81% / 14.81% / 14.81% | 0.568 | Baseline | Baseline |
| Risk-first (probability proportional to ) | 3.00% / 11.56% / 43.33% | 0.657 | +33.8% | -16% to +97% |
| Residual- and cost-aware (Section 3) | 7.89% / 19.37% / 21.84% | 0.515 | -17.8% | -9% to -32% |
Three results hold everywhere in the envelope. The uncorrected error rate in the risk-first queue reads 19.60% against a batch rate of 7.10%, an overstatement of 2.76 times, and between 1.96 and 4.14 times across the envelope; the correction of Section 2 recovers 7.10% under every policy. The residual- and cost-aware allocation always measures better than uniform sampling. And risk-first routing always measures worse than that allocation, with 62.7% more variance at the reference batch and 14% to 137% more across the envelope. How risk-first routing compares with random sampling is not stable: it is 34% worse at the reference batch and can fall on either side within the envelope. Its quality as a measurement design depends on how predicted risk happens to line up with residual uncertainty and audit cost, which nothing in the policy controls.
Three audit policies at the same expected budget
The price of precision. The budget ratio of Section 4 is : the residual- and cost-aware allocation reaches any precision target with 15.2% less audit budget than uniform sampling, and with 7.5% to 27.3% less across the envelope.
Table 3. Audit budget needed to reach each precision target.
| Target standard deviation (percentage points) | Audit budget, residual- and cost-aware | Audit budget, uniform |
|---|---|---|
| 0.50 | 2,100 | 2,475 |
| 0.40 | 2,974 | 3,506 |
| 0.30 | 4,399 | 5,185 |
| 0.25 | 5,419 | 6,387 |
Calibration tolerance. With relative errors of alternating sign and size in the three stratum residual estimates, the variance rises above the optimum as follows.
Table 4. Variance cost of errors in the stratum residual estimates.
| Relative error in the residual estimates | Variance above the optimum | Quarter-variance law |
|---|---|---|
| 10% | 0.30% | 0.30% |
| 20% | 1.23% | 1.19% |
| 30% | 2.88% | 2.68% |
| 50% | 9.2% | 7.4% |
The stress case is a residual ordering reversed across the strata, which sends the allocation to 23.00% / 11.77% / 3.00% and raises the variance to 199.7% above uniform. With a weight of on the uniform design the same wrong estimates cost 17.1%, and the bound of Section 4 guarantees no more than 50.3% for any estimates.
One budget, two products. The total budget is 5,000 cost units, 37% of the cost of auditing everything, to be divided between audits and reviews. With no handling the batch ships 710 defects.
Table 5. One budget of 5,000 cost units divided between audits and reviews.
| Plan | Audit probability by stratum | Share of each stratum reviewed | Standard deviation of the estimate (percentage points) | Defects shipped |
|---|---|---|---|---|
| Risk-first for audits and for reviews | 3.00% / 11.56% / 43.33% | 0% / 67% / 100% | 0.657 | 320.9 |
| Measurement rule of Section 3, then reviews by removable loss | 7.89% / 19.37% / 21.84% | 0% / 100% / 50% | 0.515 | 336.2 |
| Joint rule at the precision of risk-first routing | 3.31% / 9.57% / 100% | 0% / 61% / 0% | 0.657 | 197.1 |
| Joint rule at the precision of uniform sampling | 4.20% / 12.74% / 100% | 0% / 52% / 0% | 0.568 | 209.9 |
| Joint rule with no precision ceiling | 3.00% / 3.00% / 100% | 0% / 75% / 0% | 1.035 | 174.9 |
At the same budget and the same precision, the joint rule ships 38.6% fewer defects than risk-first routing, and 19% to 67% fewer across the envelope. Held to the tighter precision of uniform sampling it still ships 34.6% fewer.
Table 5 also shows why the two jobs cannot be planned one after the other. Choosing audits for measurement and then spending the rest on repair gives the best estimate in the table and ships more defects than the legacy policy, because it ignores the repair that audits perform. Held to the sequential plan's own precision of 0.515 points, the joint rule ships 34.7% fewer defects than that plan, and 12.4% to 69.3% fewer across the whole envelope. The joint rule sends every item of the hardest stratum to an independent reference solution: there a review removes 30% of the defects and a reference audit removes all of them, at a risk per unit cost of against a budget price of 0.093 to 0.096. In the two easier strata the audit probabilities follow the closed form of Section 5: the implied by the routine stratum and the implied by the intermediate stratum agree to six digits in both joint plans.
The last three rows are the exchange rate. Holding the estimate at 0.657 points costs 22 shipped defects relative to no ceiling, and tightening it to 0.568 costs 13 more. A program that knows this number can decide how much it wants to know.
One budget of 5,000 cost units: precision against defects shipped
The envelope for this table covers the eight entries that enter the comparison: the hardest stratum's share, prediction and audit cost, the defect rates and review effects of the two harder strata, and the ratio of review cost to audit cost.
7. Operating the design
Contributor records. Tiered review sends more work from weaker contributors to audit, so the number of audits a contributor has is correlated with reliability, and shrinkage toward a common prior misreads exactly that pattern (Brown 2008). The random reference audits supply the correction: their selection does not depend on the contributor's record, so reliability estimated on that stream is free of the routing effect and calibrates the posteriors built from adjudications. For decisions about coaching and review intensity the posterior is read through one number per contributor and stratum, the probability of falling below the stratum standard :
where is the regularized incomplete beta function. Contributors enter the action list in decreasing order of until the average of over the list reaches a set false-selection rate, which treats selection as the compound decision it is (Gu and Koenker). A worked example with prior and standard shows what the rule corrects. A contributor with 33 of 40 audited tasks correct has ; one with 6 of 8 correct has ; one with 57 of 60 correct has . Raw accuracy would send the second contributor to coaching first on eight audits. The tail probability puts the first one ahead, on five times the evidence.
What an auditor's error costs. The estimator measures the rate of audit labels, and audit labels are themselves a measurement. Let the auditor's sensitivity and specificity be estimated from blind re-audits, with . The true rate is recovered by the classical correction (Rogan and Gladen), at times the variance, and the budget identity of Section 4 converts that into audit hours. On the reference batch, holding the true rate to a standard deviation of 0.568 points:
Table 6. Audit budget with an imperfect auditor, at a standard deviation of 0.568 points.
| Auditor error | Audit budget relative to an error-free auditor |
|---|---|
| 5% of defects missed | +5.2% |
| 2% of clean items flagged | +43.8% |
| Both | +52.8% |
False flags cost about eight times what misses cost here. More than nine items in ten are clean, and a 2% false-flag rate on them nearly triples the residual variance of the routine stratum. The first target of auditor calibration is therefore the false flag.
Comparability over time. Each estimate belongs to one frozen criteria version. Dual-version anchors link versions when the criteria change, and the same anchor set serves to detect drift in auditors' standards between changes.
Evaluation in production. I defined separate production evaluations for the two jobs, because errors found in a targeted queue measure how well review locates defects and nothing else.
1. Inference on frozen, adjudicated batches. Uniform sampling, legacy high-risk routing, cost-aware residual allocation and defensive mixtures are each set against complete independent adjudication, and judged on bias, mean squared error (MSE) and interval coverage within task strata and rubric versions.
2. Remediation trials at matched capacity. Legacy tiered rules are set against marginal-gain () action allocation at equal total specialist hours, and judged on the severe defects remaining in the final post-training corpus and the fully loaded cost per accepted submission.
The program worked through roughly one batch a week, with more than 10,000 tasks per batch and more than 300 specialists. Contributor reliability records, tiered review workflows and random reference audits became its operating standards for quality measurement and review allocation, and later post-training programs took over the design.