Evidence Verification and Robust Comparison in Investment Research
System Architecture
Evidence
Score
Bound
Plan
Returns to 01 Evidence: Each correction triggers rescoring and attribution updates
Problem and Approach
Due diligence has a limited budget of analyst time and a long list of facts that could be checked. In late 2023 I began treating that as a decision problem: given the evidence in hand, which conclusions does it support, and which facts would most improve the decision if they were verified? AI-polished narratives, unverified sustainability claims and common reporting errors can all move ESG and investment scores, so the answer has to account for uncertainty in the corporate evidence as well as in the scoring models. The usual guide to what deserves checking is feature attribution, which measures what the model cares about. What a diligence team needs to know is which evidence is still open and able to change the decision.
The architecture I developed takes a traceable evidence state as its input and builds in model multiplicity, joint measurement error and the costs of verification. Its guarantees need no probability that any particular disclosure is exaggerated. It returns three things an investment committee can act on: the set of pairwise rankings that hold whatever the unverified evidence turns out to be, the number of simultaneous worst-case misstatements each ranking survives, and the verification plan that removes the most decision regret for the analyst hours available. I kept refining it through 2026. Sections 1 through 6 develop the limits of attribution methods, the joint uncertainty model, ranking bounds, the verification plan, versioned updates and worked examples.
Key Contributions
- I framed investment due diligence as a decision problem: which conclusions does the current evidence support, and which fact is most valuable to verify next? The framework took shape in late 2023, and I refined it through the work that followed. I represented unresolved evidence jointly with a near-optimal model set through and derived convex support-function bounds () that give ranking guarantees across every retained model and admissible resolution of the evidence. Under the same affine scoring model, common measurement errors cancel when the two companies have identical exposures. In a worked comparison, the joint model certified a 0.6-point advantage that the difference bound from separate company intervals () could not show.
- I derived a query criterion that ranks verification by its effect on the decision, and solved it exactly for affine scores. The criterion is the reduction in worst-case decision regret that a query can guarantee, and it needs no probability distribution over the scenarios inside the joint uncertainty set. With interval evidence, verifying a set of fields with combined swing leaves worst-case regret . A single field therefore has guaranteed value only if its swing exceeds the margin, fields worthless alone can be valuable together, and the best plan within an analyst budget is a knapsack on swing per unit cost.
- I gave each certified ranking a number a committee can read: how many simultaneous worst-case misstatements it survives. One inclusion event certifies every pairwise ranking at once, with no penalty for the number of comparisons, so the output is a certified partial order that verification can only extend.
- I implemented the framework with a leading global alternative asset manager, and subsequent research reused it. Institutional diligence requirements became measures of score reliability, pairwise ranking stability and verification priority. The workflow ingests evidence, scores companies across a set of near-optimal models, computes ranking bounds and allocates verification effort by minimax regret. Each correction triggers rescoring and attribution updates, so the investment committee has a traceable basis for its analysis.
- I showed where conventional bootstrap resampling, and verification priorities based on raw SHAP magnitudes, fail when inputs contain shared reporting bias or unverified claims. Under a fixed common bias, a nominal 95% interval loses coverage as the sample grows, and resampling the same records cannot identify the bias. A self-reported metric near the sector mean can carry a near-zero attribution and still hold enough unresolved error to reverse a ranking. I derived the sample size past which collecting more records is worth less than verifying the ones in hand.
- I analyzed how AI-polished corporate narratives shift LLM-assisted scoring, ESG ratings and investment analysis when the claims beneath them lack factual support. I converted operating reports, audited financial statements, sustainability disclosures, management questionnaires and third-party datasets into structured scoring rules that carry source and version records. AI-assisted checks then pick out unsupported claims and contradictions across documents, and shared reporting errors enter the same review, so that verification in private-market and ESG research can be targeted.
- I deployed gradient-boosted trees and deep tabular models to capture nonlinear thresholds and interactions, and measured how far their explanations depend on model choice. Attributions computed across the near-optimal set , read together with conditional analysis of correlated features, identify the scoring factors whose contributions stay stable across models.
1. Feature attribution and the value of verification
SHAP splits a model prediction into additive feature contributions through its local accuracy property. Conditional Shapley methods also account for dependence among features, and Rashomon-set methods check how stable explanations are across near-optimal models (SHAP; Aas et al.; Marx et al.). All of them take the corporate inputs as given. In investment diligence the inputs are the open question, because self-reported disclosures can be polished well past what the underlying facts support, and an analyst needs to know which corrections to the evidence could change a decision. Attribution was not built for that question, and there is now a theorem to that effect: for expressive model classes, attribution methods that are complete and linear, SHAP among them, can do no better than random guessing at inferring how the model behaves under changed inputs (Bilodeau et al.). The remedy those authors recommend is to evaluate the model directly on the changes of interest, which is what the bounds of Sections 2 to 4 do.
Neighboring work bears on other parts of the problem. Aggregate Confusion documents how widely ESG ratings diverge across agencies and traces most of the divergence to measurement. Robust Active Preference Elicitation selects pairwise comparison queries under set-based uncertainty, with worst-case utility and regret as objectives. My framework works in institutional research where the facts are still being checked, and it connects model disagreement, evidence changes, ranking stability and the choice of what to verify next.
The first failure case concerns verification priorities based on raw attribution magnitudes. Take a linear scoring model with a fixed marginal background distribution . The SHAP attribution for feature is . If the feature has unresolved factual uncertainty bounded by , its sensitivity to verification is , and over a box of admissible evidence errors the exact worst-case change in the company score is
The upper bound comes from the triangle inequality, and it is attained by picking the endpoint of each error interval so that every perturbation pushes the score in the same direction. Apart from the coefficient, that sum depends on how wide the errors are, whereas the attribution depends on where the feature sits relative to the background. Attribution magnitude and verification value are separate calculations. Two cases show the difference:
• A financial metric may have a large SHAP attribution and come from audited filings that have already been verified (). Further checks of that field do little to reduce score uncertainty.
• A self-reported ESG or operating metric may lie near the sector mean () and so receive a near-zero SHAP attribution. Substantial unresolved reporting error () may still be enough to reverse a company ranking.
The second failure case concerns conventional resampling: redrawing existing records cannot identify a shared reporting bias. Suppose reported observations satisfy , where are independent errors and is an unmodeled bias common to all records, as with systematic overstatement across a sector. A conventional nominal 95% normal confidence interval centered on the sample mean covers the true parameter with probability
where . With a fixed nonzero common bias, a larger sample reduces random variance while the interval's coverage of the true parameter tends to zero, and bootstrapping the same biased records adds no information that identifies . The effect is large in practice. A survey with about 250,000 responses a week overestimated US vaccine uptake by 17 percentage points, an estimate no more accurate than a simple random sample of size 10 (Bradley et al.). In ESG data, the association between corporate emissions and stock returns reported in earlier work was traced to emissions figures estimated by one data vendor from financial fundamentals, with minimal evidence of any association in the emissions firms disclosed themselves (Aswani et al.). A shared third-party estimate is the error structure the joint model of Section 2 is built to carry.
Bounding the common bias takes external evidence, and once verification supplies a bound the interval can be made both valid and short. The shortest interval with 95% coverage for every admissible bias has half-width
which is the bias-aware critical value of Armstrong and Kolesár. It also says when to stop collecting. Past
records, the bias bound is the larger part of the half-width, and halving the bound shortens the interval more than doubling the sample does. From there on, analyst time buys more precision in verification than in collection.
2. Joint uncertainty over models and evidence
I represent model multiplicity and unresolved evidence jointly to assess whether an ESG or investment ranking stays stable under admissible changes to either.
On the model side, I fix the prediction target, input definitions and output scale, and let contain candidate scoring models specified before examining the validation data. The validation set consists of i.i.d. observations from the target evaluation distribution, independent of model construction, and the loss is bounded in . For a failure probability , Hoeffding's inequality and a union bound give
Let minimize population loss within . I retain every model whose empirical validation loss is within of the empirical minimum, which defines an empirical Rashomon set:
On the simultaneous concentration event, , so and every retained model has population excess loss at most . The construction follows Marx et al.; the finite-class radius and the excess-loss bound are the two-line derivation above.
On the evidence side, represents the questions still unresolved under the current diligence state : entity boundaries, reporting periods, physical units, measurement definitions, shared third-party sources and suspected greenwashing claims. Each specifies a joint input configuration for all candidate companies at once, and that is how error dependence across companies and fields is preserved. Over the joint space , the exact worst-case bounds on the score difference between companies and are
If , the ordering holds for every retained model in and every admissible resolution of the evidence in . Suppose calibration establishes that contains the true inputs with probability at least at the specified certification stage. A union bound then gives joint inclusion of and the true inputs with probability at least , with no independence required between model validation and evidence calibration. The probability of issuing a certificate that is false for at the true inputs is at most . The object certified is the score ordering under the best model of the prespecified class at the true inputs, and for certification repeated over several stages is calibrated jointly across the stages.
One event certifies every pair. The guarantee is a statement about a single inclusion event, so on that event every pair with a positive lower bound is ordered correctly at once. Nothing is paid for the number of comparisons. Three consequences follow.
• The output of diligence is a certified partial order over the companies, and verification can only extend it: resolving evidence shrinks , so every lower bound stays where it is or rises.
• Progress has a measure, the share of company pairs that are certified, and a diligence plan can be judged by how far it moves that share per analyst hour.
• Selecting the top of companies needs only the certificates that put each selected company above each rejected one. For five of twenty that is 75 certificates in place of all 190 pairs.
Three neighboring constructions come close and stop short of this. Robust ordinal regression reports the "necessary" preferences that hold for every value function compatible with a decision maker's stated judgments, and shows that the necessary relation grows with information (Greco et al.); its uncertainty is entirely on the model side and the inputs are taken as known. Certain predictions hold for every classifier trained on every possible completion of an incomplete data set, and the companion cleaning rule chooses which row a person should fix next (Karlaš et al.); its possible worlds are independent across rows, so an error shared by two units cannot cancel, and its selection rule counts worlds, which amounts to a uniform prior over them. Confidence sets for ranks give simultaneous statistical statements about league tables (Mogstad et al.); they pay for simultaneity and treat each unit's estimate as a noisy draw, with no account of unverified inputs. The joint set carries model multiplicity and cross-company evidence error together, without a prior.
3. Computing comparison bounds with support functions
Convex support functions make the joint comparison bounds tractable (Ben-Tal and Nemirovski). Let collect the unresolved factual quantities, including common benchmark errors and company-specific allowances for reporting error, and write company inputs as . For an affine scoring model, the pairwise difference is , where , and the support function gives exact comparison bounds:
For the ellipsoidal uncertainty set , the Cauchy-Schwarz inequality gives . For an box centered at zero with half-widths , . I call the swing of field : the most the unresolved state of that field can move this comparison.
Keeping everything in one joint representation lets shared errors cancel exactly in a pairwise comparison. Let index common-source coordinates on which the two companies have identical loadings, so that . Under the same linear scoring model,
The common-source contribution cancels, and company-specific uncertainty stays in the support-function bound. If every uncertain direction cancels, and . Subtracting intervals constructed separately for companies and loses this, because it lets a shared source move in opposite directions at the same time. That combination is incompatible with the joint error structure, and it produces an unnecessarily wide difference bound.
How many misstatements a ranking survives. The box lets every unresolved field take its worst value at once. A committee is usually asking something narrower: how many things would have to be wrong together before this ranking fails? If at most fields are misstated simultaneously, the support function is the sum of the largest swings, the budgeted uncertainty set of Bertsimas and Sim. Ordering the swings as , the misstatement tolerance of the ordering is
the largest number of simultaneous worst-case misstatements the ranking survives. It turns a bound into a sentence: this ranking holds against any two unverified fields being wrong at once, and three could overturn it. It also sets a target for verification, since raising to the level a committee requires is a question about which swings to remove.
For a smooth nonlinear model, set and . If is twice continuously differentiable on and its Hessian has operator norm bounded by , Taylor's theorem gives
Nonsmooth tree ensembles get their bounds from leaf-region partitioning or mixed-integer linear programming (Kantchelian et al.). The implementation spends computation first on company pairs whose difference bounds are close to the zero decision boundary.
4. Selecting queries by the regret they can remove
When , the ranking is unresolved, since a factual correction or a resolution of ambiguous evidence could change the decision. The system then prioritizes the next verification query , such as reviewing a supplier contract, requesting original utility bills, or checking a regulatory filing.
Let be a finite set of institutional actions, such as advancing company or company to final diligence, or deferring a rating. Let be the prespecified utility of an action in scenario , where is nonempty and utilities are uniformly bounded. The baseline minimax regret over deterministic actions is
A query returns an observation and restricts the compatible scenarios to a nonempty subset . I score the query by its guaranteed reduction in worst-case decision regret:
Nonnegativity follows from : restricting the scenario set cannot increase worst-case regret for any fixed action, and minimizing over preserves that inequality. Calculating uses the set of compatible scenarios and requires no prior probability that a disclosure is exaggerated. Minimax regret is Savage's criterion, and choosing queries by it comes from utility elicitation (Boutilier et al.; Robust Active Preference Elicitation). Those methods elicit a decision maker's preferences. Here the queries are facts about companies, their answers move the inputs of every model in at once, and they cost analyst hours.
The exact plan for affine scores. For a choice between two companies under an affine score with interval evidence, the whole problem has a closed form. Write for the current margin, for the total swing, and for the swing of a set of fields. Verify the fields in , let the action be chosen again once their values are known, and take the worst values they could turn out to have. The regret that remains, and the guaranteed value of the set, are
The reason is that verification acts twice. It removes of swing, and it moves the margin by the revealed values, by as much as in the unfavorable direction. Until exceeds the two effects cancel in the worst case. Four results follow.
1. The pivot rule. A single field has guaranteed value only if , that is, only if that field alone could reverse the decision, and its value is then . Checking a field whose swing is below the margin guarantees nothing by itself, however large its attribution.
2. Complementarity, exactly. Two fields that are each worthless alone are worth together as soon as their combined swing passes the margin. A rule that scores one query at a time ranks them last; the closed form shows what such a rule gives up.
3. The plan is a knapsack. With verification costs and a budget of analyst hours, the plan that leaves the least regret maximizes verified swing within the budget. That is a knapsack on the swings, and ordering fields by solves its relaxation.
4. Shared sources are worth nothing to a pairwise choice. A common-source field on which the two companies load identically has and zero swing. This is the cancellation of Section 3 read as scheduling advice: when choosing between two companies, analyst hours spent on the benchmark they share cannot move the choice.
Across the model set the same expressions apply with the least favorable model for each action.
Verification narrows an allowance and seldom removes it. In a field experiment on third-party environmental audits, auditors chosen and paid by the audited plants systematically reported readings just below the regulatory standard (Duflo et al.). I therefore let a verified field keep a fraction of its half-width, set by the standing of the source: self-reported, limited assurance, reasonable assurance or primary document. The swing a check removes is , and that is the quantity that enters the knapsack.
With two-point evidence in place of intervals the worst observation cannot land in the middle of the range, so a check guarantees more; the third example of Section 6 is of that kind. The interval case is the conservative one. And because a verification rule the scored companies can anticipate invites embellishment where checks are unlikely (Lyon and Maxwell), the plan is held inside the diligence team and combined with a share of checks drawn at random.
5. Propagating factual corrections through scores and attributions
When diligence resolves a query and the verified evidence vector changes from to , the system follows versioned dependencies to recompute the affected scores, peer rankings and SHAP explanations. With the scoring function , background reference and rubric version held fixed, Shapley efficiency gives the exact identity (SHAP local accuracy):
Approximate Shapley outputs meet this up to a residual. Suppose their efficiency residuals are and under the same baseline . The triangle inequality then gives the recomputation check , which every update has to pass before its explanations reach the committee.
Every database update records (field version, evidence document ID, model version, background Q, rubric version). With these identifiers, a change caused by a factual correction is told apart from one caused by a model update or a change to the background distribution.
6. Worked examples
Five worked examples fix the size of each effect under stated inputs.
1. Confidence interval coverage under common reporting bias.
• Set the shared bias to and independent noise to . If the common bias is ignored, a nominal 95% interval has actual coverage of 48.40% at records and 0.000637% at . Once targeted verification shows , the half-width gives 97.50% coverage at , and the bias-aware half-width of Section 1 is against : 8.0% shorter at exactly 95%. The threshold is records. Past it, verification is the better use of the next analyst hour.
Coverage of a nominal 95% interval under a shared reporting bias
2. Cancellation of shared errors in pairwise rankings.
• Two companies have a nominal score difference of points and share an additive sector-benchmark error in . Subtracting separately constructed company intervals gives , which cannot certify the ordering, whereas the joint support-function calculation cancels the shared error and gives , which certifies the ranking.
One shared benchmark error, two ways to bound the difference
3. Minimax-regret verification compared with SHAP magnitude.
• Consider a single query, with the same cost for each query option. The example has eight scenarios, which come from two candidate models in the Rashomon set () and four unresolved evidence configurations (, ). The pairwise score difference is . A further shared feature has a large SHAP attribution but is already verified (). The action is to select company or company , and regret is measured in model-score units.
Table 1. Guaranteed regret reduction of each verification query.
| Verification query | Remaining regret under the worst observation | Guaranteed regret reduction |
|---|---|---|
| Recheck the verified field with a large SHAP attribution | 0.61 | 0 |
| Verify the smaller unresolved variable | 0.61 | 0 |
| Verify , which determines the sign of the score difference | 0 | 0.61 |
Enumerating the eight scenarios shows that rechecking the verified field with a large SHAP attribution produces no regret reduction (), while verifying takes worst-case regret from 0.61 to zero.
Eight scenarios for the score difference
Before verification, either choice can leave a regret of 0.61.
Remaining regret under the worst observation: 0.61; guaranteed reduction = 0.
Remaining regret under the worst observation: 0.61; guaranteed reduction = 0.
Remaining regret under the worst observation: 0; guaranteed reduction = 0.61.
4. A verification plan under an analyst budget.
• One company leads another by score points. Five fields are unresolved, with swings and verification costs of analyst hours. The total swing is , so the ranking is open and the worst-case regret is . Only the first field passes the pivot rule, with guaranteed value ; each of the others alone guarantees nothing, and the second and third together are worth . With four hours available:
Table 2. Two verification plans under a budget of four analyst hours.
| Plan | Fields verified | Hours | Regret remaining |
|---|---|---|---|
| Largest swing first | 1 | 4 | 1.4 |
| Swing per hour (the knapsack plan) | 2, 3, 4 | 4 | 1.1 |
The knapsack plan is the optimum over all plans within the budget, and it leaves 21% less regret than checking the largest field.
5. Misstatement tolerance.
• A ranking has margin and six unresolved fields with swings . The box bound does not certify it, since the swings sum to . The two largest sum to and the three largest to , so : the ranking holds against any two of the six fields being misstated at once, and that sentence is what the committee receives.
In the implementation with the asset manager, institutional diligence requirements became measures of score reliability, pairwise ranking stability and verification priority. The workflow ingests evidence, scores companies across the near-optimal model set, computes the certified partial order and its tolerances, and allocates verification by the plan of Section 4. Each correction triggers rescoring and the recomputation check of Section 5, so the investment committee has a traceable basis for every conclusion it is shown.