Evidence uncertainty and decision guarantees in investment research

Project 4: AI-Driven Investment Research and Explainable Risk Analysis
Author: Theodore Ouyang

An exact explanation of a prediction can still support an incorrect comparison between companies. Feature attribution describes how a model uses its current inputs. Investment research must also establish how those inputs might change after verification, which other models fit the validation data, and whether either source of uncertainty could reverse the conclusion. In the project, I carried evidence corrections through to scores, predictions, rankings, and explanations. The analysis below develops that practice into a decision protocol: represent unresolved evidence jointly, identify comparisons that remain valid across the resulting set, and allocate verification effort according to the decision regret it can remove.

1 Where existing methods leave the decision unresolved

SHAP's local accuracy property decomposes a given prediction into feature contributions. It imposes no requirement that the input facts be correct. Conditional Shapley methods address dependence among features, while sets of explanations across near-optimal models address uncertainty in model selection. These methods establish properties of attribution and its reliability. A further step is needed to determine which correction to the evidence could change the current research decision. See SHAP, Aas et al., and Marx et al..

For a linear model under a fixed marginal-background definition, the attribution to feature $j$ is $\phi_j=\beta_j(x_j-\mathbb E_QX_j)$. If its unresolved input error satisfies $|\delta_j|\leq d_j$, the corresponding sensitivity to correction is $|\beta_j|d_j$. Over a box of admissible errors, the exact worst-case change in the prediction is

The triangle inequality gives the upper bound. Choosing the endpoint of each error interval so that all contributions move the prediction in the same direction attains it. A feature with a large attribution may already have been thoroughly verified, leaving a small $d_j$. Conversely, a feature with an attribution close to zero may retain substantial unresolved error. Ranking verification work by absolute SHAP values omits this connection between uncertainty and the consequences of a correction.

Resampling the current records can also miss a common measurement bias. Suppose reported observations satisfy $X_i=\mu+b+\varepsilon_i$, where the $\varepsilon_i$ are independent $N(0,\sigma^2)$ errors and $b\neq0$ is an unmodeled bias shared by all records. A conventional normal interval centered on the sample mean covers the true $\mu$ with probability

Here $z=\Phi^{-1}(0.975)$. A larger sample reduces random error without removing the shared bias. A bootstrap that only resamples the same biased records has no additional information with which to identify $b$. Verification must therefore inform the description of input uncertainty before a stability analysis can address the relevant source of error.

2 Joint uncertainty over models and evidence

Fix the prediction target, input definitions, and output scale. To give the model set a statistical basis, consider $M$ candidate models specified before examining an independent validation set. Assume independent and identically distributed validation observations and loss bounded in $[0,1]$. For failure probability $\alpha$, Hoeffding's inequality and a union bound give

Let $f^*$ minimize population loss within this candidate set, and retain

On the simultaneous concentration event, $\widehat L_n(f^*)\leq L(f^*)+t_n\leq\min_g\widehat L_n(g)+2t_n$, so $f^*\in\mathcal F_\alpha$. The same bound shows that every retained model has population excess loss at most $4t_n$. This gives an explicit criterion for retaining multiple models. It is a finite-candidate construction based on uniform convergence; related work on sets of explanations appears in Marx et al.. Repeatedly adapting models to the same validation set requires fresh independent evaluation or a bound that accounts for the selection process.

The evidence set $\mathcal X(E)$ records unresolved questions about entity identity, reporting period, units, measurement definitions, and shared sources. Each element specifies a joint input configuration for all relevant companies. Dependencies are retained rather than represented as unrelated error bars for individual fields. Define $\Omega(E)=\mathcal F_\alpha\times\mathcal X(E)$. For companies $i,k$, compute

For a nonempty set with bounded differences, $\underline\Delta_{ik}>0$ certifies a strict ordering throughout the stated set. If calibration also establishes that the evidence set contains the true inputs with probability at least $1-\beta$, a union bound gives joint inclusion of the candidate-optimal model and the true inputs with probability at least $1-\alpha-\beta$. The unconditional probability of issuing an incorrect ordering certificate is therefore at most $\alpha+\beta$, without assuming that the two coverage events are independent. A scenario set specified solely by expert judgment supports a guarantee within that set; statistical coverage requires its own justification. The certificate concerns the defined score or prediction comparison, not future investment returns.

3 Computing comparison bounds with support functions

The joint evidence model also determines whether the calculation is tractable. Let $u\in\mathcal U$ collect shared unresolved quantities, with company inputs $x_i(u)=\bar x_i+B_i u$. Under affine predictions, the pairwise difference has the form $\Delta_f(u)=m_f+a_f^\top u$. With support function $h_{\mathcal U}(a)=\sup_{u\in\mathcal U}a^\top u$,

For $\mathcal U=\{Cz:\|z\|_2\leq r\}$, the Cauchy-Schwarz inequality, with equality attained in the direction of $C^\top a_f$, yields $h_{\mathcal U}(a_f)=r\|C^\top a_f\|_2$. When $\mathcal U$ is a box centered at zero, the support function is a weighted sum of absolute values; a nonzero center adds its inner product with the direction. These expressions replace a class of continuous scenario searches with direct calculation or convex optimization. The underlying tools are established in robust optimization. See Ben-Tal and Nemirovski.

Joint modeling can also remove unnecessary conservatism. If two companies evaluated by the same linear model share an identical additive source error, then $B_i=B_k$ and $a_f=(B_i-B_k)^\top\beta_f=0$. The common error cancels exactly in the comparison. Constructing separate intervals for the companies and subtracting them would allow the same source to vary in opposite directions, introducing a worst case that cannot occur under the stated evidence model.

For a nonlinear model, the approximation error must be bounded. Set $m_f=\Delta_f(0)$ and $a_f=\nabla\Delta_f(0)$. If the pairwise difference is twice continuously differentiable on $\|u\|_2\leq r$, with Hessian operator norm bounded by $K_f$, Taylor's remainder gives

A local linearization certifies the ordering only if the lower bound remains positive after the remainder is deducted. Tree models do not satisfy this smoothness condition; they require bounds based on leaf regions, enumeration, or a suitable global optimization method. An implementation can first use inexpensive bounds to resolve stable comparisons, then concentrate computation on company pairs close to the zero boundary.

4 Choosing verification queries by the regret they can remove

Stability intervals identify comparisons that need more evidence. Resource allocation also requires an account of how that evidence could change the decision. Let $\mathcal A$ be a finite action set, such as advancing a company to the next diligence stage, retaining a rating, or deferring judgment. For each scenario $\omega\in\Omega$, define action utilities $U(a,\omega)$ in advance on a common scale. The minimax regret over deterministic actions is

Hold the action set and utility definition fixed. A verification query $q$ yields a possible observation $o$, restricting the scenarios to a nonempty subset $\Omega_{q,o}\subseteq\Omega$. Every original scenario must remain compatible with at least one possible observation. Define the guaranteed reduction in regret as

The inequality follows from set inclusion. For any fixed action, worst-case regret cannot increase when the scenario set shrinks; minimizing over actions preserves the inequality. The definition requires no probability distribution over possible query outcomes. If a query can return no useful information, that outcome belongs in its outcome set. Noisy evidence must likewise retain scenarios compatible with the error mechanism. A valid reduction in uncertainty depends on what the observation actually establishes.

With several queries and a budget, the objective can be written as minimizing the remaining regret under the worst observation sequence, subject to total cost at most $B$ along every execution path. Selecting queries sequentially by $V(q)/c_q$ is a computable candidate policy. Complementarity between queries can make it suboptimal, so it should be compared with small batches of jointly selected queries or tractable discrete optimization. Regret reduction has an established role in information acquisition; this design applies it to evidence claims that determine company inputs and comparisons. See Robust Active Preference Elicitation.

Verification priority then depends on which unresolved claim can improve the current decision within budget. A shared source affecting several companies may be especially consequential: one query can narrow uncertainty in multiple comparisons at once.

5 Checking that corrections propagate through attribution

After deciding what to verify and obtaining a correction, the analysis must establish that the change has propagated through the outputs. Hold the prediction function $f$, background $Q$, attribution definition, and output scale fixed. Shapley efficiency gives

The second identity follows by subtracting the decompositions before and after correction. If approximate attributions have summation errors bounded by $\eta,\eta'$, respectively, the difference check permits a discrepancy of at most $\eta+\eta'$. This bound can serve directly as an acceptance criterion for recomputation. See SHAP's local accuracy property.

Each correction therefore carries versions for the affected fields, sources, model, background, and rules. Dependencies identify which scores, predictions, rankings, and explanations need to be recomputed. A change to the model or background requires a separate version comparison, so its effect is not attributed to the evidence correction. Dependency records establish the scope of recomputation; the identity checks the resulting values. Both are needed to verify a batch update.

6 Reproducible quantitative comparisons

The following analytical experiments use public, specified inputs to test the mechanisms derived above. These are constructed experiments, not client performance measurements.

The first experiment sets $b=0.2,\sigma=1$. Ignoring the common bias, a nominal 95% interval has actual coverage of 48.40% at $n=100$, falling to approximately 0.000637% at $n=1000$. If verification establishes $|b|\leq0.2$, increasing the half-width to $0.2+1.96/\sqrt n$ gives approximately 97.50% coverage in this example. Accounting for the shared evidence error restores coverage at the cost of a wider interval.

The second experiment examines a shared source. Two companies have a nominal difference of 0.6 and share the same additive source error in $[-1,1]$. Subtracting independently constructed intervals gives $[-1.4,2.6]$, which cannot certify an ordering. The joint source model gives $[0.6,0.6]$, certifying the comparison. The tighter result removes incompatible combinations of errors; it does not require a smaller error range for the source itself.

The third experiment allows one query of equal cost across eight scenarios formed from two candidate models and four input configurations. The model differences are $\Delta_s=0.15s+z_1+0.3sz_2$, where $s\in\{-1,1\}$, $z_1\in\{-0.4,0.4\}$, and $z_2\in\{-0.2,0.2\}$. A further shared field has a large attribution but has already been verified. The action is to select company $i$ or $k$, with regret measured in model-score differences.

Enumerating all eight scenarios produces these results. Attribution ranking and decision-based verification can select different queries. In this simple separable example, a reasonable interval-sensitivity baseline also selects $z_1$. Whether general regret optimization justifies its computational cost must be tested with shared sources, multiple comparisons, and sequential budgets. A production comparison should therefore include random verification, attribution ranking, interval sensitivity, and joint decision policies, counting optimization, retrieval, and expert judgment in the total cost.

Adoption should depend on the consequences for research: whether serious judgment errors, remaining regret, and error-correction rates improve under the same total budget; whether every affected conclusion is recomputed when evidence changes; and whether the advantage persists across defensible model sets and evidence ranges. The proposed protocol brings input validity, model uncertainty, and comparison stability into one computable decision model for allocating subsequent verification work.