Research note · 2026-09-19
Choosing the result that holds up
A research agent can improve its validation score while moving away from a solution that works on new data. In one causal-inference search, Gnosys's additional evaluation changed which candidate it selected. A retrospective comparison found 5.45% lower held-out test error for that selection than for the candidate with the best validation score.
This is one historical search run, not a replicated comparison of research systems. It illustrates why the choice of feedback deserves its own experiment.
The question
When a search produces several plausible solutions, which one should it return? The simplest answer is the candidate with the best validation score. But a score repeatedly used to guide search can become a target in its own right.
For the FML-Bench causal-inference task, Gnosys evaluated candidate changes with both the normal validation procedure and an auxiliary evaluation. Its selection rule favored candidates that performed well under both checks.
What we compared
After the run, we compared the validation-only choice with Gnosys's selected candidate from the same search. Lower error is better in every column.
| Candidate selection | Validation error | Auxiliary error | Held-out test error |
|---|---|---|---|
| Best validation score | 1.1466 | 1.2430 | 1.3026 |
| Gnosys selection | 1.1760 | 1.1693 | 1.2316 |
The validation-only choice looked better on the metric it optimized. The auxiliary evaluation preferred the other candidate, and the held-out test agreed with that ordering.
The relative reduction is (1.302612 − 1.231630) / 1.302612, approximately 5.45%. It is a reduction in error, not a percentage-point gain in accuracy.
Why this matters
Teams need a reliable way to choose among the changes a research process generates. Producing a candidate with an attractive development score is only part of that job.
This example suggests that additional evaluation can change a selection in a useful direction. It also provides a concrete question for the next experiment: how often does that happen, and is the benefit worth the extra evaluation cost?
What this result does not establish
- This was a retrospective comparison within one run. It does not establish a repeatable advantage across tasks or independent search seeds.
- The auxiliary evaluation informed selection. It was development feedback, not untouched confirmation evidence.
- This comparison concerns auxiliary evaluation and selection. It does not isolate MCGrad, prove unbiasedness, or establish autonomous improvement of the research method itself.
- It is not a matched-cost efficiency study. Additional checks consume resources, and their benefit must be assessed against that total cost.
- This is a separate investigation from the earlier causal-inference numbers in our September 18 draft. The two must not be counted as independent replications.
Evidence and next test
The experiment was recorded on September 19, 2026. The downloadable readout preserves the unrounded values and the digest of the retained source artifact used for this article. It is a compact numerical record, not a complete reproduction package.
Download the numerical readout
The next test is a prospective comparison with frozen selection rules, multiple runs, and independent confirmation. That will help determine when extra measurement improves the final result and when a simpler approach is sufficient.