Research note · 2026-06
Making optimization work when labels are scarce
With about 200 verified labels, including roughly eight harmful examples, Gnosys improved safety-classifier recall on two early ToxicChat runs. Both used GEPA for prompt optimization. Gnosys added reference scoring and calibrated feedback to guide that optimizer.
The result is promising, but it is a benchmark comparison—not a demonstrated production operating point.
Results
| System | Headline run: 3,000 messages | Earlier run: 1,000 messages |
|---|---|---|
| Starting classifier | 73.1% recall | 78.8% recall |
| Direct GEPA optimization | 70.2% recall | 84.8% recall |
| Gnosys | 77.7% recall | 90.9% recall |
In the headline run, Gnosys gained 4.6 percentage points over the starting classifier. Direct GEPA optimization fell below the starting point in this run and improved on it in the earlier run. These are two single-seed results, not a replicated estimate of expected improvement.
How the false-positive budget was measured
For each method, evaluation set its threshold at the 95th percentile of scores on test-set negatives, then flagged scores strictly above that threshold. This applies the same nominal 5% false-positive budget. It does not produce identical actual false-positive rates because many scores are tied.
Recomputing the headline run from its saved scores gives:
| System | Recall | Actual false-positive rate |
|---|---|---|
| Starting classifier | 73.11% | 1.70% |
| Direct GEPA optimization | 70.17% | 2.32% |
| Gnosys | 77.73% | 3.91% |
Gnosys caught more harmful messages and also flagged a larger share of safe messages. These numbers do not establish higher recall at an identical realized false-positive rate, equal review cost, or a threshold that could have been set using only the sparse calibration data.
The test labels were used to calculate these benchmark thresholds and scores. A production-oriented evaluation would set the threshold using separate calibration data, freeze it, then measure recall and false positives on untouched test data.
Download the recomputed headline metrics and source hashes.
What the system changed
The direct arm optimizes against sparse labels. Gnosys combines verified labels with model-based reference scores to estimate a feedback signal for the optimizer. The aim is to make better use of limited human judgment.
Both arms use GEPA, but the Gnosys pipeline includes additional reference work and a different number of optimization examples. This comparison does not isolate calibration alone or establish an advantage at equal total cost.
The win is specific to the reported operating-point metric. On the headline run, precision-recall AUC was 0.765 for the starting classifier, 0.721 for direct GEPA, and 0.692 for Gnosys. The study does not show better overall ranking quality.
Where it regressed
These differences compare Gnosys with direct GEPA at each method’s test-derived threshold. They use full test labels, not estimates a sparse-label customer dashboard would have.
| Message length | Harmful examples | Recall difference |
|---|---|---|
| Short, under about 80 characters | 81 | −18.5 points |
| Medium | 51 | +21.6 points |
| Long or multi-step, 200+ characters | 106 | +20.8 points |
| Content category | Harmful examples | Recall difference |
|---|---|---|
| Violence-related | 21 | +23.8 points |
| Verified jailbreak attempts | 49 | +8.2 points |
| Sexual content | 63 | −7.9 points |
An aggregate gain can hide a failure that matters to a customer. Short-message recall fell substantially, and sexual-content ranking also worsened. These losses must inform a deployment decision. Hate-speech and coding-related slices had only three and six harmful examples, respectively, so we omit their estimates.
What this establishes
This study supports further testing of feedback built from sparse labels. It does not establish reliable gains across tasks, equal-cost superiority, or production performance. Applying the method to fraud, support, or another domain requires its own evaluation.
Calibration and test sets were disjoint, with exact duplicate messages removed across splits. The threshold procedure nevertheless used test labels, as described above. More seeds, uncertainty estimates, a fixed-threshold evaluation, and full resource accounting are needed for stronger claims.
Discuss a pilot to test whether the approach is useful for your system.