HOME · RESEARCH · Safety classification · Two early single-seed runs

Research note · 2026-06

Making optimization work when labels are scarce

With about 200 verified labels, including roughly eight harmful examples, Gnosys improved safety-classifier recall on two early ToxicChat runs. Both used GEPA for prompt optimization. Gnosys added reference scoring and calibrated feedback to guide that optimizer.

The result is promising, but it is a benchmark comparison—not a demonstrated production operating point.

Results

System Headline run: 3,000 messages Earlier run: 1,000 messages
Starting classifier 73.1% recall 78.8% recall
Direct GEPA optimization 70.2% recall 84.8% recall
Gnosys 77.7% recall 90.9% recall

In the headline run, Gnosys gained 4.6 percentage points over the starting classifier. Direct GEPA optimization fell below the starting point in this run and improved on it in the earlier run. These are two single-seed results, not a replicated estimate of expected improvement.

How the false-positive budget was measured

For each method, evaluation set its threshold at the 95th percentile of scores on test-set negatives, then flagged scores strictly above that threshold. This applies the same nominal 5% false-positive budget. It does not produce identical actual false-positive rates because many scores are tied.

Recomputing the headline run from its saved scores gives:

System Recall Actual false-positive rate
Starting classifier 73.11% 1.70%
Direct GEPA optimization 70.17% 2.32%
Gnosys 77.73% 3.91%

Gnosys caught more harmful messages and also flagged a larger share of safe messages. These numbers do not establish higher recall at an identical realized false-positive rate, equal review cost, or a threshold that could have been set using only the sparse calibration data.

The test labels were used to calculate these benchmark thresholds and scores. A production-oriented evaluation would set the threshold using separate calibration data, freeze it, then measure recall and false positives on untouched test data.

Download the recomputed headline metrics and source hashes.

What the system changed

The direct arm optimizes against sparse labels. Gnosys combines verified labels with model-based reference scores to estimate a feedback signal for the optimizer. The aim is to make better use of limited human judgment.

Both arms use GEPA, but the Gnosys pipeline includes additional reference work and a different number of optimization examples. This comparison does not isolate calibration alone or establish an advantage at equal total cost.

The win is specific to the reported operating-point metric. On the headline run, precision-recall AUC was 0.765 for the starting classifier, 0.721 for direct GEPA, and 0.692 for Gnosys. The study does not show better overall ranking quality.

Where it regressed

These differences compare Gnosys with direct GEPA at each method’s test-derived threshold. They use full test labels, not estimates a sparse-label customer dashboard would have.

Message length Harmful examples Recall difference
Short, under about 80 characters 81 −18.5 points
Medium 51 +21.6 points
Long or multi-step, 200+ characters 106 +20.8 points
Content category Harmful examples Recall difference
Violence-related 21 +23.8 points
Verified jailbreak attempts 49 +8.2 points
Sexual content 63 −7.9 points

An aggregate gain can hide a failure that matters to a customer. Short-message recall fell substantially, and sexual-content ranking also worsened. These losses must inform a deployment decision. Hate-speech and coding-related slices had only three and six harmful examples, respectively, so we omit their estimates.

What this establishes

This study supports further testing of feedback built from sparse labels. It does not establish reliable gains across tasks, equal-cost superiority, or production performance. Applying the method to fraud, support, or another domain requires its own evaluation.

Calibration and test sets were disjoint, with exact duplicate messages removed across splits. The threshold procedure nevertheless used test labels, as described above. More seeds, uncertainty estimates, a fixed-threshold evaluation, and full resource accounting are needed for stronger claims.

Discuss a pilot to test whether the approach is useful for your system.

Bring us a problem worth investigating.

We’re seeking design partners with a difficult AI workflow and a clear outcome to improve. Let’s discuss a focused research pilot.

We’d like to use one cookie to count unique visits. Privacy