HOME · RESEARCH · Few-shot learning · One task, one seed

Research note · 2026-09-19

An early comparison of eight research agents

We compared eight research agents on one few-shot learning task in FML-Bench. Gnosys reached 78.45% test accuracy, compared with 79.21% for AIDE, the highest result in this run.

All eight used glm-5.3-flash with a 25-step search budget and seed 0. This was an exploratory comparison on one task, not a complete FML-Bench campaign or a study of performance at equal total cost.

The full comparison

The task was Data_Efficiency_easyfsl. The table reports test accuracy, with higher values indicating better performance.

Agent Test accuracy
AIDE 79.21%
AIRA-MCTS 78.92%
Autoresearch 78.72%
AI Scientist v2 78.59%
Gnosys 78.45%
OpenEvolve 78.07%
AdaptiveSearch 74.07%
The AI Scientist 70.12%

Gnosys finished fifth by test accuracy, 0.76 percentage points behind AIDE. A single observation cannot establish statistical equivalence between these systems.

What the timing record tells us

The historical report records 48 minutes for Gnosys and 116 minutes for AIDE, with 560,104 and 701,201 LLM tokens respectively. These observations motivated further investigation of research efficiency.

An October 5 audit found that the retained timing fields do not establish equivalent accounting boundaries across agents. The historical protocol also disclosed a shared machine. Full costs, auxiliary operations, reproduction, and final evaluation require consistent accounting before we can claim an efficiency advantage.

We therefore report those times as historical metadata, not verified speedup or dollar savings. An equal number of search steps is not necessarily an equal research budget.

A failed reproduction check belongs in the result

The retained report records that Gnosys's promotion re-execution failed because of a specification-validation error, which was fixed afterward. The recorded test score does not erase that failure. This run does not satisfy the stronger requirement that a selected result reproduce successfully before acceptance.

The agents also differed in successful execution counts: the retained summaries report 22 valid steps out of 25 for Gnosys and 25 out of 25 for AIDE. Neither execution count nor a shorter observed runtime establishes stronger research quality.

What we learned

Gnosys trailed AIDE on both its best validation score and selected test score. That makes candidate generation and productive use of the research budget important questions alongside final selection.

We are testing how exploration and verification should share that budget. The goal is better independently evaluated results for the total resources spent. A gain must survive fresh tasks and account for failed attempts as well as successful ones.

Scope and evidence

This comparison was recorded on September 19, 2026. Its task had already informed development. It establishes neither cross-domain method transfer nor autonomous recursive self-improvement, and it is not a comparison with RRSI.

Download the numerical readout

The readout transcribes the retained historical table and identifies its source digest. It is not a complete reproduction package. The timing and reproduction qualifications above are part of the result.

Bring us a problem worth investigating.

We’re seeking design partners with a difficult AI workflow and a clear outcome to improve. Let’s discuss a focused research pilot.

We’d like to use one cookie to count unique visits. Privacy