BJBrian Jalaian
All news

Sep 2026 · AI for Science Workshop · NeurIPS 2026 · Sydney

Is Your Research Loop Measuring Noise?

Accept rules for autonomous experimentation

Brian Jalaian · Venkat R. Dasari (DEVCOM ARL)

Venkat Dasari and I will present this work as a poster at the NeurIPS 2026 AI for Science Workshop in Sydney on December 12. We are still finishing the camera-ready version, so this post is about the idea rather than the numbers.

AI agents that run their own experiments decide what to keep by comparing scores. When run-to-run noise is as large as the improvements being chased, the loop ends up keeping lucky runs. An accept rule makes that decision statistically valid.

Accept rules: an autonomous research loop proposes a change and runs it, then accepts it only if its gain is larger than the run-to-run noise measured from replicate runs; otherwise it rejects the change or keeps sampling.Agent proposes a changeRun the experimentCompare with the incumbentIs the gain bigger thanwhat noise alone produces?accept ruleMeasured noisereplicate runsyesAcceptnoReject or keep samplingtry again
The loop accepts a change only when its gain clears the noise measured from replicate runs of the current best configuration.

The problem

Autonomous experimentation, where an AI agent proposes a change, runs it, and keeps it if the score improves, is becoming a common way to search for better training recipes and models. But training runs are noisy: the same configuration with a different random seed gives a different score.

If the loop compares a single new run against the incumbent, many apparent improvements are just noise. Looking repeatedly, or keeping the best of several tries, makes it worse. The loop drifts, confidently, toward configurations that only looked good once.

The idea

Measure the noise before trusting a gain. A small number of replicate runs of the incumbent tells the loop how much scores move on their own. A change is accepted only when its gain is larger than what that noise alone would produce, judged by a test that stays statistically valid for how the loop actually works.

  • When the number of runs is fixed in advance, a standard fixed-sample test is enough.
  • When the loop decides on the fly how long to keep sampling, it needs a sequential test that remains valid no matter when it stops.

Why it matters

Accept rules are cheap: a few extra runs and a few lines of code. In return, an automated research loop stops chasing noise, and the results it reports can be trusted. That is a prerequisite for using AI agents to do science at all.