Research
The engineering behind ML AI Code.
A coding agent is only as reliable as the system around it. This is how ours decides what to read, what it may change and what counts as done, and what we have measured so far.
Development evaluations on selected public tasks, graded outside the agent. Not an official leaderboard score. Method and limits.
The idea
Reliability is a systems problem.
Most failed tasks we have studied were not failures of raw ability. The agent lost the thread of a long task, edited outside what was asked, or said it was finished without proof. Each of those has an engineering answer, so we build the answer into the runtime rather than hope for it.
Six systems sit between the request and the result. Every one is ordinary, inspectable code with its own tests.
Six systems
What happens between your request and the result.
Task contract
Before the first edit, the request becomes a written contract: what to deliver, which files are in scope and which checks must pass. Progress is derived from that contract, and a task cannot finish while a deliverable is open.
Context management
Long tasks outgrow any working memory. Older tool output is summarised at a fixed threshold while the original request, every correction and the most recent steps stay word for word. Nothing is deleted: the full history can be read back.
Guarded edits
Every change is checked as it is written. An edit that breaks a file which parsed a moment ago is refused, a checkpoint is taken before each step, and parallel helpers work in their own copies so nothing collides.
Independent verification
"Done" is treated as a claim. The run records which commands ran, how many tests they covered and whether the code changed afterwards. Claims without matching evidence get one correction round, then are reported as unproven.
Safety policy
Each command is classified before it runs. A short list is always refused, a repository file sets the rules for that project, and a team can only tighten those rules, never loosen them.
Continuity
Work survives interruptions. Actions with side effects are journaled before they start, decisions and open questions carry to the next run, and specialists can be resumed with what they already found.
Verification
Done is a claim. We check it.
The agent that wrote the change is not the one that decides it works. A separate gate reads the record of the run and refuses the common ways a result can look finished without being finished.
- No tests actually ran. A command that collects zero tests is not a pass.
- A failure hidden in a passing summary. Every failed case fails the check.
- Code changed after the check. Evidence older than the latest edit is stale.
- A claim with nothing behind it. Each deliverable needs a passing check, a cited source or a real file.
The gate checks records and provenance. It does not judge whether your requirements were the right ones.
Context
Keep the thread on long tasks.
Hours into a task, most of what the agent has seen is old command output. We compact that selectively instead of truncating it.
The first request and every correction you make are pinned. If a summary would drop protected content, the run stops safely instead of continuing on a weaker memory.
Safety
A boundary the agent cannot talk its way past.
Rules are enforced by the runtime, not by instructions in a prompt. Each project runs on its own isolated machine, and every command passes a classifier before it executes.
# .mlai/policy.yml (reviewed like code)
production:
require_pr: true
require_ci_green: true
block_paths_without_review: ["payments/**"]
agent:
commands:
allow: ["npm test", "pytest*"]
deny: ["terraform apply*"]
secrets:
scan: trueAlways refused for the agent, whatever a project says: piping downloads into a shell inside a workspace, publishing packages, reading SSH keys, force pushes and production deploys outside the deploy review.
Results
Measured on real repositories.
Seventy tasks from two public benchmark families: reported issues in widely used open-source libraries, and a harder set of multi-file changes. We report first attempts and later repairs as separate numbers, because they answer different questions.
How a result is counted
- Tasks are chosen and committed before any run. No replacements after seeing outcomes.
- The reference fix is never shown to the agent. Reference tests arrive only after it has stopped and lost network access.
- Grading runs on a clean copy with only the agent’s patch applied.
- Accepted means every required test passes and nothing that passed before now fails.
- Setup failures and missing labels stay in the denominator.
How to read it
First-attempt numbers span several versions of the system as it improved, so they are not one fixed-version score. Some later repairs were given feedback from grading. That shows the system can reach a correct fix, not that it will on an unseen task.
Time to a patch, latest issue batch
Agent working time for each first attempt, in seconds. Excludes machine setup and grading.
Failures drive the roadmap
What measured failures changed.
- Long command output crowded out the task.Slow or verbose output is now saved outside the repository and read back in bounded pieces, without rerunning the command.
- Reviewers judged a summary, not the request.Read-only reviewers now receive the full original task and the current diff, and inspect the real code.
- A cut-off reply could be half a command.Truncated commands are never executed. Empty replies trigger one bounded recovery, then an explicit failure.
- The completion gate was too strict.Replaying 18 retained runs showed it blocking correct patches. It stays experimental until it stops doing that.
Not proven yet
What this evidence does not show.
- An official benchmark score. These are development runs on selected tasks.
- How we compare with other coding agents. We have not run a matched comparison and make no parity claim.
- Performance on tasks the system has never seen. A held-out set is next.
- That task-specific guidance makes runs faster. One matched pair finished about twice as fast with it; one pair is not a result.
- Live measurements for memory, repository understanding, durable continuation and visual review. They are built and unit tested; benchmarks are pending.
Standing on open work
Ideas we studied and built on.
Several of these systems follow designs published by open-source projects. We name them because the ideas are theirs; the implementations and results here are ours.
Editing interface
Refusing edits that leave a file broken, so errors are caught at the keystroke.
SWE-agent↗Repository map
Ranking files by how they depend on each other and fitting the map into a budget.
Aider↗Skills on demand
A short catalog up front, with full instructions loaded only when a task needs them.
Agent Skills↗Step checkpoints
A snapshot after each tool use, kept apart from your own history.
Cline↗Language-server feedback
Feeding diagnostics from the language server back after edits.
OpenCode↗Read less about it. Run a task.
Every run shows its plan, its commands and its evidence.