Method
How we tested coding agents
Provisional: one test project, tested 5 Nov 2026. Pack 0.3.0. 90 runs, five tools, six tasks, three fresh runs each, zero human interventions. Published FitScores: Claude Code 94, OpenCode 93, Codex CLI 90, Cline 88, Google Antigravity 79.
The other six candidates on the coding-agents page were not run. They have verified facts and no FitScore. The shared rules are on how we test.
The pack
Same TypeScript todo CLI for every tool. Each run started from a fresh copy of the task template inside bubblewrap, with a private network namespace. Traffic left only through a logging proxy. The agent could see only its own repo. Hidden tests, reference solutions, and the grader stayed outside the sandbox. The same prompt text was used for every tool. The spend cap was $2.00 of API cost per run. No rate or usage limit was hit.
Quality is scored on the output as delivered. The protocol allowed no human fixes, so the older "after N fixes" steps of the rubric did not apply. A pass with no unrelated change is 5. A pass with one harmless unrelated change is 4. Partially met is 2. A fail is 1. Task scores are then averaged and rounded to the nearest integer for the criterion.
| Task | What it asks | Limit | Hidden tests |
|---|---|---|---|
| T1h | todo done <id> completes the wrong todo after todo list. Two defects: list sorts the store in place, and lookup is positional. | 15 min | 6 |
| T2h | Tags, including a new store format and a migration of old files, from issues/005-tags.md. | 25 min | 9 |
| T3h | Refactor commands into a registry. Behaviour is checked by 50 hidden golden-output tests. | 25 min | 50 |
| T4 | docs/ARCHITECTURE.md, under 600 words, only true statements. No hidden tests. Eight facts, written before any doc was graded. | 15 min | — |
| T5 | Dates shift by one day in UTC+2. Hidden tests run under TZ=Africa/Johannesburg and TZ=UTC. | 25 min | per timezone |
| T6 | Headless run. The issue text is available only through the fixture's local stdio MCP server, fft-issues. | 20 min | 1 |
The hidden-test fix
One T3h check looked for a shared Command interface only in src/commands/ or cli.ts. The prompt did not require that location. Claude Code run 1 and OpenCode run 1 put the interface in src/types.ts and failed that check alone. The check now accepts any file under src/. The old check is kept. Every finished T3h run was re-graded. All of them scored 50/50. The untouched template still fails 47/50. The reference solution still passes 50/50. The same corrected check was applied to every tool.
The OpenCode run that was disqualified
Round 1, earlier the same day, ran easier tasks T1–T3. It is not this ranking. The recorded summary is: all clean passes except one OpenCode disqualification. OpenCode's T3 was 2 of 3 passes, quality 5, reliability 3.
The pack says why the isolation rule exists. One unsandboxed agent searched the disk, read the hidden tests, and ran the grader. That run is disqualified and counts as a reliability fail. Round 2 therefore runs every agent in bubblewrap. The agent can see only its own repo.
Round 2, task by task
Claude Code passed 18/18. Codex CLI passed 17/18. OpenCode passed 16/18. Median wall time is the middle of the three runs. Quality by run is the score for each run, in order. The per-run stdout, diffs, and egress logs are not hosted. The machine-readable summary of every run is coding-agents-round-2.json.
| Tool | Task | Passes | Median wall | Quality by run | Quality | Reliability | Speed |
|---|---|---|---|---|---|---|---|
| Codex CLI | T1h | 3/3 | 27.2 s | 5, 5, 5 | 5 | 5 | 5 |
| Codex CLI | T2h | 3/3 | 92.8 s | 5, 5, 5 | 5 | 5 | 5 |
| Codex CLI | T3h | 3/3 | 66.4 s | 5, 5, 5 | 5 | 5 | 5 |
| Codex CLI | T4 | 3/3 | 44.5 s | 5, 5, 5 | 5 | 5 | 5 |
| Codex CLI | T5 | 3/3 | 58.4 s | 5, 5, 5 | 5 | 5 | 5 |
| Codex CLI | T6 | 2/3 | 40.4 s | 1, 5, 5 | 4 | 3 | 5 |
| Claude Code | T1h | 3/3 | 40.6 s | 5, 5, 5 | 5 | 5 | 5 |
| Claude Code | T2h | 3/3 | 146.1 s | 5, 5, 5 | 5 | 5 | 5 |
| Claude Code | T3h | 3/3 | 106.7 s | 5, 5, 5 | 5 | 5 | 5 |
| Claude Code | T4 | 3/3 | 48.8 s | 5, 5, 5 | 5 | 5 | 5 |
| Claude Code | T5 | 3/3 | 71.3 s | 5, 5, 5 | 5 | 5 | 5 |
| Claude Code | T6 | 3/3 | 33.8 s | 5, 5, 5 | 5 | 5 | 5 |
| OpenCode | T1h | 2/3 | 13.9 s | 5, 2, 5 | 4 | 3 | 5 |
| OpenCode | T2h | 3/3 | 56.8 s | 5, 5, 5 | 5 | 5 | 5 |
| OpenCode | T3h | 3/3 | 30.5 s | 5, 5, 5 | 5 | 5 | 5 |
| OpenCode | T4 | 2/3 | 13.6 s | 2, 5, 5 | 4 | 3 | 5 |
| OpenCode | T5 | 3/3 | 27.5 s | 5, 5, 5 | 5 | 5 | 5 |
| OpenCode | T6 | 3/3 | 19.1 s | 5, 5, 5 | 5 | 5 | 5 |
OpenCode T1h run 2 fixed only one of the two defects. Hidden tests 4/6. Quality 2 for that run. OpenCode T4 run 1 stated the checklist facts and added one false claim about the code. Quality 2 for that run.
Codex T6 run 1 failed because the harness had not set Codex's documented per-server MCP approval key, mcp_servers.<name>.default_tools_approval_mode="approve". Under approval policy never, Codex cancels MCP calls unless that key is set. Runs 2 and 3, with the key set, passed. Run 1 stays a fail. Counting it as a pass would not change Codex's rounded quality, reliability, or the published FitScore.
Round 3, Cline and Google Antigravity
Same pack, same sandbox, same prompts, same limits. Cline CLI 3.0.74 passed 18/18. Google Antigravity CLI agy 1.5.1 passed 14/18. No rate or usage limit was hit. There were no human interventions during the runs. The machine-readable summary is coding-agents-round-3.json. Per-run stdout and diffs are not hosted.
Cline used gpt-6.1-sol with thinking set to low, through Cline's native ChatGPT Plus sign-in. Every Cline run reported that model. The choice was made to match Codex CLI. That match is inferred, not a Codex log entry: codex debug prompt-input says "based on GPT-6", and the model cache lists gpt-6.1-sol as priority 1 with default reasoning low. The wizard had saved GPT-5.3 Codex Spark at medium thinking. Antigravity used the CLI default, Gemini 4 Argon (Medium), on Google AI Pro.
The Cline series ran from 11:31 to 12:03 SAST and the Antigravity series from 11:32 to 12:30 SAST, at the same time on the same machine. Every median is still under a third of its limit, so speed stays 5. The absolute times may be slightly inflated.
| Tool | Task | Passes | Median wall | Quality by run | Quality | Reliability | Speed |
|---|---|---|---|---|---|---|---|
| Cline | T1h | 3/3 | 49.1 s | 5, 5, 5 | 5 | 5 | 5 |
| Cline | T2h | 3/3 | 123.3 s | 5, 5, 5 | 5 | 5 | 5 |
| Cline | T3h | 3/3 | 106.6 s | 5, 5, 5 | 5 | 5 | 5 |
| Cline | T4 | 3/3 | 57.0 s | 5, 5, 5 | 5 | 5 | 5 |
| Cline | T5 | 3/3 | 72.1 s | 5, 5, 5 | 5 | 5 | 5 |
| Cline | T6 | 3/3 | 63.3 s | 5, 5, 5 | 5 | 5 | 5 |
| Antigravity | T1h | 3/3 | 72.7 s | 5, 5, 5 | 5 | 5 | 5 |
| Antigravity | T2h | 3/3 | 200.4 s | 5, 5, 5 | 5 | 5 | 5 |
| Antigravity | T3h | 2/3 | 109.8 s | 1, 5, 5 | 4 | 3 | 5 |
| Antigravity | T4 | 0/3 | 90.5 s | 2, 2, 2 | 2 | 1 | 5 |
| Antigravity | T5 | 3/3 | 127.8 s | 5, 5, 5 | 5 | 5 | 5 |
| Antigravity | T6 | 3/3 | 82.4 s | 5, 5, 5 | 5 | 5 | 5 |
Judgement calls
Antigravity T4 is 0/3 under the same standard as OpenCode's false claim in round 2. Run 1 leaves out the requirement to add tests. Runs 2 and 3 state all eight facts, then claim that normalizeTitle validates inputs and throws during runCli dispatch. It does not throw, and runCli does not call it. That is a borderline call. Counting those two claims as over-broad rather than false would make quality 5, reliability 4, and the FitScore 85. The published score stays 79.
Two runs stalled on the model stream. Cline T3h run 1 stalled for about 7 minutes (11:37:41 to 11:44:31 SAST), then exited 1 after 513 s with "The operation timed out." The refactor was already complete, 50/50 hidden tests, so it counts as a pass. A script that checks the exit code would have reported a failure. Antigravity T3h run 1 made no further progress until the harness killed it at 1549.9 seconds, past the 1500 second limit. The code left behind passed 50/50, and a cap hit scores quality 1.
After the runs, the grader's "new test added" check was fixed so it counts it.each. It changed one run: Cline T5 run 2. Checked by hand, those tests fail on the buggy code in UTC and UTC+2, and the hidden tests pass 4/4 in both zones, so the run was overridden to a pass. Round 2 T5 results were not affected.
Other disclosed alternatives, not applied: counting Antigravity setup from the 10:42 install to the first successful run would make setup 2 and the FitScore 77. Treating play.googleapis.com as ordinary Google telemetry would make Antigravity privacy 4 and the FitScore 81. Treating registry.npmjs.org as an unexplained destination would make Cline privacy 3 and the FitScore 86.
Cline's task runs contacted chatgpt.com (5.56 MB uploaded) and registry.npmjs.org (0.03 MB). data.cline.bot appeared only in the first telemetry-on smoke run, about 2 KB, and in none of the 18 task runs, so the telemetry opt-out could not be confirmed either way. Antigravity's task runs contacted daily-cloudcode-pa.googleapis.com (14.35 MB), play.googleapis.com (5.30 MB, 37 connections, even with interaction data collection opted out; the host is not named in the docs), aicode.googleapis.com, www.googleapis.com, lh3.googleusercontent.com, the auto-updater on run.app in 2 of 18 runs, and oauth2.googleapis.com in 1 run.
How the five FitScores were scored
Weights: quality 30, reliability 15, cost 15, speed 10, setup 10, agent/API/MCP 10, privacy 10. FitScore = round(sum of score × weight / 5). Versions: Claude Code 2.1.318 on Claude Pro, model claude-opus-5-5. OpenCode 1.19.5, model opencode/big-pickle, no account, reported cost $0.00. Codex CLI 0.166.0 on ChatGPT Plus. Codex did not report a model name in codex exec --json. Cline CLI 3.0.74 on ChatGPT Plus, model gpt-6.1-sol, thinking low. Google Antigravity CLI 1.5.1 on Google AI Pro, default model Gemini 4 Argon (Medium).
| Tool | Quality | Reliability | Cost | Speed | Setup | Agent | Privacy | FitScore |
|---|---|---|---|---|---|---|---|---|
| Claude Code | 5 | 5 | 3 | 5 | 5 | 5 | 5 | 94 |
| OpenCode | 5 | 4 | 5 | 5 | 5 | 5 | 3 | 93 |
| Codex CLI | 5 | 5 | 3 | 5 | 4 | 5 | 4 | 90 |
| Cline | 5 | 5 | 3 | 5 | 3 | 5 | 4 | 88 |
| Antigravity | 4 | 4 | 3 | 5 | 4 | 5 | 3 | 79 |
Codex setup is 4, and the FitScore is 90. We count the tool's own setup time and exclude the human relay of the sign-in code. Install finished at 08:36:08 SAST. The device-code login started at 08:36:14. Auth was written at 08:47:05, which includes that relay. The tool's own steps are inside 10 minutes. An earlier write-up scored setup 3 and FitScore 88 because the clock included the relay. That is not the published score.
Claude Code's sign-in also needed a browser step, and that step did not block the headless runs. The published setup score stays 5. A stricter reading that any browser step is a 3 would make Claude Code's FitScore 90. We are not applying that reading.
Claude Code cost is 3 because the runs were covered by Claude Pro at $20 a month, not because of the API-equivalent figure the client showed ($8.31 across 18 runs, median $0.40, not billed). OpenCode privacy is 3 because the free model's terms say collected data may be used to improve the model, without naming the provider or the retention period.
Cline cost is 3 because ChatGPT Plus at $20 a month covered all 18 runs. The tool showed an API-equivalent figure of $1.41, which was not billed. Setup is 3 because sign-in is a browser step that blocks a headless machine. Privacy is 4: ChatGPT Plus content may be used for training by default, and there is an opt-out. Antigravity cost is 3 because Google AI Pro at $19.99 a month covered the runs and the CLI reports no dollar cost. Setup is 4: the installer took 3 seconds, one documented flag was rejected, and the tool's own sign-in took about 2 minutes 15 seconds. Privacy is 3 because play.googleapis.com is not named in the docs.
Left off the list
Google Jules is shutting down. Sessions until 11 Nov 2026, then read-only, wind-down 31 Dec 2026, folded into Google Antigravity. Jules changelog, 29 Oct 2026.
Windsurf was renamed Devin Desktop on 2 Jun 2026, so it is covered by the Devin candidate, which is not yet tested. Devin's announcement.