Which coding agent should I use to fix bugs and add features in my repo?

Provisional: one test project, tested 5 Nov 2026

Round 2, 5 Nov 2026, pack 0.3.0: 18 runs, 6 tasks, 3 runs each, zero human interventions. Claude Code passed 18/18. Claude Code 2.1.318 on Claude Pro. Model reported by the tool: claude-opus-5-5.

Output quality 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 5, rounded

Reliability 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 5, rounded

Cost and free tier 3/5

Claude Pro, $20/mo billed monthly ($17/mo annual), covered all 18 runs with no usage limit hit. The tool showed an API-equivalent figure of $8.31 total (median $0.40/run); not billed.

Speed 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 5, rounded

Setup friction 5/5

About 3 min from install to signed in (~/.claude created 08:47:19, credentials written 08:50:09 SAST); documented `claude` sign-in, worked first time.

Agent, API, and MCP support 5/5

Headless `claude -p` (documented). MCP server registered with `--mcp-config` + `--strict-mcp-config`; MCP tools pre-allowed with `--allowedTools`. T6 passed 3/3 with no prompt.

Privacy and data 5/5

Observed hosts: api.anthropic.com (model), http-intake.logs.us5.datadoghq.com (operational telemetry), mcp-proxy.anthropic.com (claude.ai connectors, on by default for claude.ai sign-ins). All three listed in Anthropic's network-config doc. With DISABLE_TELEMETRY etc. plus ENABLE_CLAUDEAI_MCP_SERVERS=false only api.anthropic.com was contacted (smoke runs). Consumer chats/code used for training only if you choose to allow it.

• cost
• agentSupport
• agentSupport
• privacy
• privacy

Provisional: one test project, tested 5 Nov 2026

Round 2, 5 Nov 2026, pack 0.3.0: 18 runs, 6 tasks, 3 runs each, zero human interventions. OpenCode passed 16/18. opencode 1.19.5, free model opencode/big-pickle, no account. Reported cost $0.00 on every run.

Output quality 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 4.667, rounded

Reliability 4/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 4.333, rounded

Cost and free tier 5/5

Free built-in model opencode/big-pickle, no account or key; $0.00 reported on all 18 runs, no limit hit.

Speed 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 5, rounded

Setup friction 5/5

8.6 s on a fresh HOME: npm install 5.5 s, first run 3.1 s, no account.

Agent, API, and MCP support 5/5

Headless `opencode run` (documented). MCP server registered as a `local` entry under `mcp` in opencode.json. T6 passed 3/3 with no prompt.

Privacy and data 3/5

Observed hosts: opencode.ai (model gateway) and once models.opencode.ai (model list; docs name models.dev and offer OPENCODE_DISABLE_MODELS_FETCH). Big Pickle terms: 'During its free period, collected data may be used to improve the model'; provider and retention not named, only opt-out is another model. Terms unclear about training = 3.

• T1h run 2 fixed only one of the two defects. Hidden tests 4/6. Quality 2 for that run.
• T4 run 1 stated the checklist facts but added one false claim about the code. Quality 2 for that run.

• cost
• agentSupport
• agentSupport
• Pricing

Provisional: one test project, tested 5 Nov 2026

Round 2, 5 Nov 2026, pack 0.3.0: 18 runs, 6 tasks, 3 runs each, zero human interventions. Codex passed 17/18. codex-cli 0.166.0 on ChatGPT Plus. The tool did not report a model name in codex exec --json. Setup is 4 and the FitScore is 90. We count the tool's own setup time and exclude the human relay of the sign-in code.

Output quality 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 4.833, rounded

Reliability 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 4.667, rounded

Cost and free tier 3/5

ChatGPT Plus, $20/month, billed monthly, covered all 18 runs with no usage limit hit. 1.88M input tokens (87% cached) and 30k output tokens across 18 runs.

Speed 5/5

round 2, mean of per-task scores over T1h,T2h,T3h,T4,T5,T6 = 5, rounded

Setup friction 4/5

Setup is 4. We count the tool's own setup time and exclude the human relay of the sign-in code. Install finished at 08:36:08 SAST and the device-code login started at 08:36:14. Auth was written at 08:47:05, which includes that relay. The tool's own steps are inside 10 minutes, so the rubric gives 4, not 3. FitScore is 90.

Agent, API, and MCP support 5/5

Headless `codex exec --json` (documented). MCP server registered with documented `-c mcp_servers.<name>.*` config keys. With approval policy `never`, MCP calls are cancelled unless the per-server `default_tools_approval_mode="approve"` key is set (documented). With it, T6 passed 2/2 with no prompt; run 1 without it failed.

Privacy and data 4/5

Observed hosts: chatgpt.com (model/backend) and ab.chatgpt.com (Statsig analytics, about 720 KB uploaded per run). `analytics.enabled=false` is documented and removed ab.chatgpt.com in our smoke run. Plus content (ChatGPT and Codex tasks) may be used to train models by default; opt out via Settings > Data controls or the Privacy Portal. Codex has a separate "Include environments" training setting.

• T6 run 1 failed: the harness had not set Codex's documented per-server MCP approval key. Counted as a fail. Runs 2 and 3, with that key set, passed.

• cost
• agentSupport
• agentSupport
• privacy
• privacy
• Pricing

Provisional: one test project, tested 5 Nov 2026

Round 3, 5 Nov 2026, pack 0.3.0: 18 runs, 6 tasks, 3 runs each, zero human interventions. Cline CLI 3.0.74 passed 18/18. Model: gpt-6.1-sol, thinking low, via ChatGPT Plus sign-in. That model was chosen to match Codex CLI in round 2. The match is inferred: Codex did not log its model. codex debug prompt-input says "based on GPT-6", and the model cache lists gpt-6.1-sol as priority 1 with default reasoning low. The Cline series (11:31 to 12:03 SAST) overlapped the Antigravity series on the same machine.

Output quality 5/5

Round 3, mean of per-task scores over T1h-T6 = 5, rounded

Reliability 5/5

Round 3, mean of per-task scores over T1h-T6 = 5, rounded

Cost and free tier 3/5

ChatGPT Plus, $20/month billed monthly, covered all 18 runs with no rate or usage limit hit. Cline showed an API-equivalent figure of $1.41 in total (gpt-6.1-sol at $2/M input and $10/M output, 1.77M input tokens with 1.34M cache reads, 39k output). This was not billed, because Plus is flat-rate.

Speed 5/5

Round 3, mean of per-task scores over T1h-T6 = 5, rounded

Setup friction 3/5

npm install of Cline CLI 3.0.74 was quick and needed no account. Sign-in to ChatGPT Plus uses the native 'Sign in with ChatGPT' (openai-codex) provider, which is a browser OAuth flow. Sarel had to finish it in a desktop browser, and the earlier non-interactive `cline auth` attempt was refused. A browser step that blocks a headless box = 3. The sign-in duration was not timed, because Sarel did it himself.

Agent, API, and MCP support 5/5

Headless `cline --json` (documented NDJSON mode). MCP servers can be added by a single documented command (`cline mcp add <name> --yes -- <cmd>`) or the documented `~/.cline/data/settings/cline_mcp_settings.json` file; our harness wrote that file. T6 passed 3/3 with no prompt; each run called the MCP tool `get_issue`.

Privacy and data 4/5

Hosts observed in the 18 task runs were chatgpt.com (model, 5.6 MB up) and registry.npmjs.org (one small request per run, consistent with an npm version check; not described in Cline's docs). data.cline.bot (Cline telemetry, documented, on by default in the CLI, opt-out `telemetryOptOut`) was seen only in the first smoke run. It was absent from all 18 task runs, whether telemetry was on or off, so we could not confirm the opt-out works either way. Cline says telemetry excludes code and prompts except for its own providers. Model side: ChatGPT Plus content may be used for training by default, with an opt-out = 4. If the npm host counts as an unexplained destination the score would be 3. GitHub issue #7068 (VS Code extension 3.34.0, now closed) reported telemetry reaching data.cline.bot despite the opt-out.

• T3h run 1: the ChatGPT stream stalled for about 7 minutes (11:37:41 to 11:44:31 SAST). Cline then exited 1 with "The operation timed out." after 513 s. The refactor was already complete (50/50 hidden tests, tsc and eslint clean), so the run counts as a pass. A script that checks the exit code would have reported a failure.
• T5 run 2: the grader marked a fail only because the new tests use it.each. Checked by hand: those tests fail on the buggy code in UTC and UTC+2, and the hidden tests pass 4/4 in both zones. Overridden to pass. The grader regex was fixed after the runs. Round 2 T5 results were not affected.

• cost
• setupFriction
• setupFriction
• agentSupport
• agentSupport
• agentSupport
• privacy
• privacy
• privacy
• Pricing

Provisional: one test project, tested 5 Nov 2026

Round 3, 5 Nov 2026, pack 0.3.0: 18 runs, 6 tasks, 3 runs each, zero human interventions. Google Antigravity CLI agy 1.5.1 passed 14/18. Model: the CLI default, Gemini 4 Argon (Medium), via Google AI Pro. The Antigravity series (11:32 to 12:30 SAST) overlapped the Cline series on the same machine.

Output quality 4/5

Round 3, mean of per-task scores over T1h-T6 = 4.333, rounded

Reliability 4/5

Round 3, mean of per-task scores over T1h-T6 = 4, rounded

Cost and free tier 3/5

Google AI Pro, $19.99/month, covered all 18 runs with no rate or usage limit message (Antigravity limits refresh every 5 hours with a weekly cap). The CLI reports tokens but not cost: 590k input tokens, 2.16M cache reads, 100k output and 33k thinking tokens across the 17 runs that returned JSON. 'Use AI Credits' is off by default, so nothing extra was billed.

Speed 5/5

Round 3, mean of per-task scores over T1h-T6 = 5, rounded

Setup friction 4/5

The official installer took 3 s with no account. The --skip-path flag in the docs was rejected, which was one snag. Sign-in is Google OAuth with a pasted code, which works without a local browser. It took about 2 min 15 s (11:22:33 to 11:24:48 SAST), after a first headless attempt timed out waiting 60 s for the code. The tool's own steps took ≤10 min with one snag = 4, using the same editorial rule as the Codex setup score. Counting from the 10:42 install to the first successful run would give 2.

Agent, API, and MCP support 5/5

Headless `agy -p --output-format json` (documented; it needs a prior sign-in). The MCP server was configured in the documented ~/.gemini/config/mcp_config.json. T6 passed 3/3 with no prompt; each run called `get_issue` for #14.

Privacy and data 3/5

Hosts observed in the 18 task runs: - daily-cloudcode-pa.googleapis.com (model, 14.4 MB up) - aicode.googleapis.com - www.googleapis.com and oauth2.googleapis.com (auth) - lh3.googleusercontent.com (account avatar) - the auto-updater on run.app (2 runs) - play.googleapis.com: 37 connections and 5.3 MB uploaded, even though Sarel opted out of interaction data collection at sign-in. play.googleapis.com is not named in Antigravity's docs, so it is one unexplained destination = 3. Its 'Enable Telemetry' setting collects interactions 'for use in evaluating, developing, and improving Antigravity and models'. For the CLI, that choice is saved with the account.

• T4 was 0/3 under the same false-claim rule as OpenCode T4 run 1 in round 2. Run 1 never says to add tests when adding a command. Runs 2 and 3 claim that during runCli dispatch the helpers, including normalizeTitle, validate inputs and throw. normalizeTitle in src/title.ts never throws, and TodoStore calls it, not runCli. Quality 2 for each run. Treating those two claims as over-broad rather than false would make quality 5, reliability 4, and the FitScore 85.
• T3h run 1 hit the cap. The model stream stalled and the harness killed the run at 1549.9 s (limit 1500 s). The code left behind passed 50/50 hidden tests, but a cap hit scores quality 1. Runs 2 and 3 took 85 s and 110 s.

• cost
• cost
• cost
• setupFriction
• setupFriction
• agentSupport
• agentSupport
• privacy
• privacy
• Pricing

• Pricing

• Pricing

• Pricing

• Pricing

• Pricing

• Pricing