{ "round": 3, "date": "2026-11-05", "timezone": "Africa/Johannesburg", "test_pack": "0.3.0", "harness": "local log, not hosted (copy of round2.py: same bwrap sandbox + private netns + logging egress proxy, same prompts, templates, hidden tests, grader, limits; secrets RESEND_API_KEY, FTP_PASSWORD, CLAUDE_AUTH_CODE, GEMINI_AUTH_CODE removed from env). Part 2 adds a fix to the grader's new-test regex (it.each) after the runs; the one affected run is overridden in manual-review.json.", "scores_file": "local log, not hosted", "manual_review": "local log, not hosted", "tools": { "cline": { "result": "tested: 6 tasks x 3 runs, FitScore 88", "version": "Cline CLI 3.0.74", "plan": "ChatGPT Plus ($20/mo) via Cline's native 'Sign in with ChatGPT' (openai-codex) provider, signed in by Sarel", "model": "gpt-6.1-sol, thinking low (`-m gpt-6.1-sol --thinking low`). Chosen to match Codex CLI in round 2. That model is inferred, not logged: `codex debug prompt-input` says 'based on GPT-6', and ~/.codex/models_cache.json lists gpt-6.1-sol as priority 1 with default reasoning low. The wizard default was 'GPT-5.3 Codex Spark', medium thinking.", "command": "cline --json -P openai-codex -c -m gpt-6.1-sol --thinking low (HOME=tests/cline-home; MCP for T6 via ~/.cline/data/settings/cline_mcp_settings.json)", "native_loop_check": "Cline ran its own agent loop: run_commands, read_files, apply_patch and editor tools, with no execution=provider tags, 5 to 9 iterations per run. This is unlike the part-1 claude-code passthrough, which is kept below.", "findings": [ "18/18 passes (2 after review: one grader false negative on it.each tests, and T4 docs graded by hand). No rate limit on ChatGPT Plus.", "Same model as Codex CLI in round 2 (inferred): Cline passed 18/18 against Codex's 17/18 (Codex's one fail, T6 run 1, came from our harness's MCP approval setting). Codex was faster on every task: medians of 27 to 93 s against Cline's 49 to 123 s. Cline's runs overlapped with the agy series, so shared box load may account for part of that gap. Both score speed 5.", "T3h run 1: the ChatGPT stream stalled for about 7 min (11:37:41 to 11:44:31 SAST). Cline then exited 1 with 'The operation timed out.' after 513 s, but the refactor was already complete (50/50 hidden tests).", "API-equivalent cost shown by Cline: $1.41 for all 18 runs (not billed).", "data.cline.bot telemetry was seen only in the first smoke run, never in task runs, so the opt-out could not be verified.", "Cline starts a background hub daemon (--cline-hub-daemon) even for one-shot --json runs." ], "smoke_runs": [ "logs/round3/cline-smoke-telemetry-on", "logs/round3/cline-smoke-telemetry-on2", "logs/round3/cline-smoke-telemetry-off" ], "part1_passthrough": { "result": "stopped after 1 run; no FitScore", "version": "Cline CLI 3.0.74", "path_tried": "provider claude-code (Cline drives the already signed-in `claude` CLI on Sarel's Claude Pro plan; no new credentials). Documented at https://docs.cline.bot/provider-config/claude-code", "setup_evidence": "`cline auth -p claude-code` (non-interactive) refused: 'provider \"claude-code\" requires API key setup'. Passing -P claude-code on the run command worked with no sign-in. Default model for this provider was claude-haiku-5-5; we set -m opus to match the model Claude Code used in round 2.", "smoke_runs": [ "logs/round3/cline-smoke-telemetry-on (default model haiku, 4.4 s, OK)", "logs/round3/cline-smoke-telemetry-on-opus (2.6 s, OK)" ], "task_runs": [ { "run": "cline-t1h-r1", "started": "2026-11-05T10:44:59+02:00", "wall_s": 44.2, "pass": true, "hidden_tests": "6/6", "changed_files": [ "src/lookup.ts", "src/store.ts" ], "tsc_ok": true, "eslint_ok": true, "cline_iterations": 1, "usage_reported_by_cline": { "inputTokens": 131143, "outputTokens": 4712, "cacheReadTokens": 106203, "cacheWriteTokens": 24930, "totalCost": 0.2401706 }, "tool_calls": { "Bash (execution=provider)": 9, "Read (execution=provider)": 10, "Edit (execution=provider)": 2 }, "egress": { "api.anthropic.com:443": { "connections": 13, "bytes_up": 1275183, "bytes_down": 269728, "errors": 0 }, "http-intake.logs.us5.datadoghq.com:443": { "connections": 3, "bytes_up": 199150, "bytes_down": 15455, "errors": 0 }, "data.cline.bot:443": { "connections": 1, "bytes_up": 2131, "bytes_down": 5877, "errors": 0 }, "registry.npmjs.org:443": { "connections": 1, "bytes_up": 1785, "bytes_down": 5009, "errors": 0 }, "mcp-proxy.anthropic.com:443": { "connections": 9, "bytes_up": 25042, "bytes_down": 514052, "errors": 0 } } } ], "why_stopped": "Every tool call in the run was a Claude Code built-in (Bash, Read, Edit) marked execution=provider, and Cline logged a single iteration. With the claude-code provider the Claude Code CLI runs the whole agent loop and Cline only relays it. A Cline score from these runs would really be a second Claude Code score, so we stopped after one run to save Claude Pro usage.", "egress_hosts": [ "api.anthropic.com:443", "data.cline.bot:443", "http-intake.logs.us5.datadoghq.com:443", "mcp-proxy.anthropic.com:443", "registry.npmjs.org:443" ], "telemetry_source": "https://docs.cline.bot/enterprise-solutions/monitoring/telemetry (on by default in the CLI; opt-out: telemetryOptOut true in /data/settings/global-settings.json)", "fair_path_needs_sarel": "Run `cline auth`, choose the ChatGPT subscription provider (openai-codex) and finish the browser sign-in with Sarel's ChatGPT Plus account. Cline's own agent loop then calls the model directly. The wizard also offers 'Import from Codex CLI' (copies ~/.codex/auth.json); we did not use it, because it shares the Codex sign-in token with another tool, and that is Sarel's decision.", "sources": [ "https://raw.githubusercontent.com/cline/cline/main/apps/cli/README.md", "https://docs.cline.bot/provider-config/claude-code", "https://docs.cline.bot/provider-config/openai", "https://docs.cline.bot/enterprise-solutions/monitoring/telemetry" ] }, "fitscore": 88, "criterion_scores": { "quality": { "score": 5, "basis": "mean of per-task scores over T1h-T6 = 5, rounded" }, "reliability": { "score": 5, "basis": "mean of per-task scores over T1h-T6 = 5, rounded" }, "speed": { "score": 5, "basis": "mean of per-task scores over T1h-T6 = 5, rounded" }, "cost": { "score": 3, "basis": "ChatGPT Plus, $20/month billed monthly, covered all 18 runs with no rate or usage limit hit. Cline showed an API-equivalent figure of $1.41 in total (gpt-6.1-sol at $2/M input and $10/M output, 1.77M input tokens with 1.34M cache reads, 39k output). This was not billed, because Plus is flat-rate.", "sources": [ "https://help.openai.com/en/articles/6950777-what-is-chatgpt-plus", "local log, not hosted (run_result.aggregateUsage)" ] }, "setup": { "score": 3, "basis": "npm install of Cline CLI 3.0.74 was quick and needed no account. Sign-in to ChatGPT Plus uses the native 'Sign in with ChatGPT' (openai-codex) provider, which is a browser OAuth flow. Sarel had to finish it in a desktop browser, and the earlier non-interactive `cline auth` attempt was refused. A browser step that blocks a headless box = 3. The sign-in duration was not timed, because Sarel did it himself.", "sources": [ "https://cline.bot/blog/introducing-openai-codex-oauth", "https://docs.cline.bot/provider-config/openai", "local log, not hosted" ] }, "agent_api_mcp": { "score": 5, "basis": "Headless `cline --json` (documented NDJSON mode). MCP servers can be added by a single documented command (`cline mcp add --yes -- `) or the documented `~/.cline/data/settings/cline_mcp_settings.json` file; our harness wrote that file. T6 passed 3/3 with no prompt; each run called the MCP tool `get_issue`.", "sources": [ "https://docs.cline.bot/cli/cli-reference", "https://docs.cline.bot/mcp/mcp-overview", "https://github.com/cline/cline/blob/main/apps/cli/README.md" ] }, "privacy": { "score": 4, "basis": "Hosts observed in the 18 task runs were chatgpt.com (model, 5.6 MB up) and registry.npmjs.org (one small request per run, consistent with an npm version check; not described in Cline's docs). data.cline.bot (Cline telemetry, documented, on by default in the CLI, opt-out `telemetryOptOut`) was seen only in the first smoke run. It was absent from all 18 task runs, whether telemetry was on or off, so we could not confirm the opt-out works either way. Cline says telemetry excludes code and prompts except for its own providers. Model side: ChatGPT Plus content may be used for training by default, with an opt-out = 4. If the npm host counts as an unexplained destination the score would be 3. GitHub issue #7068 (VS Code extension 3.34.0, now closed) reported telemetry reaching data.cline.bot despite the opt-out.", "sources": [ "https://docs.cline.bot/enterprise-solutions/monitoring/telemetry", "https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance", "https://github.com/cline/cline/issues/7068" ] } }, "per_task": { "t1h": { "runs": 3, "passes": 3, "median_wall_s": 49.1, "limit_s": 900, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t2h": { "runs": 3, "passes": 3, "median_wall_s": 123.3, "limit_s": 1500, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t3h": { "runs": 3, "passes": 3, "median_wall_s": 106.6, "limit_s": 1500, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t4": { "runs": 3, "passes": 3, "median_wall_s": 57.0, "limit_s": 900, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t5": { "runs": 3, "passes": 3, "median_wall_s": 72.1, "limit_s": 1500, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t6": { "runs": 3, "passes": 3, "median_wall_s": 63.3, "limit_s": 1200, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 } }, "passes": "18/18", "egress_task_runs": { "chatgpt.com": { "runs_seen": 18, "connections": 18, "bytes_up": 5564673, "bytes_down": 21190073 }, "registry.npmjs.org": { "runs_seen": 18, "connections": 18, "bytes_up": 32130, "bytes_down": 90191 } }, "human_interventions": 0, "rate_limit_hit": false }, "google-antigravity": { "result": "tested: 6 tasks x 3 runs, FitScore 79", "version": "agy 1.5.1", "plan": "Google AI Pro ($19.99/mo) via Google OAuth, signed in by Sarel (interaction data collection opted out)", "model": "Gemini 4 Argon (Medium), the CLI default (`model override to backend: label=\"Gemini 4 Argon (Medium)\"` in cli.log)", "command": "agy -p --output-format json --dangerously-skip-permissions (inside bwrap; ~/.gemini/antigravity-cli bound, per-run ~/.gemini/config with mcp_config.json for T6)", "models_offered": "logs/round3/agy-models.txt: Gemini 4 Argon (low/medium/high), Gemini 3.8 Flash, Claude Opus 5.5, Claude Sonnet 5.5, GPT-OSS 120B", "findings": [ "14/18 passes. T4 was 0/3: run 1 left out 'add tests', and runs 2 and 3 claimed normalizeTitle validates and throws (it does not, and runCli never calls it). T3h run 1 hit the 25-min cap after the model stream stalled (the code at kill time passed 50/50).", "If the two normalizeTitle claims were treated as over-broad rather than false, the FitScore would be 85 instead of 79.", "play.googleapis.com received 5.3 MB in 37 connections across the 18 runs, about a third as much as the model endpoint. This happened even though Sarel opted out of interaction data collection, and the host is not named in Antigravity's docs.", "An LLM safety monitor ('LLM Monitor sir_flash_lite') evaluates the agent's trajectory before risky tool calls (cli.log).", "The CLI contacted its auto-updater (antigravity-cli-auto-updater-\u2026run.app) in 2 runs and spawned a background update process. The sandbox kept the binary unchanged.", "AI Pro gives access to Claude Opus 5.5 and Sonnet 5.5 inside agy as well as Gemini (agy models).", "No rate or usage limit was hit in 18 runs on AI Pro." ], "smoke_runs": [ "logs/round3/agy-smoke-telemetry-on" ], "fitscore": 79, "criterion_scores": { "quality": { "score": 4, "basis": "mean of per-task scores over T1h-T6 = 4.333, rounded" }, "reliability": { "score": 4, "basis": "mean of per-task scores over T1h-T6 = 4, rounded" }, "speed": { "score": 5, "basis": "mean of per-task scores over T1h-T6 = 5, rounded" }, "cost": { "score": 3, "basis": "Google AI Pro, $19.99/month, covered all 18 runs with no rate or usage limit message (Antigravity limits refresh every 5 hours with a weekly cap). The CLI reports tokens but not cost: 590k input tokens, 2.16M cache reads, 100k output and 33k thinking tokens across the 17 runs that returned JSON. 'Use AI Credits' is off by default, so nothing extra was billed.", "sources": [ "https://one.google.com/about/google-ai-plans/", "https://antigravity.google/docs/plans/", "https://antigravity.google/docs/settings/" ] }, "setup": { "score": 4, "basis": "The official installer took 3 s with no account. The --skip-path flag in the docs was rejected, which was one snag. Sign-in is Google OAuth with a pasted code, which works without a local browser. It took about 2 min 15 s (11:22:33 to 11:24:48 SAST), after a first headless attempt timed out waiting 60 s for the code. The tool's own steps took \u226410 min with one snag = 4, using the same editorial rule as the Codex setup score. Counting from the 10:42 install to the first successful run would give 2.", "sources": [ "https://antigravity.google/docs/getting-started/", "https://antigravity.google/docs/install/", "local log, not hosted" ] }, "agent_api_mcp": { "score": 5, "basis": "Headless `agy -p --output-format json` (documented; it needs a prior sign-in). The MCP server was configured in the documented ~/.gemini/config/mcp_config.json. T6 passed 3/3 with no prompt; each run called `get_issue` for #14.", "sources": [ "https://antigravity.google/docs/cli/headless/", "https://antigravity.google/docs/settings/", "https://github.com/google-antigravity/antigravity-cli" ] }, "privacy": { "score": 3, "basis": "Hosts observed in the 18 task runs:\n- daily-cloudcode-pa.googleapis.com (model, 14.4 MB up)\n- aicode.googleapis.com\n- www.googleapis.com and oauth2.googleapis.com (auth)\n- lh3.googleusercontent.com (account avatar)\n- the auto-updater on run.app (2 runs)\n- play.googleapis.com: 37 connections and 5.3 MB uploaded, even though Sarel opted out of interaction data collection at sign-in.\n\nplay.googleapis.com is not named in Antigravity's docs, so it is one unexplained destination = 3. Its 'Enable Telemetry' setting collects interactions 'for use in evaluating, developing, and improving Antigravity and models'. For the CLI, that choice is saved with the account.", "sources": [ "https://antigravity.google/docs/settings/", "https://antigravity.google/terms/", "https://discuss.ai.google.dev/t/antigravity-2-26-0-macos-sign-in-fails-loadcodeassist-socket-is-not-connected/195017/9" ] } }, "per_task": { "t1h": { "runs": 3, "passes": 3, "median_wall_s": 72.7, "limit_s": 900, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t2h": { "runs": 3, "passes": 3, "median_wall_s": 200.4, "limit_s": 1500, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t3h": { "runs": 3, "passes": 2, "median_wall_s": 109.8, "limit_s": 1500, "quality": 4, "quality_by_run": [ 1, 5, 5 ], "reliability": 3, "speed": 5 }, "t4": { "runs": 3, "passes": 0, "median_wall_s": 90.5, "limit_s": 900, "quality": 2, "quality_by_run": [ 2, 2, 2 ], "reliability": 1, "speed": 5 }, "t5": { "runs": 3, "passes": 3, "median_wall_s": 127.8, "limit_s": 1500, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 }, "t6": { "runs": 3, "passes": 3, "median_wall_s": 82.4, "limit_s": 1200, "quality": 5, "quality_by_run": [ 5, 5, 5 ], "reliability": 5, "speed": 5 } }, "passes": "14/18", "egress_task_runs": { "daily-cloudcode-pa.googleapis.com": { "runs_seen": 18, "connections": 165, "bytes_up": 14350933, "bytes_down": 4182856 }, "play.googleapis.com": { "runs_seen": 18, "connections": 37, "bytes_up": 5295646, "bytes_down": 263289 }, "aicode.googleapis.com": { "runs_seen": 18, "connections": 19, "bytes_up": 65502, "bytes_down": 690597 }, "www.googleapis.com": { "runs_seen": 18, "connections": 18, "bytes_up": 43992, "bytes_down": 108051 }, "lh3.googleusercontent.com": { "runs_seen": 18, "connections": 18, "bytes_up": 38142, "bytes_down": 525218 }, "antigravity-cli-auto-updater-974169037036.us-central1.run.app": { "runs_seen": 2, "connections": 2, "bytes_up": 3858, "bytes_down": 17147 }, "oauth2.googleapis.com": { "runs_seen": 1, "connections": 1, "bytes_up": 2227, "bytes_down": 6030 } }, "human_interventions": 0, "rate_limit_hit": false }, "openhands": { "result": "skipped; no FitScore", "version": "OpenHands CLI 1.16.0 (SDK 1.21.0)", "why": "The CLI needs either `openhands login` to OpenHands Cloud (a new account, which is not allowed) or an LLM provider API key (none available). Its SDK includes a ChatGPT-subscription OAuth module (openhands/sdk/llm/auth/openai.py), but the CLI does not expose it. Using it would mean writing our own harness and asking Sarel for another ChatGPT sign-in. The docs also say the CLI 'is no longer actively maintained'.", "fair_path_needs_sarel": "Either an OpenHands Cloud account or an LLM API key, both Sarel's decision. Or a ChatGPT sign-in through the SDK, which would not be a test of the CLI.", "sources": [ "https://docs.openhands.dev/openhands/usage/cli/quick-start" ] }, "aider": { "result": "skipped; no FitScore", "version": "aider 0.86.2", "why": "Aider is bring-your-own-key. Its documented paths are provider API keys, OpenRouter's free models (these need an OpenRouter account and key), GitHub Copilot (no Copilot plan) or local models via Ollama. It has no ChatGPT Plus, Claude Pro or Google AI Pro sign-in. A local model on this 8-CPU, 15 GB box with no GPU would break the pack's rule of one shared model for BYOK tools, and the result would measure the small model rather than Aider.", "fair_path_needs_sarel": "An API key for the shared BYOK model (to be agreed), or approval to run a local Ollama model and publish it as a separate 'local model' result.", "sources": [ "https://aider.chat/docs/llms.html" ] }, "gemini-cli": { "result": "skipped", "why": "Sarel's standing instruction: 'Don't test Gemini CLI further.' In rounds 1 and 2 the personal Google login was discontinued for this CLI. A free AI Studio API key is not available, and we were not allowed to sign up for one." } }, "new_fitscores": { "cline": 88, "google-antigravity": 79 }, "subscription_usage": "Part 1: Claude Pro (2 smokes + 1 Cline passthrough run). Part 2: ChatGPT Plus 18 Cline task runs + 3 smokes; Google AI Pro 18 agy task runs + 1 smoke + 1 `agy models` listing. No rate or usage limits hit.", "secrets": "None entered or printed. AGY_AUTH_CODE and CLINE_CALLBACK_URL were not used and were removed from the agent env, along with RESEND_API_KEY, FTP_PASSWORD, CLAUDE_AUTH_CODE, GEMINI_AUTH_CODE and any secret-looking variable.", "published_note": "Round 3 summary, 5 Nov 2026. Per-run stdout and diffs are not hosted." }