{ "generated_note": "Conductor Mode benchmark — per-suite leg summaries. Regenerated directly from the live dashboard_data.json that powers 2004-conductor-benchmark.community.iamstarchild.com, so this file and the page can never disagree. Suites: AIME 2025, GPQA Diamond, LiveCodeBench (LCB-100), Everyday-mix, MMMU. Objective exact-match scoring, no LLM judge. Costs are real OpenRouter billing, not token estimates.", "regenerated_at": "2026-08-06T10:30:21+00:00", "source_generated_at": "2026-08-05T23:07:00.000000+00:00", "model_bench_updated_at": "2026-08-04T13:35:13Z", "composite_rank": { "suites": [ "AIME 2025", "GPQA Diamond", "LCB-100", "Everyday" ], "weighting": "equal", "rows": [ { "name": "Kimi K3", "composite": 93.4, "per_task": 0.0624, "accs": { "AIME": 100.0, "GPQA": 90.2, "LCB": 92.9, "Everyday": 90.4 }, "pers": { "AIME": 0.0785, "GPQA": 0.0957, "LCB": 0.1441, "Everyday": 0.0043 }, "everyday_cost": 0.00378, "suite_costs": { "AIME": 0.062, "GPQA": 0.0399, "LCB": 0.1441, "Everyday": 0.00378 } }, { "name": "GPT-5.6 Sol", "composite": 93.1, "per_task": 0.0394, "accs": { "AIME": 96.7, "GPQA": 93.4, "LCB": 96.0, "Everyday": 86.2 }, "pers": { "AIME": 0.0632, "GPQA": 0.0356, "LCB": 0.0697, "Everyday": 0.0108 }, "everyday_cost": 0.01011, "suite_costs": { "AIME": 0.0455, "GPQA": 0.0324, "LCB": 0.0697, "Everyday": 0.01011 } }, { "name": "Conductor v3 (new rules)", "composite": 92.3, "per_task": 0.0434, "accs": { "AIME": 100.0, "GPQA": 90.9, "Everyday": 89.4, "LCB": 89.0 }, "pers": { "AIME": 0.0347, "GPQA": 0.0172, "Everyday": 0.0018, "LCB": 0.0876 }, "everyday_cost": 0.00106, "suite_costs": { "AIME": 0.0347, "GPQA": 0.0172, "LCB": 0.0876, "Everyday": 0.00106 } }, { "name": "Claude Opus 5", "composite": 92.2, "per_task": 0.1075, "accs": { "AIME": 100.0, "GPQA": 94.4, "LCB": 84.7, "Everyday": 89.7 }, "pers": { "AIME": 0.087, "GPQA": 0.0953, "LCB": 0.1968, "Everyday": 0.0313 }, "everyday_cost": 0.02879, "suite_costs": { "AIME": 0.109, "GPQA": 0.0953, "LCB": 0.1968, "Everyday": 0.02879 } }, { "name": "Gemini 3.6 Flash", "composite": 91.5, "per_task": 0.0706, "accs": { "AIME": 100.0, "GPQA": 92.3, "LCB": 88.0, "Everyday": 85.5 }, "pers": { "AIME": 0.1046, "GPQA": 0.064, "LCB": 0.1249, "Everyday": 0.0075 }, "everyday_cost": 0.00726, "suite_costs": { "AIME": 0.081, "GPQA": 0.0611, "LCB": 0.133, "Everyday": 0.00726 } }, { "name": "GLM-5.2", "composite": 91.4, "per_task": 0.0138, "accs": { "AIME": 93.3, "GPQA": 92.4, "LCB": 92.3, "Everyday": 87.6 }, "pers": { "AIME": 0.0184, "GPQA": 0.0254, "LCB": 0.0231, "Everyday": 0.0014 }, "everyday_cost": 0.00117, "suite_costs": { "AIME": 0.018, "GPQA": 0.0128, "LCB": 0.0231, "Everyday": 0.00117 } }, { "name": "Qwen 3.8 Max", "composite": 88.9, "per_task": 0.0364, "accs": { "AIME": 96.7, "GPQA": 95.9, "LCB": 76.3, "Everyday": 86.5 }, "pers": { "AIME": 0.0718, "GPQA": 0.0432, "LCB": 0.0392, "Everyday": 0.004 }, "everyday_cost": 0.00398, "suite_costs": { "AIME": 0.0421, "GPQA": 0.0297, "LCB": 0.0698, "Everyday": 0.00398 } }, { "name": "KAT-Coder-Pro", "composite": 86.7, "per_task": 0.0213, "accs": { "AIME": 90.0, "GPQA": 86.7, "LCB": 83.9, "Everyday": 86.2 }, "pers": { "AIME": 0.0346, "GPQA": 0.0237, "LCB": 0.0323, "Everyday": 0.0022 }, "everyday_cost": 0.0022, "suite_costs": { "AIME": 0.025, "GPQA": 0.0175, "LCB": 0.0406, "Everyday": 0.0022 } }, { "name": "Grok 4.5", "composite": 84.2, "per_task": 0.0287, "accs": { "AIME": 76.7, "GPQA": 88.9, "LCB": 91.8, "Everyday": 79.4 }, "pers": { "AIME": 0.0452, "GPQA": 0.0232, "LCB": 0.0502, "Everyday": 0.0049 }, "everyday_cost": 0.00477, "suite_costs": { "AIME": 0.033, "GPQA": 0.024, "LCB": 0.053, "Everyday": 0.00477 } }, { "name": "Inkling", "composite": 82.9, "per_task": 0.0436, "accs": { "AIME": 80.0, "GPQA": 83.8, "LCB": 81.4, "Everyday": 86.2 }, "pers": { "AIME": 0.0619, "GPQA": 0.0478, "LCB": 0.0773, "Everyday": 0.0026 }, "everyday_cost": 0.00232, "suite_costs": { "AIME": 0.051, "GPQA": 0.038, "LCB": 0.083, "Everyday": 0.00232 } }, { "name": "KAT-Coder-Air", "composite": 78.2, "per_task": 0.0042, "accs": { "AIME": 63.3, "GPQA": 79.8, "LCB": 90.4, "Everyday": 79.4 }, "pers": { "AIME": 0.0084, "GPQA": 0.0045, "LCB": 0.0053, "Everyday": 0.0002 }, "everyday_cost": 0.00024, "suite_costs": { "AIME": 0.005, "GPQA": 0.0035, "LCB": 0.0082, "Everyday": 0.00024 } } ], "note_en": "The Everyday column is pinned to the 94-task common set every leg actually ran — same tasks, same grading. Conductor v2 (GLM-5.2 + MiniMax era) is not ranked: it only has real runs on AIME (96.7%) and Everyday (86.2%) — it never ran the GPQA or LCB-100 pools. Per-suite v3-vs-v2 deltas are in §0.", "note_zh": "Everyday 一列统一取所有腿都实际跑过的 94 题共同集，同题同判。Conductor v2（GLM-5.2 + MiniMax 时代）不参与综合排名：其真实数据仅覆盖 AIME（96.7%）与 Everyday（86.2%），从未跑过 GPQA 与 LCB-100 题库。v3 对 v2 的分套件对比见 §0。", "everyday_n": 94 }, "v3_final": { "updated": "2026-08-05T13:15:24Z", "config": { "economy": "DeepSeek V4 Flash 0731", "strong": "GLM-5.2", "escalation": "Kimi K3", "media": "Qwen 3.8 Max", "fallback": "GLM-5.2 → Qwen 3.8 Max → DeepSeek V4 Flash" }, "rows": [ { "suite": "AIME 2025", "n": 30, "total": 30, "acc": 100.0, "per": 0.0347, "status": "measured", "base_acc": 96.7, "base_per": 0.0311, "base_label": "v2 smart (real run, n=30)" }, { "suite": "GPQA Diamond", "n": 197, "total": 198, "acc": 90.9, "per": 0.0314, "status": "partial", "base_acc": null, "base_per": null, "base_label": null }, { "suite": "Everyday", "n": 94, "total": 94, "acc": 89.4, "per": 0.00106, "status": "measured", "base_acc": 86.2, "base_per": 0.00091, "base_label": "v2 smart (same 94 tasks)" }, { "suite": "LiveCodeBench-100", "n": 100, "total": 100, "acc": 89.0, "per": 0.1063, "status": "measured", "base_acc": null, "base_per": null, "base_label": null }, { "suite": "MMMU-100", "n": 96, "total": 100, "acc": 91.7, "per": 0.0168, "status": "carryover", "base_acc": 91.7, "base_per": 0.0168, "base_label": "Qwen 3.8 Max (v3 media leg, carried)" } ], "composite_acc": 92.2 }, "gpqa": { "legs": [ { "leg": "Qwen 3.8 Max", "total": 172, "passed": 165, "rate": 95.9, "cost": 7.44, "per_q": 0.0432 }, { "leg": "Claude Opus 5", "total": 197, "passed": 186, "rate": 94.4, "cost": 18.77, "per_q": 0.0953 }, { "leg": "GPT-5.6 Sol", "total": 198, "passed": 185, "rate": 93.4, "cost": 7.05, "per_q": 0.0356 }, { "leg": "GLM-5.2", "total": 185, "passed": 171, "rate": 92.4, "cost": 4.69, "per_q": 0.0254 }, { "leg": "Gemini 3.6 Flash", "total": 196, "passed": 181, "rate": 92.3, "cost": 12.55, "per_q": 0.064 }, { "leg": "Conductor v3", "total": 197, "passed": 179, "rate": 90.9, "cost": 6.19, "per_q": 0.0314 }, { "leg": "Kimi K3", "total": 194, "passed": 175, "rate": 90.2, "cost": 18.56, "per_q": 0.0957 }, { "leg": "Grok 4.5", "total": 198, "passed": 176, "rate": 88.9, "cost": 4.6, "per_q": 0.0232 }, { "leg": "DeepSeek V4 Flash", "total": 193, "passed": 170, "rate": 88.1, "cost": 0.84, "per_q": 0.0044 }, { "leg": "KAT-Coder-Pro", "total": 196, "passed": 170, "rate": 86.7, "cost": 4.64, "per_q": 0.0237 }, { "leg": "Inkling", "total": 197, "passed": 165, "rate": 83.8, "cost": 9.42, "per_q": 0.0478 }, { "leg": "KAT-Coder-Air", "total": 198, "passed": 158, "rate": 79.8, "cost": 0.89, "per_q": 0.0045 } ], "target": 198, "snapshot": "2026-08-05 14:30 HKT — live recount from result files (Conductor v3 189/198, still running)" }, "reproduce": { "task_sets": "exports/everyday_mix.jsonl (163 tasks, full text + answer keys). AIME 2025 / GPQA Diamond / LiveCodeBench / MMMU task_ids map onto the public upstream sets.", "grader": "exports/grade.py — the exact answer-extraction and matching logic used to produce results_per_task.csv.", "per_task": "exports/results_per_task.csv — 7,329 rows: suite, leg, task_id, correct, real cost_usd, routed_model." } }