# Conductor Mode benchmark — reproduction bundle

Everything here is regenerated from the same `dashboard_data.json` that renders
the live page, so the downloads and the page cannot disagree.

| File | What it is |
|---|---|
| `results_per_task.csv` | 7,329 rows — one per task per leg. Columns: `suite, leg, task_id, correct, cost_usd, difficulty, routed_model`. `cost_usd` is real OpenRouter billing, not a token estimate. |
| `results_summary.json` | Per-suite leg summaries + composite ranking, regenerated from the live dashboard data. |
| `everyday_mix.jsonl` | The full Everyday-mix task set: 163 tasks with problem text, answer keys and aliases. `task_id` joins to the CSV. |
| `grade.py` | The exact grading logic behind the `correct` column. Stdlib only, offline, no LLM judge. |

## Verifying a suite

**Everyday-mix** is fully self-contained here:

```bash
python grade.py --selftest        # 163/163 — proves the grader matches its own keys
```

Task set derived from public HF sets (boolq, gsm8k, cais/mmlu, MMLU-Pro,
ai2_arc, trivia_qa, web_questions) plus a small authored composite tier,
seed 42, deduplicated by normalized problem text. 163 built, 155 in the CSV
(8 dropped by the per-run length filter).

**AIME 2025 / GPQA Diamond / LiveCodeBench / MMMU** are unmodified public sets —
`task_id` in the CSV is the upstream identifier (`aime2025_01`, `gpqa_000`,
`abc374_e.json`, `validation_Accounting_1`), so scores can be recomputed
against the originals. LiveCodeBench is graded by executing generated code
against the public test cases, not by string match; see `lcb_grade_note()`.

## Scope

These numbers measure a **routing system's cost-efficiency**, not a single
model's ceiling. Conductor v3 publishes at rank 3 (92.3 composite) behind
Kimi K3 (93.4) and GPT-5.6 Sol (93.1) — both are in the same table.

Absolute agentic scores (Terminal-Bench and similar) are **not** comparable
across harnesses. Reasoning effort, timeout multipliers and trial-launch
failure handling move those numbers by tens of points, so only compare runs
made under an identical harness configuration.