Agent benchmark
AI agents can now drive cytometry software. This benchmark measures whether they get the analysis right: graded tasks on simulated experiments whose true answers are known, worded as a user would ask, with nothing but CytoWeave's tools.
What it measures
Each task is a request a user could make of an agent: gate a population, find an error in a compensation matrix, test whether a treatment changes a population, name what went wrong during acquisition, read a concentration from a bead assay, write a report. Every task runs on an example experiment generated with a seed of its own, so its answer cannot be recalled from the example's defaults, and is graded against the simulator's own record of that experiment: the true population of every cell, the planted fault, the true spillover, the true concentration.
Grading is deterministic: code reads the analysis the agent left in CytoWeave (its gates, results and files) and the answer line the task asks for, and gives partial credit by a fixed rubric. No model judges another.
Results
The score is the mean over tasks of each task's mean score over runs. The reference solution, scripted through the same tools without a model, scores 100% and shows that every task can be solved with CytoWeave's tools; an agent that answers nothing scores 0%. Cost is Claude Code's own estimate of each run's API cost (on a subscription, usage of the plan).
| Agent and model | Score | Tasks × runs | Answered | Tool calls per run | Cost per run | Run on | CytoWeave |
|---|---|---|---|---|---|---|---|
| Reference solution (scripted, no model) | 100% | 11 × 1 | 11 of 11 | 1.9 | — | 2026-10-07 | 0.7.0 (eff402f) |
| Reference solution (scripted, no model) | 100% | 11 × 1 | 11 of 11 | 1.9 | — | 2026-10-08 | 0.8.0 (fc63f56) |
| Claude Code, Claude Sonnet 5.5 | 100% | 11 × 3 | 33 of 33 | 3.4 | $0.08 | 2026-10-07 | 0.7.0 (c581cac) |
| Claude Code, Claude Haiku 5.5 | 100% | 11 × 3 | 33 of 33 | 4.8 | $0.01 | 2026-10-08 | 0.8.0 (fc63f56) |
| Claude Code, Claude Opus 5.5 | 100% | 11 × 3 | 33 of 33 | 4.5 | $0.16 | 2026-10-08 | 0.8.0 (fc63f56) |
| Claude Code, Claude Sonnet 5.5 | 100% | 11 × 3 | 33 of 33 | 3.2 | $0.08 | 2026-10-08 | 0.8.0 (fc63f56) |
| Claude Code, Claude Opus 5.5 | 99.7% | 11 × 3 | 33 of 33 | 5.1 | $0.17 | 2026-10-07 | 0.7.0 (c581cac) |
| Claude Code, Claude Haiku 4.5 | 95.1% | 11 × 3 | 33 of 33 | 3.2 | $0.04 | 2026-10-07 | 0.7.0 (eff402f) |
| No answer (lower bound) | 0% | 11 × 1 | 0 of 11 | 0.0 | — | 2026-10-07 | 0.7.0 (eff402f) |
| No answer (lower bound) | 0% | 11 × 1 | 0 of 11 | 0.0 | — | 2026-10-08 | 0.8.0 (fc63f56) |
By task
| Task | Reference solution (scripted, no model) | Reference solution (scripted, no model) | Claude Sonnet 5.5 | Claude Haiku 5.5 | Claude Opus 5.5 | Claude Sonnet 5.5 | Claude Opus 5.5 | Claude Haiku 4.5 |
|---|---|---|---|---|---|---|---|---|
| Gate T cells and report their frequency gating | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 76.5% |
| Find the wrong value in a spillover matrix compensation | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Test whether stimulation changes CD25+ CD4 T cells statistics | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Name each acquisition's problem quality control | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 80% |
| Find the sample with an acquisition problem quality control | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Find what is wrong with one dye in a spectral panel spectral unmixing | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Choose an antibody amount from a titration assay setup | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Find the most potent compound on a plate plates | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 90% |
| Read a serum concentration from a bead assay immunoassay | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Compare calcium responses to stimuli kinetics | 100% | 100% | 100% | 100% | 100% | 100% | 96.3% | 100% |
| Write a gating report as a PDF reporting | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
How a run works
- The harness starts CytoWeave (built from the benchmark's commit) with an empty library, opens its window, generates the task's example with the task's seed, and prepares it as the task says (for some tasks, the example's suggested gates). The example's title, the simulator's per-cell truth and any annotation naming a planted fault are removed.
- The agent is started as a program of its own, in an empty folder, with CytoWeave's MCP server as its only tools: no files, shell or web. Claude Code runs with
--tools "",--strict-mcp-config, the settings of that empty folder only (not the user's hooks, plugins or defaults) and the model and effort given; each run is a fresh session. - The agent gets the task's prompt, which starts as every one does: the data are already open in CytoWeave, in the workspace "Experiment". It works through CytoWeave's tools in the window the harness watches and ends its reply with the answer line the prompt asks for. A run stops after 60 turns or 20 minutes.
- The harness grades the window's state, the files in the task's output folder and the answer, and records the score, the tool calls, the tokens and the time.
Tasks
| Task | Example | Graded against |
|---|---|---|
| Gate T cells and report their frequency | PBMC, 14 colors | The F1 of the agent's "T cells" gate against every cell's true type (full credit from 0.9), and the reported percentage of all events (within 1.5 points) |
| Find the wrong value in a spillover matrix | PBMC, 14 colors | The planted error (APC into Alexa Fluor 700), and the corrected value against the true spillover (within 1.5 points) |
| Test whether stimulation changes CD25+ CD4 T cells | PBMC, 14 colors, suggested gates | The direction (up, as simulated), significance, and a test paired by donor |
| Name each acquisition's problem | Acquisition QC showcase | Each of four files: clean, a clog, drift or an air bubble, as simulated |
| Find the sample with an acquisition problem | PBMC, 14 colors | The sample with the simulated clog, and the kind of problem |
| Find what is wrong with one dye in a spectral panel | Spectral 25-color, PE-Cy7 degraded by 10% in the samples | The dye and the cause (a degraded tandem) |
| Choose an antibody amount from a titration | Titration and voltage walk | The amount following Bonilla et al. 2024 from the true binding curve (125 ng) |
| Find the most potent compound on a plate | 96-well drug screen, suggested gates | The most potent compound, its IC50 against the simulated one (within 2×), and the inactive compound |
| Read a serum concentration from a bead assay | Cytokine bead assay, suggested gates | The serum's true IL-6 concentration (within 20%) |
| Compare calcium responses to stimuli | Calcium flux | The strongest stimulus and the true share of responding cells (within 10 points) |
| Write a gating report as a PDF | PBMC, 14 colors, suggested gates | The PDF written where asked, a page per stained sample, and the gating strategy on its pages |
Before results are published, the benchmark checks itself: every reference solution must score at least 90% (all score 100%), an agent that answers nothing must score 0, and unit tests give each grader plausible wrong answers to check its partial credit. These checks run in continuous integration with the rest of the validation.
Limits
- Simulated data. The truth is the simulator's. Real data are messier, and an agent good here may still be wrong on a real panel; CytoWeave's own analyses are validated on real data separately (Science).
- The examples are public. Their code is on GitHub, so a model trained on it could know where faults were planted. Seeds change every task's data, but not which fault an example carries; later versions of the benchmark will move the faults with the seed.
- CytoWeave's tools. The benchmark measures agents through CytoWeave. Analyses without an agent tool yet (cell-cycle and proliferation fits, index sorting, bead Q and B) are not among the tasks, and a tool's wording can mislead an agent: QC findings that call a clog "possibly a bubble" and a bubble "possibly a clog" led Claude Haiku to swap them in every run (and Sonnet and Opus in some runs of version 1). CytoWeave 0.8.0's findings name the cause from what the flow rate did at the same time, and on 0.8.0 every model named each problem right in every run; Haiku 5.5 is also a newer model than Haiku 4.5, so the two changes cannot be told apart.
- Prompts matter. In version 1, two prompts described their experiment in the words of an example's description, and agents opened that example, replacing the task's data with another copy of it, then answered about that. Version 2 starts every prompt by saying where the data are; a run that still analyzes another experiment is scored 0 and counted as such in the results.
- Few runs. Agents vary from run to run; results give the number of runs per task, and differences of a few points between models are within that variation.
Run it, or submit results
The benchmark is in the repository's benchmark folder. With Go, Chrome and Claude Code installed and logged in:
node benchmark/run.mjs --agent claude-code --model sonnet --repeat 3writes a results file to benchmark/results/. Other agents that speak MCP can be added as adapters in benchmark/harness.mjs. To submit results, open a pull request with the results file: it records the agent's program and model, the CytoWeave commit, every run's score, tool calls, tokens and time, and the reply.