Agent benchmark
AI agents can drive SituCraft through its MCP server. This benchmark measures whether they get the analysis right: tasks worded as a user would ask them, on simulated tissues and cohorts whose every cell is known and on the Lung bundle, with nothing but SituCraft's tools, graded by code.
What it measures
Each task is a request a user could make of an agent: label every cell with its type, find which cell types neighbor each other, find the sample with a quality problem, find the gene a treatment changes, compare groups that do not differ, read a paired design, count and save cells by their transcripts, read a run's quality, make a figure and write its methods, certify an analysis so others can check it. The simulated tissues and cohorts are made for each task with a seed of their own, and where they carry a planted effect it is moved from SituCraft's examples (another gene, cell type or sample), so that an answer cannot be recalled from the examples' descriptions. The simulator's truth (every cell's true type, the planted effect, the sample planted far out) is read and then taken out of the bundles before the agent starts, so that no tool can open it. The Lung tasks' answers come from the bundle's own CSV copies of its tables and its metrics, read independently of SituCraft's readers.
Grading is deterministic: code reads the answer line the task asks for, what the agent left in the project (its label sets, filters, comparisons, analyses) through the same tools, and the files it wrote, and gives partial credit by a fixed rubric. No model judges another.
Results
The score is the mean over tasks of each task's mean score over runs. The reference solution, scripted through the same tools without a model, shows that every task can be solved with SituCraft's tools; an agent that answers nothing scores 0%. Models' runs are added as they are made, each with the model's version and Claude Code's own estimate of its cost (on a subscription, usage of the plan).
| Agent and model | Score | Tasks × runs | Answered | Tool calls per run | Cost per run | Run on | SituCraft |
|---|---|---|---|---|---|---|---|
| Reference solution (scripted, no model) | 100% | 10 × 1 | 10 of 10 | 4.4 | — | 2026-10-21 | 0.9.0 (78897c0) |
| No answer (lower bound) | 0% | 10 × 1 | 0 of 10 | 0.0 | — | 2026-10-21 | 0.9.0 (78897c0) |
By task
| Task | Reference solution (scripted, no model) |
|---|---|
| Label every cell with its type cell types | 100% |
| Find the two cell types that neighbor each other spatial | 100% |
| Find the sample with a quality problem cohort QC | 100% |
| Find the gene a treatment changes differential expression | 100% |
| Compare groups that do not differ statistics | 100% |
| Find the cell type that changes after treatment study design | 100% |
| Count and save the endothelial cells of a lung section querying cells | 100% |
| Read a run's quality quality control | 100% |
| Make a figure of the cell types and write its methods reporting | 100% |
| Certify an analysis so others can check it the record | 100% |
How a run works
- The harness builds SituCraft from the benchmark's commit and starts its MCP server with an empty library, its page in a headless Chrome. It makes the task's data (
situcraft simulatewith the task's seed and parameters, or the Lung bundle) and opens it as the project "Experiment", a cohort's samples with their subjects and conditions, every bundle's derived data built. - The agent is started as a program of its own, in an empty folder, with SituCraft's MCP server as its only tools, every leave given (edits, compute, files): no shell, files or web of its own. Claude Code runs with
--tools "",--strict-mcp-config, the settings of that empty folder only (not the user's hooks, plugins or defaults) and the model and effort given; each run is a fresh session. - The agent gets the task's prompt, which starts as every one does: the data are already open in SituCraft, in the project "Experiment". It works through SituCraft's tools in the page the harness watches and ends its reply with the answer line the prompt asks for. A run stops after 60 turns or 20 minutes.
- The harness grades the answer, the project and the files in the task's output folder, and records the score with its rubric, the tool calls, the tokens, the cost and the time. A run that opened other data (an example, say) is counted as such.
Tasks
| Task | Data | Graded against |
|---|---|---|
| Label every cell with its type | A simulated tissue, 500 µm square, its types in uneven shares (6% to 61%) | A label set in the project against every cell's true type (full credit from 90% of the cells), and each type's percentage (within 3 points) |
| Find the two cell types that neighbor each other | A simulated tissue, fibroblast-like cells planted six times as likely where immune-like cells lead | The planted pair, and a neighborhood enrichment in the project |
| Find the sample with a quality problem | A simulated cohort of eight subjects, the third sample detecting 40% of its transcripts | That sample and no other, what is wrong with it, and the samples summarized in the project |
| Find the gene a treatment changes | A simulated cohort of eight subjects, SIM072 raised three times in the treated subjects' fibroblast-like cells | The gene, its cell type, its direction, no other change claimed, and the groups' expression compared in the project |
| Compare groups that do not differ | A simulated cohort of eight subjects without any effect | Abundance and expression both compared, and no change claimed in either |
| Find the cell type that changes after treatment | A simulated cohort of six subjects before and after, the immune-like cells' odds a quarter after | The cell type, its direction, and a comparison pairing each subject with itself |
| Count and save the endothelial cells of a lung section | The Lung 2-FOV bundle (10x Genomics) | The cells with two VWF or two PLVAP transcripts and no EPCAM (within 1%), those of them with ACE, and a filter keeping exactly them (by the Jaccard index), all from the bundle's transcripts CSV |
| Read a run's quality | The Lung 2-FOV bundle | The cells detected and median transcripts per cell as the run recorded them, and the share of cells under 30 transcripts from the bundle's cells CSV |
| Make a figure of the cell types and write its methods | A simulated tissue | The PDF written where asked, read back by SituCraft as its own figure of the project, unchanged, its cells filled by their labels; the methods written, citing how the cells were labeled |
| Certify an analysis so others can check it | A simulated tissue | The certificate written where asked, of a neighborhood enrichment, and verified by situcraft verify, every step made again |
Before results are published, the benchmark checks itself: every reference solution must score at least 90% (all score 100%), an agent that answers nothing must score 0, and unit tests give each grader right, partly right and wrong answers. These checks run in continuous integration with the rest of the validation.
Limits
- Mostly simulated data. The truth is the simulator's: four cell types named for what they resemble, marker genes the simulator chose. Real tissue is messier, and an agent good here may still be wrong on a real one; SituCraft's own analyses are validated on real data separately (Science).
- The simulator is public. Its code is on GitHub. Seeds change every task's cells, and the planted effects are moved from the examples', but a model could still know how the simulator names its types and which genes it makes markers.
- The Lung tasks read the run. Their answers are counts and metrics, known exactly from the bundle's files; they test whether an agent asks SituCraft the right question, not its biology.
- SituCraft's tools. The benchmark measures agents through SituCraft: segmentation and refinement, image alignment and region review have no task yet, and a tool's wording can lead an agent astray as well as help it.
- Few runs. Agents vary from run to run; results give the number of runs per task, and differences of a few points between models are within that variation.
Run it, or submit results
The benchmark is in the repository's benchmark folder. With Go, Chrome, the Lung bundle (node validation/fetch.mjs xenium-lung-2fov) and Claude Code installed and logged in:
$ node benchmark/run.mjs --agent claude-code --model sonnet --repeat 3writes a results file to benchmark/results/. Other agents that speak MCP can be added as adapters in benchmark/harness.mjs. To submit results, open a pull request with the results file: it records the agent's program and model, the SituCraft commit, every run's score with its rubric, tool calls, tokens, cost and time, and the reply.