Running at scale

A result's recipe is every step from the bundles to it. situcraft run makes it again without the window, on your computer, a server, a cluster's nodes or in a container; on the bundle it was made from, or on others, one bundle or a hundred, with a record of each run and a certificate anyone can check.

Recipes

In the Reports view's What made a result, Recipe saves the recipe of the result shown: each step from the bundles to it, the job that made it and its parameters, as readable JSON (agents and scripts ask for it with recipe). Nothing of your computer is in it: a bundle is named by its role (its folder's name), another file a step read (a table of labels, a reference) by its name, and what an earlier step made by that step. Its format is in Formats.

Some steps carry cells the window chose: a clustering of a filter's cells, a graph of every cell, an enrichment by each cell's label. On another bundle the program chooses them again, as the window did, when it can: every cell, a saved filter's cells (its conditions or its tree, whose label sets and clusterings the recipe makes too), and each cell's label of a label set the recipe makes. The recipe says so only when the program, choosing them on the bundle it was written from, gets the very cells the window chose. The cells of a region, those a review left out, curated or consensus labels and niches are the window's alone: such a step is made only on its own bundle.

situcraft run

$ situcraft run --recipe enrichment.recipe.json --bundle /data/lung-1 --out /results/lung-1

It works in a SituCraft of its own, with its library in the output folder (library/), and builds the bundle's derived data first, in a temporary folder (--cache keeps them for another run of the same bundle).
It makes each step in turn, and writes run.json: the recipe and the bundles by SHA-256, each step's result and files by SHA-256, and why it stopped when it did not finish. The output folder also holds certificate.zip, which situcraft verify checks.
Exit status: 0 made; 1 a step failed (or the run was stopped); 2 it could not run (a wrong command line, a bundle or file missing, a step it cannot make on that bundle). --dry-run checks the recipe, the bundle and the files without making anything.
A file the recipe reads is given with --file name=path, or found in a folder by its name and SHA-256 with --data.
The same recipe on the same bundle gives the same files on any number of threads (--threads, every core if not given) and on any computer. --clock 2026-01-01T00:00:00Z records everything as made at that time, so that two runs give the same certificate, byte for byte. A clustering's, a map's and labels' files also record how long they took; their content is the same.

The report

--report also writes, in a headless Chrome, Chromium, Edge or Brave (--chrome, or CHROME), what the window writes: the result's report (report.html, its numbers, methods and certificate in one file that opens offline) and its methods (methods.md), and the run's project in its library, with the bundle and every result the run made. To look at it in the window:

$ situcraft --data-dir /results/lung-1/library

Batches

Given several bundles, it makes the recipe on each, in a folder of its own under --out named after the bundle's, and writes batch.json: each bundle's exit status, reason, result and certificate's fingerprint. The bundles are every folder holding an experiment.xenium in a folder (--bundle-dir, by name), or those a file lists, one a line (--bundles; blank lines and lines starting with # are passed over). It goes on after a bundle that fails, and exits with 0 when every bundle was made, 2 when none could be, else 1.

$ situcraft run --recipe enrichment.recipe.json --bundle-dir /data/bundles --out /results/enrichment

Array jobs on a cluster

On a cluster, one task an array job's makes one bundle: --index chooses the nth bundle of the batch (from 1), and each task writes its own folder. A Slurm script for 96 bundles:

#!/bin/bash
#SBATCH --array=1-96
#SBATCH --cpus-per-task=8
#SBATCH --mem=16G
situcraft run --recipe enrichment.recipe.json --bundle-dir /data/bundles \
  --index "$SLURM_ARRAY_TASK_ID" --threads "$SLURM_CPUS_PER_TASK" \
  --out /results/enrichment

Other schedulers number their tasks the same way: $SGE_TASK_ID (Grid Engine), $PBS_ARRAY_INDEX (PBS Pro; $PBS_ARRAYID on Torque), $LSB_JOBINDEX (LSF). AWS Batch counts from 0: give --index $((AWS_BATCH_JOB_ARRAY_INDEX + 1)). The array's size is the batch's: --dry-run lists its bundles in order.

A scheduler that stops a task (its time is up, or it is canceled) sends it SIGTERM: the run stops the step under way, says so in run.json, and exits with 1.

Containers

SituCraft is one program with no dependencies: copy it into the image. Only --report needs a browser; install Chromium in the image for it.
In a container the browser runs without its sandbox when the run is root, and does not rely on /dev/shm, which containers keep small. On Linux it ends with the run, so a stopped run leaves no browser behind.
Give the run the cores the container has (--threads), and memory as the steps need it: a refinement's preset says how much (refinement; its Low memory preset runs on small machines and gives the same result).

Checking what was made

Each run's certificate.zip is checked as any certificate: situcraft verify --data /data/bundles /results/lung-1/certificate.zip makes each step again from the bundle and compares every file. A recipe made on the bundle it was written from gives the same results, by the same ids, as the window made: run.json says so for each step (sameAsRecipe).