Error rates and scale
A test is only as good as its error rates. SituCraft measures its cohort tests on thousands of simulated cohorts whose truth is known: how often they find a difference that is not there, and how often they find one that is.
The study
Every week, SituCraft simulates 2,000 null cohorts: two groups of four subjects whose sections vary from one another as the simulated cohort's do, with no difference between the groups. It also simulates 500 cohorts with the standard simulated cohort's known effects. Each sample's true cells are labeled by their true types, summarized as the Cohort view summarizes them (see Sample summaries), and compared by condition as it compares them: abundance, expression and spatial statistics. A smaller study runs on every change to the program.
False positives
On the null cohorts, a test at 5% should find a difference in 5% of its tests, and a family adjusted by Benjamini and Hochberg should find anything in at most 5% of the cohorts:
| Family | Tests | Below 0.05 | Below 0.01 | Cohorts finding anything |
|---|---|---|---|---|
| Shares of the labeled cells | 8,000 | 4.9% | 1.0% | 4.8% |
| Cells per mm² | 8,000 | 5.0% | 1.1% | 5.5% |
| Niches' shares | 12,000 | 4.7% | 0.8% | 3.9% |
| Expression (each label's genes) | 703,142 | 5.1% | 1.0% | 5.8% |
| Neighbor enrichment | 20,000 | 5.5% | 1.2% | 6.4% |
| Nearest distances | 32,000 | 5.1% | 1.1% | 5.4% |
| Moran's I | 233,350 | 5.3% | 1.1% | 4.2% |
Each rate is within half a percentage point of its level. Neighbor enrichment's and Moran's I's are slightly above it: their statistics' variances are moderated toward one, and those of rare pairs and sparse genes vary more than the rest. The study once also tested the niches' shares by a moderated linear model of the spatial statistics, which found a difference in 6.0% of them. Niches' shares are now compared only as counts, in Differential abundance.
Cells are not replicates
The same null cohorts, tested as if each cell were a replicate (as a test of the groups' pooled cells does), find differences everywhere:
- A label's share, by a two-proportion test of the groups' cells: 58% of the tests below 0.05, and a difference found in 88% of the cohorts.
- A gene in a label, by the Wilcoxon rank-sum test of the groups' cells: 24% of the tests below 0.05, and genes found in every cohort.
The cells of a section share its subject, slide and run. The sections' own variation, which their cells cannot show, makes the groups differ by chance (Squair et al. 2021). SituCraft therefore never tests cells across groups. Every comparison is of the design's replicate units, and a design whose tests would be of cells is refused, saying why:
- a group with one unit;
- a unit in two rows of one group (its samples are combined instead);
- regions of one section given different units (see The design).
The tests of cells SituCraft does make are within one sample, such as a cluster's marker genes, and they are labeled exploratory.
Power and coverage
On the 500 cohorts with the standard cohort's effects, four subjects a group:
- A gene raised 2.5-fold in one type (SIM033 in the epithelial-like cells) is found in 98% of the cohorts. Its estimates average 1.33 log2 (the truth is 1.32), and its 95% interval covers the truth in 95.0% of them.
- The immune-like cells mixed into epithelial-like domains raise the two types' neighbor enrichment, which is found in 47% of the cohorts.
- The endothelial-like cells' odds doubled against each other type move their share of all labeled cells less (an odds ratio of 1.5 on average), and are found in 17%.
- Among the genes found, 6.3% on average are not the raised gene. That is close to the 5% false discovery rate the adjustment aims at.
Four subjects a group find a large change in a gene's expression nearly every time, but a modest change in a cell type's share or arrangement only sometimes. A larger cohort, or pairing, finds more. An interval says how much a cohort can tell.
At scale
Every week, 24 simulated sections of about 84,000 cells each (2 million cells) are summarized and compared one section at a time, as the summary job reads them: each section's graph, enrichment, nearest distances and Moran's I with the defaults the page offers, its summary, then the niches across the sections, the summary written and checked, and the three comparisons. On four cores of a laptop this takes 4.2 minutes, most of it each section's Moran's I and nearest distances, with a peak of 0.65 GB of memory. The same steps for 6 of the sections peak at 0.54 GB: the memory needed barely grows with the sections, because one is held at a time. What each section leaves for the summary is small: its numbers, its pseudobulk, and its cells' positions and labels for the niches across the sections. Summarizing, writing the summary and comparing the groups take under a second.
How it is checked
- Weekly: the study of 2,000 null cohorts and 500 with effects. Each family's rates must be within four standard errors (and a fifth of the level) of 5% and 1%; each family must find anything in at most 5% of the null cohorts, within the binomial margin; the raised gene's interval must cover its truth 95% of the time and the gene must be found in at least 80% of the cohorts. The cohort at scale must stay within its budget (20 minutes and 1.5 GB on four cores, its peak at most 1.5 times 6 sections').
- On every change: 24 null cohorts and 6 with effects, held to the same rates within their wider standard errors, and the cell-by-cell tests far above them.