Building Training Datasets for Single-Cell Foundation Models
Training a single-cell foundation model starts with a data problem. Most of the engineering effort we spent in the early months at Relation Therapeutics went into corpus curation, not model architecture. That ratio surprised us at first. It should not have. The model can only generalize what is in the training data, and what is in the training data is directly determined by the curation decisions you make before training starts.
Public single-cell datasets are abundant. CELLxGENE alone contains data from hundreds of studies. The problem is not raw availability. The problem is that publicly available datasets vary dramatically in quality, completeness of metadata, annotation consistency, and sequencing depth. Naive aggregation of everything produces a training corpus that is dominated by the most common tissue types and cell types in the public literature, underrepresents rare populations, and contains variable levels of technical noise from different experimental protocols.
The Quantity vs. Quality Tradeoff
Cell count is not a reliable proxy for training quality. A dataset of ten million cells from a single tissue type processed with a single protocol will train a model that knows that tissue well but generalizes poorly to other tissues. A dataset of two million cells drawn from diverse tissue types, disease states, and sequencing protocols will train a model with much broader generalization, even though it is smaller.
This seems obvious in principle but creates real operational tension. More cells means faster convergence during training and lower variance in the learned representations. Less diverse cells means overfitting to whatever biology is most represented in the corpus. Neither failure mode is immediately obvious during training. The diversity problem only appears when you apply the model to cell types or tissues that were underrepresented during training, which you often discover during evaluation rather than training.
The approach we settled on was to treat corpus composition as a first-class design decision with explicit targets: representation targets for tissue type diversity, cell type diversity, donor diversity, and disease state diversity. These targets guide which datasets to include and, where specific categories are underrepresented, guide the decision to seek out or prioritize datasets from underrepresented contexts.
Source Dataset Selection Criteria
Not every public single-cell dataset is suitable for foundation model training. The minimum bar for inclusion covers several dimensions.
Metadata completeness matters more than most groups initially expect. A dataset without consistent tissue labels, donor metadata, or cell-type annotations is difficult to incorporate in a way that the model can learn from. Metadata is how you tell the model what biological context a cell came from. Without it, the model may learn to represent the technical variation between experiments rather than the biological variation between cell types.
Sequencing depth affects what the model can learn about each cell. Very shallow sequencing, fewer than 500 genes detected per cell in typical 10x Chromium data, produces sparse profiles where most of the variation is technical dropout rather than genuine biology. We apply a minimum gene detection threshold during dataset screening, but the threshold is protocol-dependent. What constitutes adequate depth for Drop-seq is different from what constitutes adequate depth for Smart-seq2.
Cell viability filtering is a step that some datasets have applied carefully and others have not. Damaged cells, empty droplets, and doublets all look like cells but do not represent genuine single-cell biology. Rigorous QC before training removes these artifacts. If source datasets have not applied rigorous QC, it needs to be applied at ingestion time, which means building QC reproducibly into the data pipeline rather than relying on upstream processing.
Annotation Harmonization
Cell type labels across public datasets are inconsistent. Two studies from the same tissue type often use different names for the same cell population, or use the same name for populations that differ in important ways. Training a model on labels as-is means training on the inconsistency, which degrades the model's ability to represent cell type identity reliably.
Harmonization means mapping source dataset labels to a controlled vocabulary. The Cell Ontology is the standard here, but it is not granular enough to capture the fine-grained subtypes that many modern single-cell studies identify. We use a hierarchical approach: broad Cell Ontology terms for the coarse level, study-specific fine-grained terms for the subtype level, with explicit mappings and notes on where the mapping is uncertain.
The uncertain mappings are important to track. A cell labeled "activated macrophage" in one study and "inflammatory monocyte-derived macrophage" in another might refer to the same population or to genuinely distinct populations. Treating them as identical in training may collapse a meaningful distinction. Treating them as entirely distinct may artificially inflate the apparent diversity of macrophage subtypes in the corpus. Both errors have downstream consequences for what the model learns about macrophage biology.
Handling Batch Effects in the Training Corpus
A training corpus assembled from hundreds of studies is, definitionally, heavily batched. Each study is a batch in the statistical sense: a distinct combination of experimental protocol, reagent lot, processing facility, and analysis pipeline. Training a model on this corpus without accounting for batch variation produces a model that has learned, at least partially, to distinguish batches rather than cell types.
The standard approach is to include batch covariates explicitly during training, so the model can learn batch-invariant representations. This works well when the batch structure is known and documented, which requires accurate metadata about which study and which processing protocol each cell came from. It works less well when metadata is incomplete or when batch structure is confounded with biological variation, which happens when a single study has both unique cell types and a unique processing protocol.
We treat batch effect handling as a continual process rather than a one-time preprocessing step. As new datasets are added to the corpus, they introduce new batch effects. The model retraining pipeline includes batch mixing diagnostics that check whether the new version of the model represents cell types consistently across batches of origin, not just on the aggregate metrics.
Rare Cell Type Coverage
Rare cell types are the hardest curation challenge. A population that constitutes half a percent of cells in any given tissue sample will appear in only a handful of cells per dataset, even in large studies. Across hundreds of datasets, the total count might be several thousand cells. That is enough to train a recognizable representation, but the variance is high and the generalization to new samples is lower than for common cell types.
The curation response is to actively seek datasets with enriched rare cell type content. Some studies profile specific rare populations using FACS enrichment before sequencing. Others focus on conditions where a rare cell type transiently expands. Including these studies gives the model more examples of rare biology than passive corpus assembly would provide.
The tradeoff is a mild overrepresentation of enriched datasets for rare cell types, which needs to be balanced against the natural distribution so the model does not learn to expect rare cells at higher frequency than they actually occur in tissue. We weight training examples during corpus assembly to maintain approximate proportionality while ensuring rare cell types have enough representation to train a reliable encoder.
The Curation Process Is Ongoing
The single-cell literature publishes new datasets continuously. Atlas projects are expanding their tissue and disease coverage. Building a training corpus is not a one-time event that produces a fixed dataset. It is a continuous process of evaluation, ingestion, harmonization, and retraining.
What this means practically is that corpus curation infrastructure is as important as model architecture. Reproducible pipelines for ingestion, QC, harmonization, and staging of new datasets reduce the engineering cost of keeping the training corpus current. Without that infrastructure, adding new data becomes a manual effort that either does not happen, or happens inconsistently in ways that introduce regressions.
The connection between corpus quality and model performance is direct. We learned early on that a poorly curated dataset included in training does not just fail to help, it can actively degrade the model's representation of the cell types that dataset was supposed to add. That experience made quality checking at ingestion a hard gate in the pipeline rather than a soft suggestion.
