Foundation Models: Masked Language Modeling

This page describes the Foundation Models: Masked Language Modeling row of the Papers.md matrix: pretrained, transferable models trained with a masked, BERT-style objective, where part of the input is hidden and the model reconstructs it from the surrounding context. The row’s authoritative scope is its Taxonomy.md definition; this page synthesizes what currently sits in it.

Scope boundary

Masked pretraining at scale plus downstream transfer is the whole test. It separates this row from Next-Token Prediction, which differs only in objective, and from every architecture row, because a model here is defined by how it was pretrained rather than by what it is made of. A BERT-style encoder trained end-to-end on one labelled task is not in this row. A paper that merely invokes such a model, as an agent’s tool call or a queried service, keeps its own row rather than earning this one.

The row’s recurring intellectual problem is that a cell has no word order. Genes are not a sequence, so masked pretraining borrowed from text needs a substitute for position, and each paper below solves it differently: gene2vec embeddings as a relative encoding, rank-value ordering, or learned value embeddings with no discretization at all. The variety is the row’s real content.

Cellular Engineering

#112 scBERT (Yang et al. 2022, Nature Machine Intelligence, Tencent AI Lab): pretrains on unlabelled scRNA-seq and fine-tunes for cell-type annotation, aimed at three named weaknesses of prior annotation methods: dependence on curated marker-gene lists, poor batch-effect handling, and inability to use gene-gene interaction structure. Two design decisions carry it. Because shuffling gene columns does not change a cell’s meaning, absolute position embeddings are meaningless, so gene2vec embeddings are used instead as a relative encoding that already carries semantic similarity between genes. And because vanilla attention is quadratic while scRNA-seq can exceed 20,000 genes, the backbone is a Performer rather than a standard transformer. Expression is discretized by binning into 200-dimensional token embeddings. The paper reports results on annotation, novel-cell-type discovery, batch robustness and interpretability. Code at TencentAILabHealthcare/scBERT; catalogued in Software.md. Its later re-evaluation (#113) and the authors’ reply (#114) are covered in Benchmarks & Evaluation Frameworks, and are worth reading before taking any number here at face value.
#116 scFoundation (Hao et al. 2024, Nature Methods, BioMap Research): 100 million parameters over about 20,000 genes, pretrained on more than 50 million human single-cell profiles collected from GEO, the Human Cell Atlas, Single Cell Portal and EMBL-EBI, deduplicated by study DOI and unified to 19,264 HGNC protein-coding and mitochondrial genes. It departs from its predecessors by refusing to discretize: the embedding module converts each expression scalar directly into a learned value embedding, as a weighted sum over a set of embeddings with weights learned from the scalar, avoiding binning entirely. The architecture is deliberately asymmetric, a large vanilla-transformer encoder that sees only the non-zero, non-masked genes and a small Performer decoder that sees all of them, which the authors’ ablations credit for the memory savings. Reported across expression enhancement, tissue and single-cell drug response, perturbation prediction, annotation and gene-module inference. Code at biomap-research/scFoundation; catalogued in Software.md.
#235 CellFM (Zeng et al. 2025, Nature Communications): 800 million parameters trained on 100 million human cells gathered from GEO, ENA, GSA and ImmPort between April 2021 and August 2023, standardized through the SynEcoSys workflow with a 200-gene-per-cell floor and HGNC symbol normalization. The backbone is a modified RetNet rather than a transformer, built on MindSpore, with LoRA used to cut the parameter count during adaptation, and each cell truncated or padded to a fixed 2,048 genes with the highest-expressed retained. The masking rate is stated plainly at 20% of genes. Reported to outperform existing models on annotation, perturbation prediction, gene-function prediction and gene-gene relationship capture.

All three corpora are human. Applying any of them to bovine, porcine or piscine cells is transfer beyond what the papers evaluate, which is why the cross-species models in LM + Biological Priors and Next-Token Prediction matter disproportionately for cellular agriculture.

Sensory Prediction

#322 Umami-BERT (Zhang et al. 2023, Food Research International): predicts umami peptides from amino-acid sequence using a two-stage strategy that is a clean illustration of why this row exists. Stage one pretrains attention over a large, heterogeneous pool of bioactive peptides assembled from six sources (anticancer peptides from UCI, neuropeptides from NeuroPedia, anti-tubercular peptides from AntiTbPdb, fermented-food peptides from FermFooDb, food-derived bioactives from DFBP, and signalling peptides from PeptideDB), none of which is labelled for umami. Stage two re-trains on UMP789, just 275 umami and 514 non-umami sequences. Reported accuracy 93.23% and MCC 0.78 on the balanced set, 95.00% and MCC 0.85 on the unbalanced one. The attention weights are then read directly, identifying alanine, cysteine, aspartate and glutamate as the largest contributors, so the transfer buys interpretability as well as accuracy on a dataset far too small to train from scratch. Directly relevant to savoury-flavour work in cultivated meat.

AI Tooling / Methodology

#6 DNABERT (Ji et al. 2021, Bioinformatics): applies masked pretraining to genomic DNA, tokenizing sequence as overlapping k-mers and pretraining on 10-to-510-length windows sampled from the human genome. The adaptation from text is explicit: next-sentence prediction is removed, sequence length is adjusted, and instead of masking scattered single tokens the model masks contiguous runs of k tokens covering 15% of the sequence, because adjacent k-mers overlap and independent masking would leak the answer. Fine-tuned with small labelled sets for promoter, splice-site and transcription-factor-binding-site prediction, with nucleotide-level attention that surfaces conserved motifs and candidate functional variants, and the authors show the human-pretrained model transfers to other organisms. That transferability is the cell-ag hook: reading regulatory grammar in a livestock genome without pretraining a new model. Code at jerryji1993/DNABERT.
#111 Geneformer (Theodoris et al. 2023, Nature, Broad Institute and Gladstone): pretrained on Genecorpus-30M, 29.9 million human single-cell transcriptomes drawn from 561 datasets, with two curation choices that shape what the model can be used for. Cells with high mutational burden, malignant cells and immortalized lines, were excluded because network rewiring without companion genome sequencing would be uninterpretable. And only droplet-based platforms were included, so expression units are comparable across the corpus. Cells are rank-value encoded rather than binned. The paper’s central claim is about data efficiency: fine-tuning on limited task-specific data consistently improved accuracy across chromatin and network-dynamics tasks, and applied to cardiomyopathy with limited patient data it nominated candidate therapeutic targets. That argument, pretraining as a substitute for scarce labels, is the one most directly transferable to livestock cell biology. Distributed at Hugging Face; catalogued in Software.md, corpus in Datasets/HumanReference.md.
#117 scGPT (Cui et al. 2024, Nature Methods): described under Cell-State & Perturbation Prediction. It is in this row and in Next-Token Prediction because it uses both objectives.

Adjacent methods

Foundation Models: Next-Token Prediction: the autoregressive alternative, and the direction the field moved after this wave.
Foundation Models: Cell-State & Perturbation Prediction and LM + Biological Priors: the target-defined and prior-injecting foundation-model rows.
Deep Learning: where a task-specific network belongs, however transformer-shaped.
Benchmarks & Evaluation Frameworks: the scBERT re-evaluation and reply, and the broader question of whether these models beat simple baselines.