Foundation Models: Next-Token Prediction

This page describes the Foundation Models: Next-Token Prediction row of the Papers.md matrix: large pretrained models trained with an autoregressive, GPT-style objective and applied to biology. The row’s authoritative scope is its Taxonomy.md definition; this page synthesizes what currently sits in it.

Scope boundary

Three things have to hold. The objective must be autoregressive, predicting the next token given those before it, which separates this row from Masked Language Modeling where the model fills in a hidden token from both sides. The model must be a pretrained foundation model rather than a task-specific network, which is the same test every foundation-model row applies. And the paper must have built or adapted that model rather than called it: invoking one as an agent’s tool call or a queried service keeps the paper in its own row.

The interesting wrinkle is that “next token” needs a sequence, and a cell is not one. Each paper here answers that differently: rank the genes by expression and call the ordering a sentence, or make the sampling order itself the generative process. Reading them together is really a survey of how the field turned an unordered expression vector into something a language model can consume. Some references appear in more than one foundation-model row, because scGPT in particular uses both objectives.

Cellular Engineering

#92 TranscriptFormer (Pearce et al. 2026, Science, Chan Zuckerberg Initiative): a family of generative models trained on up to 112 million cells across 12 species spanning roughly 1.53 billion years of evolution. It treats a cell as a “bag of transcripts” and defines the generative process explicitly: sample a gene from a categorical distribution conditioned on the genes already selected, then sample its count from a zero-truncated Poisson. Gene embeddings come from ESM-2, an assay token encodes the sequencing technology, and attention is expression-aware, with counts entering as bias terms so highly expressed genes weigh more. Three variants differ only in training data (TF-Metazoa across 12 species, TF-Exemplar across 5 model organisms, TF-Sapiens human only), trained on roughly 3.5 trillion tokens with low-resource species upweighted. The authors report state-of-the-art cell-type classification even across species separated by 685 million years, and note that developmental trajectories and phylogenetic relationships emerge in the representations without being trained on. The species span is why this is the most cell-ag-relevant model in the row. Catalogued in Software.md.
#115 tGPT (Shen et al. 2023, iScience): earlier and smaller, and methodologically the origin point for this row in single-cell work. Genes are sorted by descending expression and the resulting list is fed to an autoregressive model as a token sequence bracketed by start and end symbols. Trained on 22.3 million single cells plus bulk transcriptomes from TCGA, GTEx and a urothelial-carcinoma trial, with mitochondrial, ribosomal and non-protein-coding genes discarded from the single-cell data, and evaluated against author-provided cell labels on datasets including the Human Cell Atlas immune census, the Human Cell Landscape and Tabula Muris. Its significance is the demonstration that a next-token objective works at all for single-cell biology, the lineage that later includes State and Cell2Sentence. Code at deeplearningplus/tGPT; catalogued in Software.md.
#120 C2S-Scale (Rizvi et al. 2026, bioRxiv): takes the rank-ordering idea to its conclusion by making the cell sentence literal text and training a 27-billion-parameter LLM on it alongside biological prose and metadata. Described in full under Cell-State & Perturbation Prediction, which is its target; it is here for its objective and in Reinforcement Learning for its fine-tuning.

Sensory Prediction

#285 (Steuer et al. 2026, npj Science of Food, Di Pizio lab): designs bitter and non-bitter peptides de novo, which matters directly for cultivated-meat and alternative-protein ingredients because protein hydrolysis releases bitter peptides unpredictably. The generative half is ZymCTRL, a conditional protein language model built for generating artificial enzymes by EC class, fine-tuned on 478 sensory-validated bitter sequences curated from the BPS-1000 database with mixed-report peptides excluded. Conditioning is done with a neat trick: an unassigned BRENDA EC number, 1.2.3.16, is assigned arbitrarily to every bitter sequence so the model has a class token to generate against. It produced 1,504 novel sequences, which were filtered by BitterPep-GCN, a graph convolutional classifier, and a second library of every overlapping tripeptide was generated from BPS-1000 by sliding window. Of 31 peptides purchased and put to an expert sensory panel, 25 were correctly classified, 15 confirmed bitter and 10 confirmed non-bitter. This is one of very few references in the matrix where a generative model’s output was synthesized and tasted. Also in GNN for the classifier. Code at DiPizio-Lab/bitter-peptide-design; catalogued in Software.md.

AI Tooling / Methodology

#88 SpectraLLM (Su et al. 2026, arXiv): predicts molecular structure end-to-end by reasoning over spectra, representing continuous modalities (IR, Raman, UV-Vis, NMR) and discrete ones (MS) in one shared language space rather than running a separate pipeline per modality. Trained and evaluated on QM9s, the Multimodal Spectroscopic dataset, MassSpecGym and MassBank, and scored not only on exact structure recovery but on functional-group overlap, Tanimoto similarity over ECFP4 and MACCS fingerprints, maximum common edge substructure and Fraggle similarity, so partial correctness is measured rather than discarded. It beats single-modality baselines and improves further when several spectra are supplied together, with the paper’s case studies showing Raman correcting IR-only errors on polarizability-dependent substructures and IR being necessary for carbonyl identification. Relevant to any sensomics workflow that has to get from a spectrum to a compound identity. No cell-ag application, hence the tooling column.
#117 scGPT (Cui et al. 2024, Nature Methods): described under Cell-State & Perturbation Prediction. It appears in this row and in Masked Language Modeling because its pretraining uses both objectives, which is a good reminder that these rows classify objectives rather than partitioning models.

Adjacent methods

Foundation Models: Masked Language Modeling: the other pretraining objective, and the one that dominated the first wave of single-cell foundation models.
Foundation Models: Cell-State & Perturbation Prediction: where these models’ downstream target is classified.
Foundation Models: LM + Biological Priors: protein-language-model tokenization, which TranscriptFormer also uses.
Foundation Models (other modalities): pretrained models over images, spectra and molecules rather than transcriptomes.
GNN: the classifier half of the bitter-peptide design loop.
Benchmarks & Evaluation Frameworks: how these models are scored, and the dispute over whether the scores hold up.