CpGPT: a Foundation Model for DNA Methylation

Abstract

DNA methylation is a critical epigenetic modification that regulates gene expression and plays a significant role in development and disease processes. Here, we present the Cytosine-phosphate-Guanine Pretrained Transformer (CpGPT), a novel foundation model pretrained on over 1,500 DNA methylation datasets encompassing over 100,000 samples from diverse tissues and conditions. CpGPT leverages an improved transformer architecture to learn comprehensive representations of methylation patterns, allowing it to impute and reconstruct genome-wide methylation profiles from limited input data. By capturing sequence, positional, and epigenetic contexts, CpGPT outperforms specialized models when finetuned for aging-related tasks, including chronological age prediction, mortality risk, and morbidity assessments. The model is highly adaptable across different methylation platforms and tissue types. Furthermore, analysis of sample-specific attention weights enables the identification of the most influential CpG sites for individual predictions. As a foundation model, CpGPT sets a new benchmark for DNA methylation analysis, achieving strong performance in the Biomarkers of Aging Challenge, where it placed second overall in chronological age estimation and first on the public leaderboard in methylation-based mortality prediction.

HighlightsO_LICpGPT is a novel foundation model for DNA methylation analysis, pretrained on over 1,500 datasets encompassing 100,000+ samples. C_LIO_LIThe model demonstrates strong performance in zero-shot tasks including imputation, array conversion, and reference mapping. C_LIO_LICpGPT achieves state-of-the-art results in mortality prediction and chronological age estimation. C_LIO_LISample-specific interpretability is enabled through analysis of attention weights. C_LI