Have We Hit the Scaling Wall for Protein Language Models?

pascalnotin.substack.com · 3,253 words · saved by 1 readers

This is not a Gary Marcus-style post for protein language models. I personally hope the answer is no, and there are a few emerging ideas (see end of post) with the potential to overcome current limitations. But, for now, naively scaling pLMs has led to underwhelming performance on the tasks that matter. We just released ProteinGym v1.3, now featuring over 90 baseline models (including recent models Progen3, ESM3/C, and xtrimoPGLM) The new leaderboard tells a clear story: multimodal models that combine Multiple Sequence Alignments (MSAs) and structure outperform others for zero-shot fitness predictions. Both modalities are useful in different settings Even very simple methods leveraging these two modalities significantly outperform billion-parameter sequence models Scaling protein language models (pLMs) does not seem to help beyond 1-4B parameters We conclude with some thoughts on what the field should focus on going forward to continue driving progress The protein sequence-function rel

Have We Hit the Scaling Wall for Protein Language Models? Beyond Scaling: What Truly Works in Protein Fitness Prediction Pascal Notin May 07, 2025 30 8 2 Share Disclaimer This is not a Gary Marcus-style post for protein language models. I personally hope the answer is no, and there are a few emerging ideas (see end of post) with the potential to overcome current limitations. But, for now, naively scaling pLMs has led to underwhelming performance on the tasks that matter. TLDR We just released ProteinGym v1.3, now featuring over 90 baseline models (including recent models Progen3, ESM3/C, and x

Have We Hit the Scaling Wall for Protein Language Models?

related reading