Synthetic Pretraining | Vintage Data

Old data, new models

Synthetic Pretraining | Vintage Data Synthetic pretraining Pierre-Carl Langlais, February 1, 2026 Pretraining data infrastructure used to be the most conservative part of a fast-moving AI world. Since GPT-3 we have been mostly scaling the usual mix of web crawls peppered with a few more select sources (including, controversially, digitized books). This is finally changing. In 2025, several major releases used extensive synthetic datasets before mid-training happens: Minimax , Trinity , K2 /K2.5, Nemotron-3 and, more speculatively, GPT-OSS . At Pleias we even experimented with full synthetic tr

related reading

• Training as we know it will end | Vintage Datavintagedata.org
• Simulators — LessWronglesswrong.com
• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingdatologyai.com
• How to Generate and Use Synthetic Data for Finetuningeugeneyan.com
• Pretraining progress is mostly coming from datadwarkesh.com
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
• Alignment pretraining could backfire — LessWronglesswrong.com
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment — LessWronglesswrong.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Training Imperativesdan.io
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com