[2603.10055] Training Language Models via Neural Cellular Automata

Abstract:Pre-training is crucial for large language models (LLMs), as it is when most representations and capabilities are acquired. However, natural language pre-training has problems: high-quality text is finite, it contains human biases, and it entangles knowledge with reasoning. This raises a fundamental question: is natural language the only path to intelligence? We propose using neural cellular automata (NCA) to generate synthetic, non-linguistic data for pre-pre-training LLMs--training on synthetic-then-natural language. NCA data exhibits rich spatiotemporal structure and statistics resembling natural language while being controllable and cheap to generate at scale. We find that pre-pre-training on only 164M NCA tokens improves downstream language modeling by up to 6% and accelerates convergence by up to 1.6x. Surprisingly, this even outperforms pre-pre-training on 1.6B tokens of natural language from Common Crawl with more compute. These gains also transfer to reasoning benchmarks, including GSM8K, HumanEval, and BigBench-Lite. Investigating what drives transfer, we find that attention layers are the most transferable, and that optimal NCA complexity varies by domain: code benefits from simpler dynamics, while math and web text favor more complex ones. These results enable systematic tuning of the synthetic distribution to target domains. More broadly, our work opens a path toward more efficient models with fully synthetic pre-training.

[2603.10055] Training Language Models via Neural Cellular Automata Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Machine Learning arXiv:2603.10055 (cs) [Submitted on 9 Mar 2026] Title: Training Language Models via Neural Cellular Automata Authors: Dan Lee , Seungwook Han , Akarsh Kumar , Pulkit Agrawal View a PDF of the paper titled Training Language Models via Neural Cellular Automata, by Dan Lee and 3 other authors View PDF HTML (experimental) Abstract: Pre-training is crucial

related reading

• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• Large Language Diffusion Modelsarxiv.org
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Generalized Language Models | Lil'Loglilianweng.github.io
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment — LessWronglesswrong.com
• Self-Adapting Language Modelsarxiv.org
• Recent Advances in Language Model Fine-tuningruder.io
• Synthetic Pretraining | Vintage Datavintagedata.org
• Language Modelinglena-voita.github.io
• Modern Pretraining Strategies: A Hands-On Guidetheneuralmaze.substack.com
• [2502.19402] General Reasoning Requires Learning to Reason from the Get-goar5iv.labs.arxiv.org
• Understanding Large Language Modelsmagazine.sebastianraschka.com