ZeitgeistLM

ZeitgeistLM — a time-conditioned language model trained on Reddit memes.

Abstract We train ZeitgeistLM, a 124M-parameter decoder-only transformer from scratch on timestamped Reddit submissions and comments. ZeitgeistLM uses ordinary causal next-token prediction, augmented with a learned continuous-time prefix embedding projected from a scalar timestamp and prepended to the token sequence. This lets the model condition every prediction on time, interpolate smoothly between observed dates, and extrapolate beyond the training interval — capabilities that discrete learned time tokens do not naturally provide. The inclusion of this prefix embedding provides a single…

saved by

• Harihar Prasad

related reading

• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• Esoteric Language Modelsarxiv.org
• Can gzip be a language model?nathan.rs
• Productizing Large Language Modelsblog.replit.com
• Large Language Diffusion Modelsarxiv.org
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
• A History of Large Language Modelsgregorygundersen.com
• [2302.13939] SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networksarxiv.org
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io
• Diffusion will be everywherevarunneal.github.io