Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs

I got nerdsniped when I first heard about Semantic IDs. The idea is simple: Instead of using random hash IDs for videos or songs or products, we can use semantically meaningful tokens that an LLM can natively understand. I wondered, could we train an LLM-recommender hybrid on the rich behavioral data that makes today’s recommender systems so effective? To my surprise, we can! The result is a language model that can converse in both English and item IDs, not with retrieval or other tools, but as a single, “bilingual” model where items (i.e., semantic IDs) are part of its vocabulary. Like a recommender model, it can recommend items given historical interactions. But the big surprise—and capability unlock—was when I found that I could simply chat with the model to steer its recommendations, and it could reason about its choices, offer explanations, and creatively name product bundles. Here’s a demo video. And here’s the code to prep the data, train the model, and chat with it. (Note: This

Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs eugeneyan Start Here Writing Speaking Prototyping About Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs [ llm recsys learning 🛠 🩷 ] · 23 min read I got nerdsniped when I first heard about Semantic IDs. The idea is simple: Instead of using random hash IDs for videos or songs or products, we can use semantically meaningful tokens that an LLM can natively understand. I wondered, could we train an LLM-recommender hybrid on the rich behavioral data that makes today’s recommender systems so effective? To my surpr

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Read in smrysmry.ai
• Deep Dive into LLMs like ChatGPTyoutube.com
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• What language do language models speak?tcz.hu
• Auditing language models for hidden objectives — LessWronglesswrong.com
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com