Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs

I got nerdsniped when I first heard about Semantic IDs. The idea is simple: Instead of using random hash IDs for videos or songs or products, we can use semantically meaningful tokens that an LLM can natively understand. I wondered, could we train an LLM-recommender hybrid on the rich behavioral data that makes today’s recommender systems so effective? To my surprise, we can! The result is a language model that can converse in both English and item IDs, not with retrieval or other tools, but as a single, “bilingual” model where items (i.e., semantic IDs) are part of its vocabulary. Like a recommender model, it can recommend items given historical interactions. But the big surprise—and capability unlock—was when I found that I could simply chat with the model to steer its recommendations, and it could reason about its choices, offer explanations, and creatively name product bundles. Here’s a demo video. And here’s the code to prep the data, train the model, and chat with it. (Note: This

Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs eugeneyan Start Here Writing Speaking Prototyping About Training an LLM-RecSys Hybrid for Steerable Recs with Semantic IDs [ llm recsys learning 🛠 🩷 ] · 23 min read I got nerdsniped when I first heard about Semantic IDs. The idea is simple: Instead of using random hash IDs for videos or songs or products, we can use semantically meaningful tokens that an LLM can natively understand. I wondered, could we train an LLM-recommender hybrid on the rich behavioral data that makes today’s recommender systems so effective? To my surpr

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Read in smrysmry.ai
• LLM Daydreaming · Gwern.netgwern.net
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• GenAI Handbookgenai-handbook.github.io
• Beyond “People Also Liked”: Building Intelligent Food Recommenders with LLMs | by Faisal Hussain Sabir | Mediummedium.com
• “Next-token predictor” is the wrong mental model for LLMsgmcgoldr.github.io
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• How LLMs Actually Work | 0xkato0xkato.xyz
• Embeddings: What they are and why they mattersimonwillison.net
• OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignmentarxiv.org
• Llama 2 · Hugging Facehuggingface.co