building a bigger qwen out of two qwen models

a crash course in model chimerism

tl;dr: Take a giant Qwen3 MoE model, collapse its experts into a dense stack that matches Qwen3-8B’s width, then interleave those dense MoE blocks with the 8B blocks to create a deeper hybrid. Validate that the transplanted blocks actually matter, fix the ‘hot’ ones, and end with a model that’s both larger and reasonably sane. the layers of the final model Why does this work at all? Qwen3-8B and Qwen3-235B-A22 share the same hidden size and attention head dimension. If hidden size matches, you can line up linear layers across both networks without inventing sketchy adapters. The other…

related reading

• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Qwen3: Think Deeper, Act Faster | Qwenqwenlm.github.io
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Quantization from the ground upngrok.com
• Cross-Model KV Cache Transfer in LLM Families:A Closed-Form Linear Mapping for Prefill Reusearxiv.org
• ali (@waterloo_intern) on Xx.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Papers I’ve read this week, Mixture of Experts editionfinbarrtimbers.substack.com
• Tensor-Transformer Variants are Surprisingly Performant — LessWronglesswrong.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• How trillion-parameter AI models work and how to serve themyoutube.com
• Extending Context is Hard | kaiokendevkaiokendev.github.io