Generalizing an LLM from 8k to 1M Context using Qwen-Agent
We’ve created an agent using Qwen2 models with an 8k context size to understand documents with 1M tokens, surpassing RAG and native long-context models. This agent was also used to generate data for training new long-context Qwen models.
Qwen-Agent TLDR: We've created an agent using Qwen2 models with an 8k context size to understand documents with 1M tokens, surpassing RAG and native long-context models. This agent was also used to generate data for training new long-context Qwen models. Introduction # Recently, there has been a hype trend in LLMs that can natively process sequences of millions of tokens. Most work has been focusing on sophisticated mathematical tweaks like RoPE-based extrapolation or architectural overhauls such as non-transformer LLMs. However, preparing fine-tuning data that is sufficiently long is a less d
saved by
related reading
- LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- Patterns for Building LLM-based Systems & Productseugeneyan.com
- How Long Contexts Faildbreunig.com
- [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
- QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Managementarxiv.org
- Extending Context is Hard | kaiokendevkaiokendev.github.io
- Qwen3: Think Deeper, Act Faster | Qwenqwenlm.github.io
- Long Context RAG Performance of LLMs | Databricks Blogdatabricks.com
- Advanced RAG Techniques: What They Are & How to Use Themfalkordb.com
- Quantization from the ground upngrok.com
- [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org