Instant LLM Updates with Doc-to-LoRA and Text-to-LoRA

Recent LLM agents have shown impressive capabilities on complex computer use and long-horizon tasks. Yet, they still struggle with long-term memory and adaptation—two of the most important cognitive capabilities that still limit LLMs today. Without long-term memory, users have to provide LLMs with relevant content at the start of every new session, creating friction, discontinuity, and longer time-to-response. Additionally, due to the lack of adaptation, they do not learn from mistakes or user preferences from previous sessions, making each interaction as cumbersome as the first. Traditionally, these two problems are tackled by “updating” the model. 1. LLM knowledge update (memory). When a user provides a long document, e.g., a policy, a report, or a private PDF, the standard solution is to put it in the context window. This works, but it means every new query re-reads the same document, paying the full latency and VRAM cost each time. Practical workarounds like KV-cache pre-filling he

--> This page requires JavaScript. Please enable it to view the website. Instant LLM Updates with Doc-to-LoRA and Text-to-LoRA TL;DR Long-term memory and continual adaptation of Large Language Models (LLMs) are two key challenges of current agentic systems. Here, we propose the usage of auxiliary modulator networks (so-called "hypernetworks" ) that modify LLM weights on the fly to compress document information and master new skills. Doc-to-LoRA enables knowledge updates by turning documents into LoRA adapters, allowing a model to internalize new factual content without retraining. Text-to-LoRA

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• The Continual Learning Problemjessylin.com
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
• [2605.12484] Learning, Fast and Slow: Towards LLMs That Adapt Continuallyarxiv.org
• LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• [2603.23516] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokensarxiv.org