Transformer²: Self-Adaptive LLMs

Transformer²: Self-Adaptive LLMs

--> Summary Adaptation is one of the most remarkable phenomena in nature. From the way an octopus can change their skin color to blend into its surroundings, to how the human brain rewires itself after an injury , allowing individuals to recover lost functions and adapt to new ways of thinking or moving. Living organisms exhibit adaptability that allows life to flourish in diverse and ever-changing environments. In the field of AI, the concept of adaptation holds a similar allure. Imagine a machine learning system that could adjust its own weights dynamically to thrive in unfamiliar settings,

saved by

• Winnie Xu

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Self-Adapting Language Modelsarxiv.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
• Transformer Circuits Threadtransformer-circuits.pub
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• GenAI Handbookgenai-handbook.github.io
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• What I've Learned About AI in the Past Two Months.sheracaolity.ghost.io
• Can LLMs Be Computers?percepta.ai
• PEFT Method Overview [implementing Adapters in PyTorch] | Eva Korolevaxmarva.github.io