Transformer²: Self-Adaptive LLMs

Transformer²: Self-Adaptive LLMs

--> Summary Adaptation is one of the most remarkable phenomena in nature. From the way an octopus can change their skin color to blend into its surroundings, to how the human brain rewires itself after an injury , allowing individuals to recover lost functions and adapt to new ways of thinking or moving. Living organisms exhibit adaptability that allows life to flourish in diverse and ever-changing environments. In the field of AI, the concept of adaptation holds a similar allure. Imagine a machine learning system that could adjust its own weights dynamically to thrive in unfamiliar settings,

saved by

• Winnie Xu

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Self-Adapting Language Modelsarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Transformer Circuits Threadtransformer-circuits.pub
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• GenAI Handbookgenai-handbook.github.io
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• What I've Learned About AI in the Past Two Months.sheracaolity.ghost.io
• PEFT Method Overview [implementing Adapters in PyTorch] | Eva Korolevaxmarva.github.io
• Explore | alphaXivalphaxiv.org
• Recent Advances in Language Model Fine-tuningruder.io