Llama Scope: Extracting Features from Llama 3.1-8B with SAEs

N/A

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders L LAMA S COPE : E XTRACTING M ILLIONS OF F EA - TURES FROM L LAMA -3.1-8B WITH S PARSE AUTOEN - CODERS Zhengfu He1,2 Wentao Shu1 Xuyang Ge1 Lingjie Chen1 Junxuan Wang1 Yunhua Zhou1,3 Frances Liu Qipeng Guo1,2,3 Xuanjing Huang1 Zuxuan Wu1…

related reading

• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelsarxiv.org
• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• GitHub - PaulPauls/llama3_interpretability_sae: A complete end-to-end pipeline for LLM interpretability with sparse autoencoders (SAEs) using Llama 3.2, written in pure PyTorch and fully reproducible.github.com
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Sparse Autoencoders Reveal Universal Feature Spaces Across Large Language Modelsarxiv.org
• OpenAI SAE Training Paperarxiv.org
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• pdfopenreview.net
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• [2409.14507] A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencodersar5iv.labs.arxiv.org