asherps/EasyNLA: Minimal codebase for efficiently training Natural Language Autoencoders (NLAs). Built on Celeste's nanoNLA: https://github.com/ceselder/nanoNLA ·
Minimal codebase for efficiently training Natural Language Autoencoders (NLAs). Built on Celeste's nanoNLA: https://github.com/ceselder/nanoNLA - asherps/EasyNLA
EasyNLA — fast, distributed training of Natural Language Autoencoders EasyNLA trains Natural Language Autoencoders (NLAs) — models that read a residual-stream activation and write a natural-language explanation of it, then reconstruct the activation back from that text. The focus here is fast, distributed training. It scales with data parallelism: launch under torchrun with one rank per GPU, each running its own single-GPU vLLM rollout engine + trainer on a slice of the batch, gradients all-reduced every step. Weights resync to the sampler each step over fast GPU→GPU IPC, and the whole NLA…
saved by
related reading
- Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
- Natural Language Autoencoders \ Anthropicanthropic.com
- GitHub - kitft/natural_language_autoencoders · GitHubgithub.com
- Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — LessWronglesswrong.com
- GitHub - karpathy/autoresearch: AI agents running research on single-GPU nanochat training automaticallygithub.com
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.github.com
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- The Annotated Transformernlp.seas.harvard.edu
- GitHub - huggingface/nanotron: Minimalistic large language model 3D-parallelism training · GitHubgithub.com
- I Trained a Language Model. Then I Built a Brain Scanner and Looked Inside It. | by Caleb DeLeeuw | Mediummedium.com
- GitHub - PaulPauls/llama3_interpretability_sae: A complete end-to-end pipeline for LLM interpretability with sparse autoencoders (SAEs) using Llama 3.2, written in pure PyTorch and fully reproducible.github.com