Add NTK-Aware interpolation "by parts" correction by bloc97 · Pull Request #1 · jquesnelle/scaled-rope

This PR adds the new and improved "by parts" correction to the NTK-aware interpolation method. This corrected method improves from previous methods fourfold: Decreases PPL in all context lengths w...

This PR adds the new and improved "by parts" correction to the NTK-aware interpolation method. This corrected method improves from previous methods fourfold: Decreases PPL in all context lengths when used on non-finetuned models compared to previous NTK-Aware method, especially for higher context sizes as alpha value can be set much lower for same context size. Removes the alpha parameter, which did not accurately predict effective context length and was variable across different models. Now uses same scale parameter as linear interpolation which is much more intuitive and less prone to…

saved by

• Bryan Chiang

related reading

• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Tinkerthinkingmachines.ai
• Parsed | Custom, interpretable AI systems that continuously learnparsed.com
• [2108.12409] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org
• Llama 2 · Hugging Facehuggingface.co
• Annotated Research Paper Implementations: Transformers, StyleGAN, Stable Diffusion, DDPM/DDIM, LayerNorm, Nucleus Sampling and morenn.labml.ai
• Goodfire AIgoodfire.ai
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org
• Neuronpedianeuronpedia.org
• nnsightnnsight.net
• Topicslearnmechinterp.com