Add NTK-Aware interpolation "by parts" correction by bloc97 · Pull Request #1 · jquesnelle/scaled-rope

This PR adds the new and improved "by parts" correction to the NTK-aware interpolation method. This corrected method improves from previous methods fourfold: Decreases PPL in all context lengths w...

This PR adds the new and improved "by parts" correction to the NTK-aware interpolation method. This corrected method improves from previous methods fourfold: Decreases PPL in all context lengths when used on non-finetuned models compared to previous NTK-Aware method, especially for higher context sizes as alpha value can be set much lower for same context size. Removes the alpha parameter, which did not accurately predict effective context length and was variable across different models. Now uses same scale parameter as linear interpolation which is much more intuitive and less prone to…

saved by

• Bryan Chiang

related reading

• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• Self-Adapting Language Modelsarxiv.org
• Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
• 2502.11089arxiv.org
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• [2108.12409] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org
• Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelsarxiv.org
• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations — LessWronglesswrong.com