2410.06205
arxiv.org · 8,427 words · saved by 1 readers
N/A
Published as a conference paper at ICLR 2025 ROUND AND ROUND W E G O ! L W HAT MAKES ROTARY P OSITIONAL E NCODINGS USEFUL ? Federico Barbero˚ Alex Vitvitskyi Christos Perivolaropoulos University of Oxford Google DeepMind Google DeepMind Razvan Pascanu Petar…
related reading
- [2410.06205] Round and Round We Go! What makes Rotary Positional Encodings useful?arxiv.org
- On N-dimensional Rotary Positional Embeddingsjerryxio.ng
- Rotary Embeddings: A Relative Revolution | EleutherAI Blogblog.eleuther.ai
- You could have designed state of the art positional encodinghuggingface.co
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
- Jane Street Blog - Using group theory to explore the space of positional encodings for attentionblog.janestreet.com
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
- How LLMs Actually Work | 0xkato0xkato.xyz
- Extending Context is Hard | kaiokendevkaiokendev.github.io
- Extending the Context of Pretrained LLMs by Dropping their Positional Embeddingspub.sakana.ai
- Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com