Character Prefix Conditioning | Cursor - The AI Code Editor
cursor.com · 242 words · saved by 2 readers
A clever algorithm for more accurate code completion sampling.
The first in a series of problems that give a glimpse into the work we do at Cursor. Setup When using a language model for code completion, we typically want the model to produce a completion that begins with what the user has typed. However, modern language models operate on sequences of tokens, not characters, so naively tokenizing the user's input and sending it to the model produces wrong results if the user's cursor doesn't happen to lie on a token boundary. Instead, we need an algorithm that samples a sequence of tokens conditional on a prefix of characters, rather than the more…
saved by
related reading
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- More problemscursor.com
- Sampling from Your Language Model One Byte at a Timearxiv.org
- [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
- Improving Cursor Tab with online RL · Cursorcursor.com
- Efficient Guided Generation for Large Language Modelsarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Shaping capabilities with token-level data filteringarxiv.org
- Speculative Decoding - philkravphilkrav.com
- Compression and Intelligencegreene.sh
- Prefix-Tuning: Optimizing Continuous Prompts for Generationarxiv.org
- Conditioning, Prompts, and Fine-Tuning — LessWronglesswrong.com