Improving Composer through real-time RL · Cursor
We apply online reinforcement learning to Composer, serving model checkpoints to production and using real user interactions as reward signals to ship an improved checkpoint multiple times a day.
Blog / research We are observing unprecedented growth in the usefulness and adoption of coding models in the real world. In the face of 10–100x increases in inference volume, we consider the question: how can we take these trillions of tokens and extract from them a training signal to improve the model? We call our approach of using real inference tokens for training "real-time RL." We first used this technique to train Tab and we found it was highly effective. Now we're applying a similar approach to Composer. We serve model checkpoints to production, observe user responses, and aggregate tho
saved by
related reading
- Composer2.pdfcursor.com
- Improving Cursor Tab with online RL · Cursorcursor.com
- Training Composer for longer horizons · Cursorcursor.com
- Coding vs thinking — Paradigm 3paradigm3.org
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- Towards self-driving codebases · Cursorcursor.com
- How we compare model quality in Cursorcursor.com
- Training a Misaligned Reward Seekeralignment.anthropic.com
- Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
- Debugging Reinforcement Learning Systemsandyljones.com
- Improving Cursor Tab with online RL · Cursorcursor.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org