Estimating the Productivity of an Autonomous AI Software Engineer | Cognition
Engineering leaders want to know how much value AI is actually providing. We built a system that measures the number of human-equivalent hours of Devin's work.
Six months ago, every CTO was concerned their team wasn't using enough tokens. That trend has reversed as token usage and AI spend have skyrocketed. Engineering leaders are now trying to figure out how to measure actual output, because not every token delivers real value. Some save engineering hours and accelerate projects; others are wasted on useless sessions and bad prompting. For any organization of sufficient scale, it's essentially impossible to measure value over thousands of sessions & billions of tokens. We set out to automate this with AI. Predicting real ROI is hard, for reasons we’
saved by
related reading
- Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPTmetr.org
- How much I'm paying for AI productivity software (and the future of AI use)jacquesthibodeau.com
- AI should earn its keep: Introducing the AI Productivity Guaranteecognition.ai
- AI Timelines — LessWronglesswrong.com
- When AI builds itself \ Anthropicanthropic.com
- My picture of the present in AI — LessWronglesswrong.com
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity - METRmetr.org
- DX Research Archivegetdx.com
- claude code is not making your product betterethanding.substack.com
- I underestimated AI capabilities (again) - by Ajeya Cotraplanned-obsolescence.org
- Contributions to OpenAI's Codex codebase show signs of AI upliftepoch.ai
- AI is removing the middle class of software engineeringblog.florianherrengt.com