✳flâneur — a map of the web's best reading

Estimating the Productivity of an Autonomous AI Software Engineer | Cognition

cognition.ai · 2,528 words · saved by 1 readers

Engineering leaders want to know how much value AI is actually providing. We built a system that measures the number of human-equivalent hours of Devin's work.

Six months ago, every CTO was concerned their team wasn't using enough tokens. That trend has reversed as token usage and AI spend have skyrocketed. Engineering leaders are now trying to figure out how to measure actual output, because not every token delivers real value. Some save engineering hours and accelerate projects; others are wasted on useless sessions and bad prompting. For any organization of sufficient scale, it's essentially impossible to measure value over thousands of sessions & billions of tokens. We set out to automate this with AI. Predicting real ROI is hard, for reasons we’

Explore this link on the map →

saved by

related reading