Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase | Databricks Blog
Databricks shares results from its internal coding benchmark, evaluating coding agents on a multi-million line codebase to optimize engineering cost and performance.
At Databricks, the way we build software is changing quickly as we aggressively adopt AI for engineering. The landscape of models and harnesses for code authoring has rapidly expanded in the last year, giving developers more choices than ever. With more options, it has become increasingly important to understand which coding agents offer the best performance on real-world coding tasks as well as understanding how task-performance varies with price. This article shares the results and methodology of the internal coding benchmark we built at Databricks, which evaluates tools on actual coding…
saved by
related reading
- Scaling long-running autonomous coding · Cursorcursor.com
- How coding agents read your code (and how to write for them)modem.dev
- If coding is solved, what now?: Measuring the sloppiness of code | EARENDILearendil.com
- Building Effective AI Agents \ Anthropicanthropic.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- claude code is not making your product betterethanding.substack.com
- Towards self-driving codebases · Cursorcursor.com
- Introducing FrontierCode | Cognitioncognition.ai
- Code was our medium for thoughtwattenberger.com
- Introducing FrontierCode | Cognitioncognition.com
- Shipping at Inference-Speed | Peter Steinbergersteipete.me
- Quantifying infrastructure noise in agentic coding evals \ Anthropicanthropic.com