BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity? | alphaXiv
Researchers from FAIR at Meta and Inria introduced BigO(Bench), a new benchmark that assesses large language models' capacity to generate functionally corr
Submitted 20 Mar 2025 Abstract We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generating code with specified time and space complexities. This benchmark addresses the gap in current evaluations that often overlook the ability of models to comprehend and produce code constrained by computational complexity. BigO(Bench) includes tooling to infer the algorithmic complexity of any Python function from profiling measurements, including human- or LLM-generated solutions. BigO(Bench) also includes of set…
saved by
related reading
- [2501.01257] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingsarxiv.org
- Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org
- Measuring what Matters: Construct Validity in Large Language Model Benchmarksalphaxiv.org
- [2605.02421] AOCI: Symbolic-Semantic Indexing for Practical Repository-Scale Code Understanding with LLMsarxiv.org
- ZeroCoder: Can LLMs Improve Code Generation Without Ground-Truth Supervision?alphaxiv.org
- MirrorCode: Evidence AI can already do some weeks-long coding tasks | Epoch AIepoch.ai
- [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
- KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
- Program Synthesis with Large Language Modelsarxiv.org
- [2605.20873] PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Modelsarxiv.org
- Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebasedatabricks.com
- Introducing FrontierCode | Cognitioncognition.ai