BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity? | alphaXiv
Researchers from FAIR at Meta and Inria introduced BigO(Bench), a new benchmark that assesses large language models' capacity to generate functionally corr
Submitted 20 Mar 2025 Abstract We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generating code with specified time and space complexities. This benchmark addresses the gap in current evaluations that often overlook the ability of models to comprehend and produce code constrained by computational complexity. BigO(Bench) includes tooling to infer the algorithmic complexity of any Python function from profiling measurements, including human- or LLM-generated solutions. BigO(Bench) also includes of set…
saved by
related reading
- [2501.01257] CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingsarxiv.org
- MirrorCode: Evidence AI can already do some weeks-long coding tasks | Epoch AIepoch.ai
- AI excels at code competitions, struggles with real workblog.peterwildeford.com
- Composer2.pdfcursor.com
- [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
- My benchmark for large language modelsnicholas.carlini.com
- 2025: The year in LLMssimonwillison.net
- LiCoEval: Evaluating LLMs on License Compliance in Code Generationarxiv.org
- ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering | alphaXivalphaxiv.org
- crawshaw - 2025-01-06crawshaw.io
- Lean Software Scaling Laws · Gwern.netgwern.net
- KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu