Have We Seen an Acceleration in Discoveries? - METR
Tom Cunningham and Nate Rush look at public time series of discoveries in cyber, math, and algorithms to see whether LLMs have accelerated the discovery rate.
Q: How much has discovery accelerated? LLMs have shown the ability to make novel discoveries across many domains. How much has this affected the aggregate discovery rate? In the figures below we plot all the data sources we can find, and make some very loose observations: Discovery of cyber vulnerabilities has accelerated sharply. Discovery of math results has accelerated somewhat. However this is harder to objectively measure. Discovery of optimizations has not shown a dramatic acceleration. These conclusions are based only on public discoveries. It is quite plausible that AI labs are…
saved by
related reading
- Evidence on AI R&D Progress from NanoGPT - METRmetr.org
- Does AI Progress Have a Speed Limit?—Asteriskasteriskmag.com
- An Apple-Picking Model of AI R&D | Tom Cunningham – Tom Cunninghamtecunningham.github.io
- There Are No New Ideas in AI… Only New Datasetsblog.jxmo.io
- Will AI R&D Automation Cause a Software Intelligence Explosion?forethought.org
- My picture of the present in AI — LessWronglesswrong.com
- AI progress is about to speed up | Epoch AIepoch.ai
- The nature of LLM algorithmic progress (v2) — LessWronglesswrong.com
- Existential Risk from AI: An Exposition for Mathematiciansalkjash.github.io
- How fast is AI improving? - AI Digesttheaidigest.org
- The Need for Verification in AI-Driven Scientific Discoveryalphaxiv.org
- The Most Important Time in History Is Now - by Tomas Pueyounchartedterritories.tomaspueyo.com