How should we evaluate progress in AI? | Better without AI
betterwithout.ai · 7,435 words · saved by 1 readers
Improving artificial intelligence research with scientific testing, design practice, and meta-rational choice of methods and criteria
Wolpertinger image courtesy Rainer Zenz The evaluation question is inseparable from questions about what sort of thing AI is—and both are inseparable from questions about how best to do it. Most intellectual disciplines have standard, unquestioned criteria for what counts as progress. Artificial intelligence is an exception. It has always borrowed criteria, approaches, and specific methods from at least six fields: 1. Science 2. Engineering 3. Mathematics 4. Philosophy 5. Design 6. Spectacle This has always caused trouble. The diverse evaluation criteria are incommensurable. They suggest diver
related reading
- 2023 letter | Zhengdongzhengdongwang.com
- Sporadicamichaelnotebook.com
- Intentionally Designing the Future of AIgoodfire.ai
- Toward a Critical Technical Practicepages.gseis.ucla.edu
- Designing AI for Disruptive Scienceasimov.press
- Thoughts — Jason Weijasonwei.net
- Are AI benchmarks doomed? - by Anson Ho and Greg Burnhamepochai.substack.com
- [2603.26524] Mathematical methods and human thought in the age of AIarxiv.org
- The Universe from an Intentional Stancecasparoesterheld.com
- Things I learned at OpenAI - by Karina Nguyen - sémaphoresemaphore.substack.com
- Assessing skeptical views of interpretability research | Christopher Pottsweb.stanford.edu
- AI Safety Seems Hard to Measurecold-takes.com