GPT-4 and professional benchmarks: the wrong answer to the wrong question
normaltech.ai · 1,933 words · saved by 1 readers
OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots.
OpenAI didn’t release much information about GPT-4 — not even the size of the model — but heavily emphasized its performance on professional licensing exams and other standardized tests. For instance, GPT-4 reportedly scored in the 90th percentile on the bar exam. So there’s been much speculation about what this means for professionals such as lawyers. We don’t know the answer, but we hope to inject some reality into the conversation. OpenAI may have violated the cardinal rule of machine learning: don’t test on your training data. Setting that aside, there’s a bigger problem. The manner in…
related reading
- GPT-4openai.com
- gpt-4.pdfcdn.openai.com
- 2303.08774.pdfarxiv.org
- gpt-4-system-card.pdfcdn.openai.com
- Noam Brown on X: "Implications of Large-Scale Test-Time Compute" / Xx.com
- Learning to reason with LLMs | OpenAIopenai.com
- AI hype is built on high test scores. Those tests are flawed. | MIT Technology Reviewtechnologyreview.com
- Late Takes on OpenAI o1alexirpan.com
- Is GPT-4 getting worse over time?normaltech.ai
- Is GPT-4 getting worse over time?aisnakeoil.com
- GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision, MoEsemianalysis.com
- What will GPT-2030 look like? — AI Alignment Forumalignmentforum.org