flâneur

OpenAI O3 breakthrough high score on ARC-AGI-PUB | Hacker News

news.ycombinator.com · 121,731 words · saved by 1 readers

~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute went into this task. We can compare this roughly to a human doing ARC-AGI puzzles, where a human will take (high variance in my subjective experience) between 5 second and 5 minutes to solve the task. (So i'd argue a human is at 0.03USD - 1.67USD per puzzle at 20USD/hr, and they include in their document an average mechancal turker at $2 USD task in their document) Going the other direction: I am interpreting this result as human level reasoning now costs (approximately) 41k/hr to 2.5M/hr with current compute. Super exciting that OpenAI pushed the compute out this far so we could see he O-series scaling continue and intersect humans on ARC, now we get to work towards making this economical! reply So, considering that the $3400/task

OpenAI O3 breakthrough high score on ARC-AGI-PUB | Hacker News Hacker News new | past | comments | ask | show | jobs | submit login OpenAI O3 breakthrough high score on ARC-AGI-PUB ( arcprize.org ) 1724 points by maurycy on Dec 20, 2024 | hide | past | favorite | 1755 comments bluecoconut on Dec 20, 2024 | next [–] Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute w

related reading