Part 1: The Map Was Wrong — Nemostation
nemostation.com · 3,952 words · saved by 1 readers
A 2B open-source video model audits the standard video-caption benchmarks and finds 70% of the ground-truth captions are wrong.
A month ago, we came across the Tarsier model , which was specifically trained for dense video captioning. Something felt off, though. The captions were dense, but there was no information about the time frames where those events occurred. We searched through the existing video-understanding benchmarks for models that were good at this, but the results were subpar. Most benchmarks don't focus on the temporal axis at all, and the ones that do try to measure it use text-based judges, which introduce several biases. So we built Marlin-2B and a new audit pipeline. The rest of this post is what we
saved by
related reading
- GitHub - openai/mle-bench: MLE-bench is a benchmark for measuring how well AI agents perform at machine learning engineeringgithub.com
- Seeing Is Not Reasoning: How VLMs and Their Benchmarks Lean on Textharvey-fin.github.io
- PostTrainBenchposttrainbench.com
- 2403.09611.pdfarxiv.org
- 2409.02908arxiv.org
- Video models are zero-shot learners and reasonersarxiv.org
- Kimi K2.5 Tech Blog: Visual Agentic Intelligencekimi.com
- Video Generation Models Explosion 2024yenchenlin.me
- [2506.09985] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningarxiv.org
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelsarxiv.org
- FrontierSWEfrontierswe.com
- A Dive into Text-to-Video Modelshuggingface.co