Deep Learning Experiments and Claims | Yuxin's Blog
这几年来, 学习到的最多的是对待 research 的态度. 因此说说写 paper 和做实验的体会.实验与 claims实验是为了证明或强化文章里给出的 claim / hypothesis 的.Ross ICCV 2019 tutorial 最后谈了谈怎么写 paper. 第126页说, 文章中所有的 claim, 理想情况下都应该要么是文献中已有的 claim, 要么是有实验能够证明的 cl
这几年来, 从 FAIR 的几位大佬身边学习到的最多的是对待 research 的态度. 因此说说写 paper 和做实验的体会. 实验与 claims¶ 实验是为了证明或强化文章里给出的 claim/hypothesis 的. Ross ICCV 2019 tutorial 最后谈了谈怎么写 paper. 第 126 页说, 文章中所有的 claim, 理想情况下都应该 要么是参考文献中已有的, 要么是被实验证明的. 举个例子, BatchNorm paper 的实验可以 claim 很多东西, 包括 BatchNorm 让结果很好, 对初始化不敏感, 大 learning rate 也不炸. 但是文章 claim BatchNorm "reduce internal covariate shift", 就遭到了一些人的质疑. 如著名的 Ali Rahimi Neurips 2017 test-of-time award presentation (B 站) 里的五连问 (第 17 分钟). BatchNorm paper 中把 "internal covariate shift" 粗略定义为 feature distribution 的变化, 但唯一一个与此相关的实验是 Fig.1 (b)(c). 虽然实验是合理的, 但结果确实不算不太显著. 甚至…
saved by
related reading
- How to win a best paper award (or, an opinionated take on how to do important research)nicholas.carlini.com
- Highly Opinionated Advice on How to Write ML Papers — LessWronglesswrong.com
- Tips for Empirical Alignment Research — AI Alignment Forumalignmentforum.org
- [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
- Maybe I was too harsh on deep learning theory (three days ago) — LessWronglesswrong.com
- Mediumpub.towardsai.net
- The Decade of Deep Learning | Leo Gaobmk.sh
- Fake Journal Club: Teaching Critical Reading · Gwern.netgwern.net
- Templates for machine learning research papers | Neel Guhaneelguha.github.io
- Fake Journal Club: Teaching Critical Reading · Gwern.netgwern.net
- A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
- Make It Work, Then Prove It Works: A Framework for Research - Vincent Sitzmannvincentsitzmann.com