Raayan Dhar

2 followers · 5 following · 869 views

on the atlas · 46

highlights · 195

the oversimplified narrative
The Bottlenecks to Scaling Foundation Models for Robotics | ICLR Blogposts 2026
(ii) the philosophy espoused in Richard Sutton’s influential blog post, The Bitter Lesson
The Bottlenecks to Scaling Foundation Models for Robotics | ICLR Blogposts 2026
(i) the empirical success of increasingly large transformer models trained on internet-scale data at solving language understanding and generation tasks (e.g., the so-called scaling laws in large language models)
The Bottlenecks to Scaling Foundation Models for Robotics | ICLR Blogposts 2026
On-chip memory units (CMEM, VMEM, SMEM) on TPUs are much larger than L1, L2 caches on GPUs
TPU Deep Dive
128x128 systolic array
TPU Deep Dive
when using Tensor Cores as the fundamental Tensor Core operation is an 8 by 8 by 128b matrix multiply
Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.ca
we want at least 4 warps per thread block (not strictly required as multiple thread blocks can run concurrently on one SM
Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.ca
CUTLASS’s permuted shared memory layout avoids bank conflicts and how to set up an n-stage global to shared memory pipeline.
Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.ca
the compiler can only apply optimizations on a single source file rather than on the whole program
Compilers - What Every Programmer Should Know About Compiler Optimizations | Microsoft Learn
whole program optimizations (WPO) on C/C++ code
Compilers - What Every Programmer Should Know About Compiler Optimizations | Microsoft Learn
We basically want one object for each construct in the language, and the AST should closely model the language.
2. Kaleidoscope: Implementing a Parser and AST — LLVM 20.0.0git documentation
IdentifierStr global variable holds the name of the identifier. If the current token is a numeric literal (like 1.0), NumVal holds its value.
1. Kaleidoscope: Kaleidoscope Introduction and the Lexer — LLVM 20.0.0git documentation
he traditional way to do this is to use a “lexer” (aka ‘scanner’) to break the input up into “tokens”. Each token returned by the lexer includes a token code and potentially some metadata (e.g. the numeric value of a number).
1. Kaleidoscope: Kaleidoscope Introduction and the Lexer — LLVM 20.0.0git documentation
CUDA code is written from a single-thread perspective
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
If it hurts, you're probably doing it wrong
Pain is not the unit of Effort
because you've just created another knife they feel pressured to cut themselves with.
Pain is not the unit of Effort
optimize for the appearance of suffering
Pain is not the unit of Effort
directly translated in his mind to accusing him of low pain tolerance and asking him to hurt himself more
Pain is not the unit of Effort
superhuman pain tolerance
Pain is not the unit of Effort
what incentives were they responding to
Pain is not the unit of Effort
harrowing. He told me that players who fell behind the pack during for daily jogs would be singled out and publicly humiliated.
Pain is not the unit of Effort
To this day there's a gremlin on my shoulder that whispers, "If you're happy, you're not trying your best
Pain is not the unit of Effort
our model is like a discrete analog of a recurrent convolutional network. It’s “convolutional” because at any given step the same rule is applied—locally—throughout an array of elements. It’s “recurrent” because in effect data is repeatedly “passed through” the same rul
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
particularly in generating diversity both in behavior, and in the paths necessary to reach it
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
none of them seem to be using a clear “mechanical procedure” (of the kind we might construct through traditional engineering) to reach their goal.
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
won’t be a traditional (say, mathematical) “general science” of machine learning (or, for that matter, probably also neuroscience)
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
things end up being effectively random enough that the adaptive process of training a machine learning system can reach success without getting stuck
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
Well, it’s only because of computational irreducibility that there’s all that richness in the computational universe
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
Machine learning, it seems, isn’t building structured mechanisms; rather, it’s basically just sampling from the typical complexity one sees in the computational universe, picking out pieces whose behavior turns out to overlap what’s needed
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
. One might have imagined that even though the training of a machine learning system might be circuitous, somehow in the end the system would do what it does through some kind of identifiable and “explainable” mechanism
What’s Really Going On in Machine Learning? Some Minimal Models—Stephen Wolfram Writings
Deleting (and relearning) dangerous representations
Circumventing interpretability: How to defeat mind-readers — LessWrong
The most commonly used are L0 and Loss Recovered. L0 is the average number of nonzero elements in the SAE's encoded intermediate representation. Loss Recovered is where we replace the GPT's original activation with our reconstructed activation and measure the additional loss from the imperfect reconstruction
An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWrong
decoder vectors correspond to linear representations of features in residual stream space
An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWrong
If we also wanted to analyze various intermediate activations within each layer, this would require hundreds of SAEs.
An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWrong
We train individual SAEs on only one location in the model. For example, we could train a single SAE on intermediate activations between layers 26 and 27. T
An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWrong
Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king. If you're Sam Altman, you don't have to be profitable to convey to investors that you'll succeed with or without them. (He wasn't, and he did.) Not everyone has Sam's deal-making ability.
A Fundraising Survival Guide
that the mind itself is a computational system
The Computational Theory of Mind (Stanford Encyclopedia of Philosophy)
curiosity may be the more powerful of the two. Ambition tends to make you climb existing peaks, but if you stick close enough to an interesting enough question, it may grow into a mountain beneath you.
Superlinear Returns
It's an intimidating prospect to enter a field where a few big winners outperform everyone else. Some people do this deliberately, but you don't need to. If you have sufficient natural ability and you follow your curiosity sufficiently far, you'll end up in one. Your curiosity won't let you be interested in boring questions, and interesting questions tend to create fields with superlinear returns if they're not already part of one.
Superlinear Returns
Learning, for example. When you first start learning something, you feel lost. But it's worth making the initial effort to get a toehold, because the more you learn, the easier it will get.
Superlinear Returns
the remaining list is exactly the same as the list of fields where you have to be independent-minded to succeed — where your ideas have to be not just correct, but novel as well.
Superlinear Returns
look for fields where a few big winners outperform everyone else. A kind of work where everyone does about the same is unlikely to be one with superlinear returns.
Superlinear Returns
he solution is to take multiple shots. Which is another reason to start taking risks early.
Superlinear Returns
Luck is always a factor, but it's even more of a factor when you're working on your own rather than as part of an organization. And though there are some valid aphorisms about luck being where preparedness meets opportunity and so on, there's also a component of true chance that you can't do anything about
Superlinear Returns
Choose work you have a natural aptitude for and a deep interest in. Develop a habit of working on your own projects; it doesn't matter what they are so long as you find them excitingly ambitious. Work as hard as you can without burning out, and this will eventually bring you to one of the frontiers of knowledge. These look smooth from a distance, but up close they're full of gaps. Notice and explore such gaps, and if you're lucky one will expand into a whole new field. Take as much risk as you can afford; if you're not failing occasionally you're probably being too conservative. Seek out the b…
Superlinear Returns
surprising amount of technique to doing great work
Superlinear Returns
if there's a kind of work that inherently requires a huge expenditure of time and energy, it will be increasingly valuable to do it when you're young and don't yet have children.
Superlinear Returns
in a world with superlinear returns, it's even more valuable to know what you're interested in, and to find ways to work on it
Superlinear Returns
to do something exceptionally well, you have to be interested in it. Mere diligence is not enough.
Superlinear Returns
obvious way to take advantage of superlinear returns for performance is by doing exceptionally good work. At the far end of the curve, incremental effort is a bargain. All the more so because there's less competition at the far end — and not just for the obvious reason that it's hard to do something exceptionally well, but also because people find the prospect so intimidating that few even try. Which means it's not just a bargain to do exceptional work, but a bargain even to try to.
Superlinear Returns