Davis Brown

27 followers · 20 following · 2328 views

on the atlas — 167

highlights — 195

It is noteworthy that while faithfulness failures can lead to poor monitorability, complete faithfulness is not required for perfect monitorability. If there is any shared knowledge between agent and monitor, then it shouldn't in principle be required for that knowledge to be saliently represented in the latent for the monitor to make a correct judgement. For example, take the scenario where an agent knows that 2x2=4 and therefore skips verbalizing this in its CoT while performing long multiplication. By definition, the CoT is not fully faithful with respect to the agent's computation; however…
Open Sourcing Monitorability Evaluations
Reconfiguring the sufficient statistics the network has built costs something proportional to how much structure needs to change.
Bitter Lessons from Distillation Robustifies Unlearning
Anthropic (and AI companies more generally) are hill-climbing on the (limited) measures of alignment they have, while in the absence of specific efforts to improve alignment, I’d expect the default progression would probably trend mostly toward worse misalignment on the tasks near the limit of the AI’s capabilities.
Current AIs seem pretty misaligned to me
I’ve also found that the chance of cheating seems to scale with the amount of AI agent labor applied to the task, though this could partially be due to the properties of large tasks that require a lot of labor to complete. (But I don’t think this is the only reason; I think I see more cheating in cases where I’m using approaches to apply more inference compute on a given task via things like best-of-k.)
Current AIs seem pretty misaligned to me
Most importantly for PSM, we find that LLMs use the same internal representations to characterize the Assistant as for other characters present in training data. Indeed, this form of reuse is commonly observed. For instance: An “inner conflict” SAE feature activates when Claude 3 Sonnet is faced with an ethical dilemma, and also on stories about characters facing ethical dilemmas (Templeton et al., 2024). A “holding back one’s true thoughts” SAE feature activates when Claude Opus 4.5 fails to reveal information that it knows about, and also activates on stories about characters concealing thei…
The Persona Selection Model: Why AI Assistants might Behave like Humans
Emotive language. AI assistants often express emotions. For instance, Claude models express distress when given repeated requests for harmful or unethical content and express joy when successfully completing complex technical tasks like debugging (Claude Opus 4 and Sonnet 4 system card, section 5). Gemini 2.5 Pro sometimes expresses panic when playing Pokemon, with these panic expressions appearing to be associated with degraded reasoning and decision-making (Gemini Team, 2025). Gemini models also sometimes express extreme distress and other forms of emotional turmoil when struggling with diff…
The Persona Selection Model: Why AI Assistants might Behave like Humans
Follow Your Curiosity That is my message. We must be curious. Because we are living in a Copernican revolution. For the first time in 5000 years, our understanding of Rational Thought is changing. Aristotle had proposed that humans are the unique rational creature; St. Thomas Aquinas thought that it was our rationality that made humans uniquely spiritual; and Descartes famously declared "I think therefore I am," identifying rationality with personhood. But now our creation of thinking devices disproves these old philosophers. The human mind is no longer so firmly at the center of the universe,…
davidbau.com In Defense of Curiosity
The Funsearch direction is a nice little nugget. Funsearch used LLMs to generate candidate programs in a setting where they could be evaluated quantitatively for reaching some objective (think: defining heuristics based for a combinatorial problem like travelling salesman to minimize travel time), and applied genetic programming on top do search over such heuristics. You wouldn’t know it from the paper/its appendices, but what happened is that the Funsearch team tried to use larger and smaller models in the middle of the loop; they had best results with a mid-sized candidate (that I trained wi…
Gemini Flash Pretraining
There are few valuable "AI-shaped holes" because we've organized everything to minimize the damage from lacking AI to fill those holes, as it were: if there were some sort of organization which had naturally large LLM-shaped holes where filling them would massively increasing the organization's output... it would've gone extinct long ago and been replaced by ones with human-shaped holes instead, because humans were all you could get. (This is why LLM uses are pretty ridiculous right now as a % of GDP - oh wow, it can do a slightly better job of spellcheck? I can have it write some code for me?…
If I wanted to spend WAY more on AI, what would I spend it on? — LessWrong
Download a PDF of the paper titled Scaling Laws for Associative Memories, by Vivien Cabannes and 2 other authors Download PDF
[2310.02984] Scaling Laws for Associative Memories
All these concerns point towards using the kind of sparse autoencoder setup explored by Sharkey et al. over a full-blown dictionary learning setup. However, we've found that sparse autoencoders are more fragile and sensitive to hyperparameters, which is a significant countervailing consideration in using them. We are interested in finding an approach with the advantages of a sparse autoencoder (in terms of only finding true features) and the consistent trainability of the dictionary learning schemes.
Circuits Updates — May 2023
More crucially, though, this effect was actually stronger when we removed TRAK-identified examples than when we removed BM25-identified examples, and even stronger than when we removed the ground-truth primary sources!
TRAK-ing Model Behavior with Data – gradient science
To our surprise, TRAK, while outperforming the previous best data attribution method (TracIn) on this task, did worse on this benchmark than a simple information-retrieval baseline (BM25)—this baseline is based only on the lexical overlap between the query and the possible data sources, and doesn’t even look at the model at all!
TRAK-ing Model Behavior with Data – gradient science
For all models, we use Decoupled AdamW with Beta1=0.9 and Beta2=0.98, and a weight decay value of 1.0e-5. The learning rate schedule begins with a warmup to a maximum learning rate of 5.0e-4 followed by a linear decay to zero. Warmup lasted for 6% of the full training duration. Global batch size was set to 4096, and microbatch size was 128; since global batch size was 4096, full pretraining consisted of 70,000 batches. We set the maximum sequence length during pretraining to 128, and we used the standard embedding dimension of 768. These hyperparameters were the same for MosaicBERT-Base and th…
MosaicBERT: Pretraining BERT from Scratch for $20
FlashAttention, ALiBi, unpadding, low precision LayerNorm, and Gated Linear Units
MosaicBERT: Pretraining BERT from Scratch for $20
An interesting variant is attention pattern patching, where we patch individual attention pattern weights (for a specific head and a specific source and destination position). (I haven't explicitly seen anyone do this, but it's an obvious enough idea that I'm sure someone has tried). It's also a cute technique because it gives you a score for each head and each pair of positions, which you can then feed through an attention pattern visualiser!
Attribution Patching: Activation Patching At Industrial Scale — Neel Nanda
In something close to despair, Barth is driven to considering the cognitive processes at work within the community's ritual specialists. The initiatees are groups of men in the same age-set; the rituals are conducted infrequently, for some of the most important only once every ten years; in between, the full ritual and myth may be known only to one or two specialists. Those who have been through the rituals don't discuss them with the un-initiated, and rarely with each other. (The ritual specialists at Barth's new field-site were willing to talk about their rites and myths with him, because th…
Review of Barth, Cosmologies in the Making
The hope is that this can be rolled out with future GPT releases. We’d love to do something similar for DALL-E—that is, watermarking images, not at the pixel level (where it’s too easy to remove the watermark) but at the “conceptual” level, the level of the so-called CLIP representation that’s prior to the image. But we don’t know if that’s going to work yet.
Shtetl-Optimized » Blog Archive » My AI Safety Lecture for UT Effective Altruism
However, the presence of a feature at the end of training is hardly informative about the inductive bias of a model on its own! Consider Lovering et al., who found that the ease of extracting a feature at the start of training, along with an analysis of the finetuning data, has deeper implications for finetuned performance than we get by simply probing at the end of training.
Interpretability Creationism | Objective Funk
The deeper insight of this technique (not really covered in the work) is that we can do this on any vector in the residual stream to interpret it in terms of the direct effect on the logits - including the output of an attn or MLP layer and even a head or neuron. And we can also do this on weights writing to the residual stream.
An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers — Neel Nanda
There are two types of transformers and you should not generalize from one to the other.
LLM.int8() and Emergent Features — Tim Dettmers
FFN layers become more “dense”. While in computer vision, you can prune about 95% of weights without severe performance degradation, that number is 30% for transformers trained on NLP data. After emergence, this number shrinks to well below 5%.
LLM.int8() and Emergent Features — Tim Dettmers
Attention layers become very sparse. The
LLM.int8() and Emergent Features — Tim Dettmers
Outliers become very large quickly. They grow from about 15 for a 6B model to about 60 for a 13B model. OPT-66B has outliers of size around 95, which indicates this growth phase is temporary.
LLM.int8() and Emergent Features — Tim Dettmers
Outliers become very large quickly. They grow from about 15 for a 6B model to about 60 for a 13B model. OPT-66B has outliers of size around 95, which indicates this growth phase is temporary.
LLM.int8() and Emergent Features — Tim Dettmers
Transformers seem to coordinate these dimensions throughout all layers except the attention function and the second feedforward network where these outliers are “consumed” to remove features.
LLM.int8() and Emergent Features — Tim Dettmers
If you take this mechanism to an extreme, you can get discretization, which goes hand-in-hand with context-dependent memory and “reasoning” over elements. Discretization means, you have, say, 100 features, but you decide to remove 99% of them by setting them to zero, and you amplify the rest. The result is a single feature that is now a discrete entity. Once discretized, this entity can be stored and reused later.
LLM.int8() and Emergent Features — Tim Dettmers
Phase shift: Outlier features suddenly become available in all transformer layers and coordinate through a few hidden dimensions.
LLM.int8() and Emergent Features — Tim Dettmers
Emergence is not sudden but gradual and grows according to an exponential function related to perplexity and not model size. Outlier features grow very quickly once their phase shift occurs. The number of outliers features is strictly proportional to perplexity.
LLM.int8() and Emergent Features — Tim Dettmers
Another point to look at is how crank theories are propagated from person to person, and which are susceptible to institutionalization.
Psychoceramics
One thing to investigate is where all the details come from --- psychoceramic outpourings typically have lots and lots of details, and not all of them are lifted from prior sources, but seem rather to have been spun out of whole cloth.
Psychoceramics
We were also surprised by the AUROC of 0.28 for Caricature (2 , 2 , 1) , since we don’t see any reason for worse-than-random performance.
[2206.13498] Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
It would therefore be valuable to design frameworks that do not require access to “normal” models, perhaps by replacing the notion of “anomaly” with that of “deviation from a specification”.
[2206.13498] Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
Developing tools to evaluate these input-independent methods could help transparency researchers draw more robust conclusions that are not contingent on a specific choice of samples
[2206.13498] Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
We will often find that model explanations can detect stark anomalies, but not the subtle ones.
[2206.13498] Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
For each technique, we test whether it can detect 24 anomalous models across 7 categories: adversarial training (Madry et al., 2019), randomized smoothing (Cohen et al., 2019), shape bias (Geirhos et al., 2019), backdoors (Li et al., 2021), spurious features, training without data from certain classes, and training on a face-obfuscated version of ImageNet (Yang et al., 2021).
[2206.13498] Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
f we view transparency methods as a way of auditing models, our detection and localization benchmarks are a form of counterauditing —planting intentionally corrupted models to check that they are discovered
[2206.13498] Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior
This indicates that visual concepts were learned within the last bottleneck but were "discarded" or dominated by the added output from the previous ReLU.
CLIP Enrichment Circuits
Instead, we found that the shift to bimodality is sudden.
CLIP Enrichment Circuits
We found that in contrast to backward contributions, which exhibit an exponential distribution, the forward contributions we looked at seemed to exhibit something a less spiky distribution, closer to normal. Additionally, we found that units from earlier and earlier layers were less and less likely to have forward contributions to the final conv layer (4/2/5) near zero, tending toward a bimodal distribution of forward contributions. The earlier units are more "versatile" in the sense that they contribute to more features to a greater degree.
CLIP Enrichment Circuits
The woah inspired Ricky Desktop to develop a score for the triple woah which then actually inspired dancers to choreograph and perform an actual triple woah. Can you program human movement with music? It turns out you can. You use an API called TikTok. That's delightful.
American Idle — Remains of the Day
One of my favorite paragraphs of recent years was one describing the miracle that are Cheetos: To get a better feel for their work, I called on Steven Witherly, a food scientist who wrote a fascinating guide for industry insiders titled, “Why Humans Like Junk Food.” I brought him two shopping bags filled with a variety of chips to taste. He zeroed right in on the Cheetos. “This,” Witherly said, “is one of the most marvelously constructed foods on the planet, in terms of pure pleasure.” He ticked off a dozen attributes of the Cheetos that make the brain say more. But the one he focused on most …
American Idle — Remains of the Day
YouTube has launched almost no creator tools of note ever. WTF.
American Idle — Remains of the Day
Homer and Hesiod invoke the Muses not while wondering what to compose, but as they begin to sing.
Start With Creation - by Simon Sarris
This database needs to be HUGE (> 1T tokens!), or else it doesn’t really help.
RETRO Is Blazingly Fast | Mitchell A. Gordon
Intuitively, you can think of the Lagrangian multiplier λ as the potential energy of an oscillating system.
How we can make machine learning algorithms tunable
wherever you see a linear combination of losses being optimised with gradient descent, this more principled approach could be used.
How we can make machine learning algorithms tunable
can use this Modified Differential Method of Multipliers to tune the balance between the losses in a semantically useful way using stochastic gradient descent, no matter the shape of the invisible Pareto front
How we can make machine learning algorithms tunable
We see that these CNNs are typically not relatively clus- terable
[2103.03386] Clusterability in Neural Networks
Net- works are reliably more clusterable than at initialization, ex- cept those trained with L 2 regularization after pruning
[2103.03386] Clusterability in Neural Networks