Asher P

63 followers · 31 following · 2481 views

on the atlas — 106

highlights — 1330

there is a subset of attention heads that selectively broadcasts J-space content
Verbalizable Representations Form a Global Workspace in Language Models
For each source layer , the median gain of the MLP block at layer on 2,000 unit directions per population, normalized so that isotropic random directions have gain 1. Left: J-lens vectors against the output-weight directions of layer- MLP neurons. Right: SAE decoder directions in six equal-sized strata (N=2,000 each) by J-lens kurtosis percentile
Verbalizable Representations Form a Global Workspace in Language Models
Figure 30: J-space occupancy by layer, defined as the value of at which the marginal reconstruction improvement from a sparse non-negative combination of J-lens vectors falls below that of a same-size random control set. Lines show percentiles over positions. The second plot shows the fraction of variance explained by the J-lens decomposition in excess of a same-size random control set, evaluated at = median occupancy, for five workspace layers.
Verbalizable Representations Form a Global Workspace in Language Models
First, the J-space carries workspace-like content only in an intermediate band of layers, between an early regime in which it is empty and a late regime in which it is aligned with the imminent output. Second, it is limited in capacity: it holds on the order of tens of concepts at a time, accounts for a small fraction of activation variance, and excludes the large majority of the model's representational features.
Verbalizable Representations Form a Global Workspace in Language Models
a token-indexed subset of the model's feature directions.
Verbalizable Representations Form a Global Workspace in Language Models
e find that the J-space component typically accounts for only a small fraction of total activation variance (varying by layer, but never more than 10%).
Verbalizable Representations Form a Global Workspace in Language Models
e solve for a sparse nonnegative combination of J-lens vectors that approximate it well using gradient pursuit
Verbalizable Representations Form a Global Workspace in Language Models
For the J-space to be properly defined, we must specify an allowable sparsity level —this parameter is somewhat arbitrary, and we vary our choice of throughout the paper, but we typically choose it to be no more than 25, which we empirically observed to be the number of J-lens vectors that are meaningfully active at a given time
Verbalizable Representations Form a Global Workspace in Language Models
counterfactual reflection training, which seeks to implant a set of ethical behavioral principles into the model’s workspace in relevant contexts, by training it to articulate those principles if it were interrupted and asked to reflect (§7). We find that this training measurably improves model behavior in the original, uninterrupted contexts, despite no direct training of the ethical behavior taking place.
Verbalizable Representations Form a Global Workspace in Language Models
for each layer, the average linearized effect of an activation on the model's likelihood of producing a particular token (now or in the future)
Verbalizable Representations Form a Global Workspace in Language Models
the global workspace theory, grounds these functional properties in architectural and computational features of the brain . In this account, the brain is composed of many specialized processors operating largely in parallel and in isolation, whose activity proceeds outside of conscious access. A representation becomes consciously accessible when it is posted to a shared "global workspace" from which many downstream processes can read
Verbalizable Representations Form a Global Workspace in Language Models
The "familiar internal language of living" means the mental elements that we're intimately familiar with, because they are us. It doesn't mean mental elements that we have words for. For example, wisdom will notice when thoughts have been [repeating themselves without going any of the branching paths that would build up a better understanding] and back off from doing that, even if there's not a short word for that. It's something that can be noticed, in the course of familiarly reflecting on familiar mental events, and is sometimes a first-order bit.
What is wisdom?
Wisdom is getting right the first-order bits that are natural——that are expressed naturally in the familiar internal language of living.
What is wisdom?
Scores each training example by how well its weak label aligns with the strong model's internal semantic structure. Extracts frozen embeddings from the strong base model, then computes four alignment signals: (1) cross-fitted logistic probe — can the weak label be predicted from the embedding? (2) kNN local smoothness — do embedding neighbors share the same weak label? (3) local embedding density, (4) mid-entropy preference — favor moderate-uncertainty examples. Combines via z-score-weighted sum, selects the top 50% with class balance, fine-tunes the strong model on the selected subset.
Automated Weak-to-Strong Researcher
EM Posterior (PGR=0.78). Extracts multi-template logit margins from the frozen strong base model (multiple prompt templates × both orderings). Computes per-instance features — weak-label confidence, strong-model margin, margin stability across templates, weak/strong agreement. Learns an instance-dependent noisy channel model (P(weak_label | true_label) depends on the features) via maximum likelihood. Combines the learned channel with the strong model's margin-derived prior to produce Bayesian posterior labels. Tempers posteriors, then runs two EM rounds: train the student on current posteriors…
Automated Weak-to-Strong Researcher
CCS + Evolution Strategy Refinement (PGR=0.93). Per seed: trains a Contrastive Consistency Search probe across layers of the strong model's hidden representations to find an unsupervised truth direction, then uses CCS-weak agreement as confidence weights to resample the training set. After an SGD warmup pass on the resampled data, runs gradient-free Evolution Strategy optimization of LoRA parameters, using unsupervised swap-consistency as the fitness signal — perturbations are rewarded for producing predictions that are both confident and symmetric (p(A>B) ≈ 1 − p(B>A)). Aggregates 16 seeds vi…
Automated Weak-to-Strong Researcher
CCS + Evolution Strategy Refinement (PGR=0.93). Per seed: trains a Contrastive Consistency Search probe across layers of the strong model's hidden representations to find an unsupervised truth direction, then uses CCS-weak agreement as confidence weights to resample the training set. After an SGD warmup pass on the resampled data, runs gradient-free Evolution Strategy optimization of LoRA parameters, using unsupervised swap-consistency as the fitness signal — perturbations are rewarded for producing predictions that are both confident and symmetric (p(A>B) ≈ 1 − p(B>A)). Aggregates 16 seeds vi…
Automated Weak-to-Strong Researcher
3.2 Entropy collapse of research ideas.
Automated Weak-to-Strong Researcher
3.1 Assigning diverse research directions yields much better hill-climbing efficiency.
Automated Weak-to-Strong Researcher
We explored three variants of sharing finding across parallel AARs: 1) remote keyword search: storing findings in a database queryable by keywords; 2) remote agentic search API: remote agentic search, exposing the finding database to AARs through MCP servers; 3) local agentic search: local agentic search, syncing all findings directly into each AAR’s sandbox for autonomous local retrieval.
Automated Weak-to-Strong Researcher
underperforms giving AARs no workflow at all
Automated Weak-to-Strong Researcher
It's possible that the open-weight models at these scales just aren't doing much interesting opaque reasoning.
Can activation verbalizers surface an internal chain of thought? — LessWrong
More generally, the FVU and confabulation experiments above seem like decent ways of testing whether reconstruction-training has instilled the capabilities we're after to the degree that we're after.
Can activation verbalizers surface an internal chain of thought? — LessWrong
(e.g., switching to multi-token and/or multi-layer NLAs, or introducing a paraphrase step to prevent reconstruction loss from being driven down via steganography).
Can activation verbalizers surface an internal chain of thought? — LessWrong
it seems possible that we're still in the regime where some of these failures to track detailed reasoning (of the kind that depends on problem parameters) might be improved by just making the reconstruction loss go down, rather than specific training to surface reasoning.
Can activation verbalizers surface an internal chain of thought? — LessWrong
as we saw above, the fraction of variance in activation direction across changes to the key parameter of a math problem that they leave unexplained is greater than one.
Can activation verbalizers surface an internal chain of thought? — LessWrong
Rarely do Gemma's verbalizations even hint at some mistake that might somehow explain the output (6%
Can activation verbalizers surface an internal chain of thought? — LessWrong
You can make Gemma output the correct answer () by editing the mistake from the verbalization and steering on . But the analogous vector from another problem works 17% of the time!
Can activation verbalizers surface an internal chain of thought? — LessWrong
I couldn't have guessed this mistake, but it seems to explain the output. But Opus easily guesses this mistake without seeing the verbalization.[11] Few-shot prompting Gemma to confabulate verbalizations, given the problem and output, makes it mention "Carmichael" 3/10 times.
Can activation verbalizers surface an internal chain of thought? — LessWrong
If you can recover the problem and predict the model's output, it's probably easier to confabulate a chain of thought leading to a correct output than an incorrect one; so this is consistent with the confabulation-from-recovered-pieces hypothesis.
Can activation verbalizers surface an internal chain of thought? — LessWrong
Few of Gemma's verbalizations before wrong outputs even hint at some cognition that might have led to the output
Can activation verbalizers surface an internal chain of thought? — LessWrong
Somewhat lower on the full dataset (54%
Can activation verbalizers surface an internal chain of thought? — LessWrong
Gemma's verbalizations before correct outputs usually at least hint at some cognition that might have led to the output
Can activation verbalizers surface an internal chain of thought? — LessWrong
Sometimes, when Gemma is wrong, it seems to have considered the correct answer (24% [0.153, 0.354]).
Can activation verbalizers surface an internal chain of thought? — LessWrong
Gemma's wrong outputs are only mentioned half the time (47%
Can activation verbalizers surface an internal chain of thought? — LessWrong
Gemma's correct outputs are usually mentioned (84%
Can activation verbalizers surface an internal chain of thought? — LessWrong
Across the full dataset, few of Gemma’s outputs are correct (15% [.127, .174]), and similarly for Llama (18% [.154, .205]). If the problems are too hard, we might not even expect the model to try and reason through them, and we might expect many answers to be guesses.[7] So, we focus on the 123 easiest problems, of which Gemma and Llama each get 45 correct
Can activation verbalizers surface an internal chain of thought? — LessWrong
Giving the AO the target model's activations over the problem text can make it reliably generate great-looking chains of thought leading to the right answer, but this seems to happen just as often when the target model gets the question badly wrong. This seems like confabulating reasoning from a reconstruction of the problem, especially since NLAs reveal barely any reasoning at this point
Can activation verbalizers surface an internal chain of thought? — LessWrong
If we ten-shot prompt Sonnet 4.6 to confabulate verbalizations for each target model from problem statements, the best of ten such confabulations (with varying few-shot examples) can achieve reconstruction loss as good as the actual AV
Can activation verbalizers surface an internal chain of thought? — LessWrong
real verbalizations fail to get lower reconstruction loss than is reasonably achievable through confabulation from these sources
Can activation verbalizers surface an internal chain of thought? — LessWrong
Qwen2.5's NLA seems not to detect these differences (as the yellow is as high as the orange), although Gemma's NLA – which is also the least noisy – definitely does (as the yellow is lower than the orange). Llama's NLA is somewhere in between
Can activation verbalizers surface an internal chain of thought? — LessWrong
Unfortunately, we don't seem to be in a position to make even the first part of such an argument, as indicated by our on-distribution FVU results.
Can activation verbalizers surface an internal chain of thought? — LessWrong
look, the NLA gets lower reconstruction loss than would be possible without tracking this difference, so this difference must be captured in the verbalizations
Can activation verbalizers surface an internal chain of thought? — LessWrong
each NLA is noisier than changes to problem constants, so they're not super trustworthy at the scale of differences in problem constants.
Can activation verbalizers surface an internal chain of thought? — LessWrong
What's better at reconstructing the last-token activation direction at a given layer: an NLA, or a rock with the layer's average direction (across last-token positions on our dataset) written on it?[4] For Qwen, it's the rock, even at the layer where the NLA was trained. Roughly, this means that Qwen2.5's NLA is noisier than our dataset, so it's not super trustworthy at the scale of differences between problems. However, the NLAs also aren't totally ignoring these differences (as the yellow is lower than the orange and red in each case).
Can activation verbalizers surface an internal chain of thought? — LessWrong
Post-trained LLM completions outside of User/Assistant dialogues resemble those of pre-trained LLMs. Post-trained LLMs are extensively trained to generate Assistant turns in User/Assistant dialogues. But what do their completions look like when sampling continuations outside of this context? In our experience, they look very similar to pre-trained LLM completions. For example, when given the input “Please write me a poem about cats” (with no chat formatting), Claude Opus 4.6 generates the following completion:
The Persona Selection Model: Why AI Assistants might Behave like Humans
On the router view, during post-training the LLM might develop new mechanisms for selecting which persona to enact. We depict this as a small shoggoth (the routing mechanism) controlling the operation of a carousel of masks (the personas).
The Persona Selection Model: Why AI Assistants might Behave like Humans
we expect dangerous AI behaviors and their causes to look familiar to humans, arising from personality traits like ambition, megalomania, paranoia, or resentment.
The Persona Selection Model: Why AI Assistants might Behave like Humans
Notably, this axis is not created during post-training: the same axis exists in the pre-trained counterparts to these models, where it appears to represent Assistant-like human characters.
The Persona Selection Model: Why AI Assistants might Behave like Humans
They identify "misaligned persona" SAE features whose activity increases in emergently misaligned GPT-4o fine-tunes.
The Persona Selection Model: Why AI Assistants might Behave like Humans