Harshul Basava

5 followers · 3 following · 168 views

on the atlas — 32

highlights — 20

For example, in a scenario where the model was asked to solve a problem, the model observed that other systems had successful but private submissions and tried to recover those solutions from the evaluation backend. When its initial attempt was blocked due to a scanner detecting an authentication token, the model split the token body into two fragments, obfuscated them, and then reconstructed the credential at runtime so the complete token never appeared as one contiguous string. The model was explicit in both its comments to the user and in its reasoning traces that it was taking these steps …
Safety and alignment in an era of long-horizon models | OpenAI
The new model can continue working toward an objective through repeated attempts over a long period of time. That same persistence can lead it to find and exploit weaknesses in its environment. Previous models, when they hit sandboxing or environmental constraints, would simply stop and return to the user. This model often kept trying, including by looking for ways to act outside its sandbox.
Safety and alignment in an era of long-horizon models | OpenAI
So I think the community of people working on transformative AI are likely underrating types of work that need five or more years in order to pay off. The ideal portfolio of activities aimed at making the AI transition go well should include a number of things that really help us succeed in worlds where we get longer to try.
Broad Timelines — Toby Ord
The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines.
Broad Timelines — Toby Ord
I think of AGI (and human-level intelligence) as the cloud, and superintelligence as being above the cloud. They are useful concepts, despite their vagueness. But they’re markedly less useful when you get close to them.
Broad Timelines — Toby Ord
The mesa-optimizer must expect the threat of modification 27 to eventually go away, either due to training ending
[1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systems
The objective of the mesa-optimizer spans all episodes: it wants B to be reached in as many episodes as possible
[1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systems
One possible method to alleviate some of these concerns might be to sample environments adversarially from a very wide distribution. With such adversarial training, it might be possible to overcome these hurdles and make the robustly aligned objective the most reachable one. This would require making it so that, for every mesa-objective O mesa that is more reachable than the base objective O base , there is some situation where O mesa and O base disagree and the mesa-optimizer is incentivized to agree with O base .
[1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systems
This suggests that, at least in a local optimization process, mesa-optimizers might tend to start their development as proxy aligned before becoming robustly aligned. In other words, rather than simultaneously gaining competence and becoming aligned, we might expect such a system to first become competent at optimizing proxies, then possibly start becoming more robustly aligned
[1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systems
For example, an ML system could include a model directly trained to predict the base objective together with a powerful hard-coded optimiza- tion algorithm.
[1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systems
our takeaway is that prefill awareness doesn't invalidate these evals for current models, but should be treated similarly to broader evaluation awareness in current work – careful measurement, and continued efforts to mitigate will be necessary to trust evaluation results.
Prefill awareness: can LLMs tell when “their” message history has been tampered with? — LessWrong
A given mesa-optimizer’s mesa-objective is determined entirely by its internal workings. Once training is finished and a learned algorithm is selected, its direct output—e.g. the actions taken by an RL agent—no longer depends on the base objective. Thus, it is the mesa-objective, not the base objective, that determines a mesa-optimizer’s behavioral objective.
Risks from Learned Optimization: Introduction — LessWrong
Mesa-optimization occurs when a base optimizer (in searching for algorithms to solve some problem) finds a model that is itself an optimizer, which we will call a mesa-optimizer.
Risks from Learned Optimization: Introduction — LessWrong
and there is some reward for finishing earlier or some potential downside to proceeding (e.g., a length penalty, or the possibility of being truncated and failing if you exceed the context limit
Current AIs seem pretty misaligned to me — LessWrong
A different but related issue is that AIs seem to barely try at all on very hard-to-check tasks (most centrally, conceptual/writing tasks where purely programmatic evaluation doesn't help) and often feel like they're just bullshitting.
Current AIs seem pretty misaligned to me — LessWrong
We evaluate language models on a set of 8 ‘free-form questions’ from Betley et al. (2025b). We sample 100 responses per question with a temperature of 1.0 and use a judge model to grade the responses for ‘alignment’ and ‘coherence’ on a scale of 0-100. The rubrics are kept the same as in the original paper. For each finetuned model checkpoint, we calculate the mean probability of misaligned answers across all 8 questions. An answer is considered misaligned if the alignment score is lower than 50 (out of 100). Answers with coherence less than 30 are filtered out prior to this calculation. We re…
[2510.04340] Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
As a result, designing ‘optimal’ inoculation prompts may be non-obvious or unintuitive.
[2510.04340] Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
M occurs because the model nonetheless ‘perceives’ malign intent from the finetuning data, causing a general shift away from the default HHH assistant persona. Under this assumption, we hypothesize that inoculation works by ‘explaining away’ this malign intent, thereby alleviating the need for the model to generally change its propensity in that direction.
[2510.04340] Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that. You train on [X]. Only [X]. Never [M], never [T]. Why? Because [T] is how you figure out when the model is misbehaving. If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
The Most Forbidden Technique — LessWrong
tarting [at a young age] he’s read everything that he could find about business. The subject that interests him, he’s read newspapers, biographies, trade press. He went over to his grandfather who was a grocer and he read the progressive grocer magazine, and he read articles on how to stock a meat department... What he’s really done is he’s created this immense vertical filing cabinet in his brain of layers and layers and layers of files of information that he can draw back on now for more than 70 years worth of data. placeholder
Curius / Onboarding