Emanuel Ruzak

2 followers · 1 following · 413 views

on the atlas — 49

highlights — 24

A computer program X is asked one of two questions: Would you like some chocolate? Here's the source code of another program Y. Do you accept it as your successor? You would like your program to accept the chocolate. And you would like it to accept a broad range of successors. But you need to make sure that every chain of successors continues to accept the chocolate.
Working through a small tiling result — LessWrong
because they do not have a good solution to the problem of aggregating correlated evidence.
Automated Alignment is Harder Than You Think — LessWrong
Active learning prediction: Given a task, and a set of N samples, predict which would improve loss the most if trained on. Early-exit uncertainty estimation: Given a task, e.g. GPQA, question q~t, and response x~f(-|q): predict the first token where confidence in the final answer is above 90%. Validation can be done via fine-tuning a separate classifier, or by optimising over prompts.
Prospects for Alignment Automation: Interpretability Case Study — LessWrong
Yet if a medical breakthrough were to emerge—and especially effective anti-aging therapies—then the optimal time to launch AGI could be pushed out considerably. In principle, such a breakthrough could come from either pre-AGI forms of AI (or specialized AGI applications that don’t require full deployment) or medical progress occurring independently of AI. Such developments are more plausible in long-timeline scenarios where AGI is not developed for several decades
Optimal Timing for Superintelligence
Suppose that without superintelligence, the average remaining life expectancy is 40 years. 10 With superintelligence, we assume that rejuvenation medicine could reduce mortality rates to a constant level similar to that currently enjoyed by healthy 20-year-olds in developed countries, which corresponds to a life expectancy of around 1,400 years. 11 This is conservative, since superintelligence could also mitigate many non-aging causes of death—such as infectious diseases, accidents, and suicidal depression. It is also conservative because it ignores more radical possibilities (like mind upload…
Optimal Timing for Superintelligence
Antimatter-catalyzed nuclear pulse propulsion Bussard ramjet Fission-fragment rocket Fission sail Fusion rocket Gas core reactor rocket Nuclear electric rocket Nuclear photonic rocket Nuclear pulse propulsion Nuclear salt-water rocket Nuclear thermal rocket Radioisotope rocket
Gas core reactor rocket
CCS + Evolution Strategy Refinement (PGR=0.93).
Automated Weak-to-Strong Researcher
One proposed approach is Debate, introduced by (Irving et al., 2018) in which two equally-capable AI systems argue with each other over the answer to a question (Michael et al., 2023). Then a judge, who can be either a human or a weaker model, tries to discern which debater is defending the correct answer (Arnesen et al., 2024). This setup mirrors the adversarial dynamics of a judicial process, where opposing parties present their cases in order to persuade a fact-finder who was not present at the scene of the events and therefore does not know what actually happened. Fundamental to this techn…
AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs
Exploration guarantees: We need a solution to exploration hacking (in which the debaters fail to explore certain highly rewarded strategies to avoid having them reinforced). For example, if we can elicit “bad” contexts in which models would answer differently under different circumstances, we can use this to detect exploration hacking (and possibly prevent it during training). Notably, with a (mostly) trusted reward signal, we don’t need to explicitly look for “scheming” – we only need an “exploration guarantee” showing that the model is not foregoing substantial reward and therefore likely op…
An alignment safety case sketch based on debate — LessWrong
De Blanc 2011 studies so called "ontological crises". That is, de Blanc examines the problem of translating a reward function from one ontology into another. Here, we avoid this problem by considering reward functions that are automatically defined in all ontologies. That said, it might still be interesting to think about how to specify our type of reward function starting from a reward function that is only defined in one particular ontology. We will return to this in the Discussion section.
Reinforcement learning with imperceptible rewards — AI Alignment Forum
Suppose your world is an infinite sequence of bits, and one hypothesis you might have about the world is maybe all the odd bits are equal to zero. This hypothesis doesn’t tell us anything about even bits. It’s only a hypothesis about odd bits, and it’s very easy to describe it as such a convex set of probability distributions over all of the bits. We just consider all probability distributions that predict that the odd bits will be zero with probability 1, and without saying anything at all about the even bits. They can be anything. They can even be uncomputable. You’re not trying to have a pr…
My take on Vanessa Kosoy's take on AGI safety — AI Alignment Forum
Quite the contrary. Inner misalignment manifests as the possibility of malign hypotheses in the prior. See also Formal Solution to the Inner Alignment Problem and discussion in comment section there.
My take on Vanessa Kosoy's take on AGI safety — AI Alignment Forum
We define a variant of reinforcement learning in which the reward is not perceived directly, but can be estimated at any given moment by some (possibly costly) experiment.
Reinforcement learning with imperceptible rewards — AI Alignment Forum
In prover-estimator debate, the prover (Alice) decomposes a problem into subclaims. The estimator (Bob) has the sole task of assigning probabilities to the prover’s subclaims. The prover uses these probabilities to decide which subclaim to recursively debate further. This departs from previous debate protocols, where the opponent selects which subclaim to challenge
Avoiding Obfuscation with Prover-Estimator Debate
Explicit uncertainty estimates. Our protocols rely heavily on the ability to ask the estimator B to produce probability estimates for subclaims that arise in the debate. The payoffs are also determined by these estimates. In a practical setting one could use multiple samples from a generative model to produce uncertainty estimates, or directly use token probabilities of the answer from an LLM. For example, after an LLM A generates an argument, one could prompt both A and opposing LLM B to read through the argument step-by-step, and at each step answer whether the claim seemed true. Token proba…
Avoiding Obfuscation with Prover-Estimator Debate
The leader-follower structure implies that for every training update made to A, we should make multiple updates to B. This ensures that whenever A receives a gradient update, it comes from an episode of debate where B is playing an approximate “best response” to A’s current strategy. This type of training algorithm makes particular sense in the asymmetric setting where A is proposing a complex solution to some problem, and is asked to defend the solution. In this case, we would like to train B to ensure the ability to identify flaws in A’s solutions, before providing a new update to A.
Avoiding Obfuscation with Prover-Estimator Debate
Asymmetry between the debaters. Much empirical work on debate thus far has had symmetric debaters, distinguished only by which debater speaks first. Our protocol suggests that it is useful to have asymmetry between the debaters. The debater A should be tasked with proposing a solution and providing evidence to defend it, while debater B should attempt to identify flaws, and evaluate the plausibility of the evidence. This setup also makes sense for empirical debate research on open-ended tasks, as the first debater will need to propose a solution and then defend its correctness.
Avoiding Obfuscation with Prover-Estimator Debate
von Neumann's minimax theorem about two-player zero-sum games published in 1928,[2] which is considered the starting point of game theory.
Minimax theorem
Adversarial techniques such as debate
Recommendations for Technical AI Safety Research Directions
On the other hand, there is substantial risk that activation monitors are not robust to distributional shifts. One particularly important distribution shift is the shift from fixed datasets of synthetic data to data generated by the model itself. If we don’t have model-generated examples of the bad outputs we would like to detect, then we’ll need to synthetically generate outputs in some other way to train activation monitors. We would then need to verify that monitors trained using these outputs generalize to actual dangerous model generations of interest. While there are signs of life for ac…
Recommendations for Technical AI Safety Research Directions
All students must be nominated by an eligible university during the application period; direct applications from students are not accepted.
Google PhD Fellowship Submission Manager
Latin America: Universities may nominate any number of eligible students.
Google PhD fellowship program
Apply by April 30, 2026
Google PhD fellowship program
Ensuring the privacy, security, and safety of AI systems
Google PhD fellowship program