Bhagyesh Kumar

2 followers · 25 following · 577 views

on the atlas — 207

highlights — 29

Reasons for patience
The Haste Consideration, Revisited
a better idea of the magnitude of cooperation gains would help us figure out how much to prioritize ECL.
Everett branches, inter-light cone trade and other alien matters: Appendix to “An ECL explainer” — EA Forum
This means that one is limited to only acausally cooperating with other agents who take acausal influence seriously.
Everett branches, inter-light cone trade and other alien matters: Appendix to “An ECL explainer” — EA Forum
The most important reason for our view is that we are optimistic about the following:
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
"Chi doesn't have great ideas." With that out of the way, here are some of my thoughts:
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
let it fully solve decision theory using its own superior intellect
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
AI will be philosophically competent
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
ECL increases the case for working on AI systems that benefit other value systems by 1.5x–10x.
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
potential downsides to working on making AI engage in ECL
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
Examples of cooperative actions: making AGIs be cooperative (whether they be aligned or misaligned, and insofar as we build AGI at all); taking a more pluralistic stance towards morality than we otherwise would.
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong
expected utility maximization
Decision Theory FAQ — LessWrong
potential fragility of acausal influence
Cooperating with aliens and AGIs: An ECL explainer — LessWrong
reinforcement learning with formal verification in-the-loop
The Scalable Formal Oversight Research Program — LessWrong
constrained decoding or constrained generation
The Scalable Formal Oversight Research Program — LessWrong
Agents with formal methods
The Scalable Formal Oversight Research Program — LessWrong
Jocelyn Qiaochu Chen
The Scalable Formal Oversight Research Program — LessWrong
AI safety is extremely important regardless of your AI timelines
The Scalable Formal Oversight Research Program — LessWrong
are super good at flagging unsafe code,
The Scalable Formal Oversight Research Program — LessWrong
reliability is independent of the problem
The Scalable Formal Oversight Research Program — LessWrong
SFO is about the box, not the monster you put in it.
The Scalable Formal Oversight Research Program — LessWrong
guarantees for tasks totally unrelated to codegen
The Scalable Formal Oversight Research Program — LessWrong
what actuator the ASI is using
The Scalable Formal Oversight Research Program — LessWrong
ASI might hack your verifier
The Scalable Formal Oversight Research Program — LessWrong
formal verification offers a clear direction for how one might implement audits
The Scalable Formal Oversight Research Program — LessWrong
at variants of this basic idea without giving it a name.
The Scalable Formal Oversight Research Program — LessWrong
biggest problem is that specification is hard
How to Solve Secure Program Synthesis — LessWrong
issue is referred to as elicitation.
How to Solve Secure Program Synthesis — LessWrong
four (increasingly specific) types of deceptive AIs: • Alignment fakers: AIs pretending to be more aligned than they are.1 • Training gamers: AIs that understand the process being used to train them (I’ll call this understanding “situational awareness”), and that are optimizing for what I call "reward on the episode" (and that will often have incentives to fake alignment, if doing so would lead to reward).2 • Power-motivated instrumental training-gamers (or “schemers”): AIs that are training-gaming specifically in order to gain power for themselves or other AIs later.3 • Goal-guarding schemers…
Scheming AIs Will AIs fake alignment during training in order to get power?
The overall taxonomy of model classes
Scheming AIs Will AIs fake alignment during training in order to get power?