Lawrence Feng

1 followers · 1 following · 323 views

on the atlas — 29

highlights — 21

prioritizing long-term understanding over short-term output.
The machines are fine. I'm worried about us.
Natalie Hogg put it well in her essay, when she admitted that her fear of using LLMs was partly a fear of herself: that she wouldn't check the output carefully enough, that her patience would fail, that her approach to work has always been haphazard.
The machines are fine. I'm worried about us.
value of scaling AI capabilities whil
You can’t imitation-learn how to continual-learn — LessWrong
But many people seem to take it for granted that completely ceding frontier AI work to people who don't care about AI risk would be preferable because it would slow down timelines! This seems insane to me. How much time would possibly need to be saved for that to be worth it?
Anthropic's leading researchers acted as moderate accelerationists — LessWrong
I don’t believe there is a plausible path to achieving that kind of robustness on that kind of time frame, except by either pausing AI development (potentially for years), or prioritizing security to such a degree that it has a similar effect (e.g., deploying models only in very limited settings).
Responsible Scaling Policy v3 — LessWrong
I feel pretty uncertain about whether the goal of achieving robustness to jailbreaks for these particular uses of concern was worth all of the energy and prioritization it got.
Responsible Scaling Policy v3 — LessWrong
Looking for patterns in the above, I’ve come to the take that the best kind of forcing function is one that sets an ambitious but achievable target.
Responsible Scaling Policy v3 — LessWrong
Like, let’s say I want to work at OpenAI, with the idea that I’m going to advocate for safety-promoting causes, and take actions that are minimally bad for timelines. OK, now I’ve been at OpenAI for a little while. How’s it going so far? What exactly am I working on? Am I actually advocating for the things I was hoping to advocate for? What are my prospects going forward? Am I being indoctrinated and socially pressured in ways that I don’t endorse?
I Would Have Solved Alignment, But I Was Worried That Would Advance Timelines — LessWrong
every young STEM nerd worried about AI risk who decides not to get a PhD in AI because they'd have to publish a paper
I Would Have Solved Alignment, But I Was Worried That Would Advance Timelines — LessWrong
Right now it seems likely that the first AGI and later ASI will be built with utmost caution by people who take AI risk very seriously.
I Would Have Solved Alignment, But I Was Worried That Would Advance Timelines — LessWrong
If this is a crux for you––if you/Anthropic think a pause is so undesirable/unlikely that it's important for the safety of the human race to publicly disparage the possibility of a pause
Responsible Scaling Policy v3 — LessWrong
I think a “balancing” attitude is ultimately better for risk reduction.
Responsible Scaling Policy v3 — LessWrong
If they believe that risks from other actors’ continued scaling are unacceptably high, and they have exhausted other avenues for preventing these risks, including advocating intensively for regulatory action, then in some scenarios they may continue scaling themselves
Responsible Scaling Policy v3 — LessWrong
One way RSPs could fail to reduce risk — or even increase it — would be if they resulted in the following dynamic: “Cautious AI developers end up slowing down in order to avoid risks, while incautious AI developers move forward as fast as they can.”
Responsible Scaling Policy v3 — LessWrong
When reasoning-oriented RL converges, we utilize the resulting checkpoint to collect SFT (Supervised Fine-Tuning) data for the subsequent round
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
we collect thousands of cold-start data to fine-tune the DeepSeek-V3-Base as the starting point for RL.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
We fine-tune DeepSeek-V3-Base for two epochs using the above curated dataset of about 800k samples.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
It seems hard to scan a trained neural network and locate the AI’s learned “tree” abstraction. For very similar reasons, it seems intractable for the genome to scan a human brain and back out the “death” abstraction, which probably will not form at a predictable neural address. Therefore, we infer that the genome can’t directly make us afraid of death by e.g. specifying circuitry which detects when we think about death and then makes us afraid. In turn, this implies that there are a lot of values and biases which the genome cannot hardcode…
The shard theory of human values - LessWrong
CoT-as-rationalization, where the model explains an answer it has already computed, and CoT-as-computation, where the CoT is a necessary part of the computation itself.
When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors
The ultimate property we need for safety is monitorability: whether we can use the CoT to determine if a model is taking dangerous actions
When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors
Despite this intuitive appeal, the reliability of CoT monitoring has been cast into doubt by a significant body of recent research on “unfaithfulness” (Turpin et al., 2023; Lanham et al., 2023; Chua and Evans, 2025; Chen et al., 2025; Arcuschin et al., 2025). This work shows that models can fabricate justifications, omitting the true causes of their decisions. This is a critical issue for many applications; for instance, a fabricated CoT could undermine an audit that’s looking for bias in a model used for hiring decisions (Karvonen and Marks, 2025).
When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors