[2602.16703] Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology
Abstract:Large language models (LLMs) perform strongly on biological benchmarks, raising concerns that they may help novice actors acquire dual-use laboratory skills. Yet, whether this translates to improved human performance in the physical laboratory remains unclear. To address this, we conducted a pre-registered, investigator-blinded, randomized controlled trial (June-August 2025; n = 153) evaluating whether LLMs improve novice performance in tasks that collectively model a viral reverse genetics workflow. We observed no significant difference in the primary endpoint of workflow completion (5.2% LLM vs. 6.6% Internet; P = 0.759), nor in the success rate of individual tasks. However, the LLM arm had numerically higher success rates in four of the five tasks, most notably for the cell culture task (68.8% LLM vs. 55.3% Internet; P = 0.059). Post-hoc Bayesian modeling of pooled data estimates an approximate 1.4-fold increase (95% CrI 0.74-2.62) in success for a "typical" reverse genetics task under LLM assistance. Ordinal regression modelling suggests that participants in the LLM arm were more likely to progress through intermediate steps across all tasks (posterior probability of a positive effect: 81%-96%). Overall, mid-2025 LLMs did not substantially increase novice completion of complex laboratory procedures but were associated with a modest performance benefit. These results reveal a gap between in silico benchmarks and real-world utility, underscoring the need for physical-world validation of AI biosecurity assessments as model capabilities and user proficiency evolve.
[2602.16703] Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology --> Computer Science > Computers and Society arXiv:2602.16703 (cs) [Submitted on 18 Feb 2026] Title: Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology Authors: Shen Zhou Hong , Alex Kleinman , Alyssa Mathiowetz , Adam Howes , Julian Cohen , Suveer Ganta , Alex Letizia , Dora Liao , Deepika Pahari , Xavier Roberts-Gaal , Luca Righetti , Joe Torres View a PDF of the paper titled Measuring Mid-2025 LLM-Assistance on Novice Performance in Biology, by Shen Zhou Hong and 11 other authors View PDF Abstrac
saved by
related reading
- Thoughts on AI in academiatheinfinitesimal.substack.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Large Language Diffusion Modelsarxiv.org
- Language Models can Solve Computer Tasksarxiv.org
- Predicting Empirical AI Research Outcomes with Language Modelsarxiv.org
- Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com
- [2506.13023] A Practical Guide for Evaluating LLMs and LLM-Reliant Systemsarxiv.org
- [2606.07527] Post-training is (Massive) Supervised Learningarxiv.org
- Can LLMs generate novel research ideas?arxiv.org
- [2603.08640] PostTrainBench: Can LLM Agents Automate LLM Post-Training?arxiv.org
- [2605.20873] PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Modelsarxiv.org
- [2605.27288] It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertaintyarxiv.org