[2510.05056] Modeling Student Learning with 3.8 Million Program Traces

Abstract:As programmers write code, they often edit and retry multiple times, creating rich "interaction traces" that reveal how they approach coding tasks and provide clues about their level of skill development. For novice programmers in particular, these traces reflect the diverse reasoning processes they employ to code, such as exploratory behavior to understand how a programming concept works, re-strategizing in response to bugs, and personalizing stylistic choices. In this work, we explore what can be learned from training language models on such reasoning traces: not just about code, but about coders, and particularly students learning to program. We introduce a dataset of over 3.8 million programming reasoning traces from users of Pencil Code, a free online educational platform used by students to learn simple programming concepts. Compared to models trained only on final programs or synthetically-generated traces, we find that models trained on real traces are stronger at modeling diverse student behavior. Through both behavioral and probing analyses, we also find that many properties of code traces, such as goal backtracking or number of comments, can be predicted from learned representations of the students who write them. Building on this result, we show that we can help students recover from mistakes by steering code generation models to identify a sequence of edits that will results in more correct code while remaining close to the original student's style. Together, our results suggest that many properties of code are properties of individual students and that training on edit traces can lead to models that are more steerable, more predictive of student behavior while programming, and better at generating programs in their final states. Code and data is available at this https URL

Modeling Student Learning with 3.8 Million Program Traces Alexis Ross⋆1[0009−0005−1861−0928] , Megha Srivastava*2[0000−0003−0985−7062] , Jeremiah Blanchard3[0000−0003−2995−5102] , and Jacob Andreas1[0000−0002−3141−5845] 1…

saved by

• Laerdon Kim

related reading

• [2511.04427] Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectsarxiv.org
• SWE-chat: Coding Agent Interactions From Real Users in the Wildarxiv.org
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• [2203.07814] Competition-Level Code Generation with AlphaCodearxiv.org
• [2608.20319] Inducing Task Models from Computer-Use Tracesarxiv.org
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Deep Knowledge Tracingstanford.edu
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Program Synthesis with Large Language Modelsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org