✳flâneur — a map of the web's best reading

Does Muon improve regulatory DNA learning? Part 1. — Origin Bio

origin.bio · 8,905 words · saved by 1 readers

Optimizers dictate the parameter update rules that govern the convergence trajectory of Stochastic Gradient Descent (SGD). At present, variations of Adam1 dominate DNA sequence modeling (e.g., AlphaGenome3, Enformer4, Evo 25). In natural language, Muon2 has been shown to achieve faster convergence, but it is unclear whether this advantage transfers to nucleotide modeling. The reason is that these domains differ in several fundamental ways. DNA uses a vocabulary of just 4 nucleotides whose token frequencies are far more uniform than the heavy-tailed Zipfian distribution over ~100k tokens in language22. The syntactical rules are explicit and local (transcription factor binding motifs typically span ~6–20bp), and at the short context lengths we study, the signal is dominated by these independent local motif grammars rather than the long-range dependencies that characterize language modeling. In this blog, we explore the behavior of recent optimizers on DNA sequence modeling in regulatory

Your browser does not support the video tag. Does Muon improve regulatory DNA learning? Part 1. March 5, 2026 · Viraj Doshi Highlights AdamW = Adam with Decoupled Weight Decay MuonW = Muon with Decoupled Weight Decay AdamH = Adam with Hyperball MuonH = Muon with Hyperball The DNA domain differs from natural language in many ways; a much smaller vocabulary (4 tokens vs ~100k) with near-uniform token frequencies, and local 6-20bp motif grammar that dominates sequence structure, so the observed efficiency gains of Muon in NLP are not guaranteed to transfer. We benchmark Muon and Adam varia

Explore this link on the map →

saved by

related reading