flâneur

Frontier-scale RL with Kimi K3 | Applied Compute

appliedcompute.com · 1,530 words · saved by 1 readers

Kimi K3, with just shy of 3T parameters, pushes performance of open weight base models closer to the frontier. At such a large scale, we needed to re-architect portions of our stack to ensure that we could train with high efficiency. Full fine-tuning support for Kimi places an extraordinary amount of stress on our training, communication, and storage systems. In this blog post, we'll share some of the improvements we implemented to train Kimi cheaply and efficiently. All of these optimizations are live when you fine-tune Kimi K3 on AC2. To derisk our implementation, we trained a coding agent using Kimi K3 as the base on a challenging long-horizon task. After 50 steps of training completed within a day, we matched the performance that a smaller 300B-class model reached in multiple days. Empirically, we observe a general trend where larger models tend to have higher slope on new tasks. The primary challenge in large-scale model training is the sheer amount of data that needs to be stored

Kimi K3, with just shy of 3T parameters, pushes performance of open weight base models closer to the frontier. At such a large scale, we needed to re-architect portions of our stack to ensure that we could train with high efficiency. Full fine-tuning support for Kimi places an extraordinary amount of stress on our training, communication, and storage systems. In this blog post, we'll share some of the improvements we implemented to train Kimi cheaply and efficiently. All of these optimizations are live when you fine-tune Kimi K3 on AC2. To derisk our implementation, we trained a coding…

saved by

related reading