Very Simple MoE Intro

1a3orn.com · 2,842 words · saved by 1 readers

I've written a notebook containing a model and training code for a deliberately-simple character-level Mixture-of-Experts Transformer model. I've removed as many complications as I can from it while still retaining better performance on a per-FLOP basis than a similar dense Transformer. The only non-default dependencies are PyTorch, and WanDB. The following explains how this model works, and how Mixture-of-Experts work in general. (Note: The implementation is closest to a Switch Transformer. But it isn't really that. For instance, my implementation is trained solely on an autoregressive language modeling task, while Switch Transformer uses a masked token language modeling task; I use a different position encoding method; and so on. And a big chunk of the contribution of Switch Transformer is the routing across multiple GPUs in a particular way, while I completely ignore multi-GPU training or inference.) Why would we care about MoEs? Well, MoE Transformers generally take less compute th

Very Simple MoE Intro 1 A 3 O R N 1 A 3 O R N Very Simple MoE Intro Created: 2024-02-03 Wordcount: 3k Tags: essays machine-learning effort-post I've written a notebook containing a model and training code for a deliberately-simple character-level Mixture-of-Experts Transformer model. I've removed as many complications as I can from it while still retaining better performance on a per-FLOP basis than a similar dense Transformer. The only non-default dependencies are PyTorch, and WanDB. The following explains how this model works, and how Mixture-of-Experts work in general. ( Note: The implement

Very Simple MoE Intro

related reading