flâneur

DeltaNet Explained (Part II) | Songlin Yang

sustcsonglin.github.io · 2,689 words · saved by 1 readers

This blog post series accompanies our NeurIPS ‘24 paper - Parallelizing Linear Transformers with the Delta Rule over Sequence Length (w/ Bailin Wang, Yu Zhang, Yikang Shen and Yoon Kim). You can find the implementation here and the presentation slides here. Ok, we’ve seen in the previous section that DeltaNet does really well on these diagnostic synthetic tasks. So now we just need to scale it up to modern LMs, right? Well, it turns out it’s not that simple. In particular, the original DeltaNet treated DeltaNet as a pure RNN which required O(L) sequential steps, which is inefficient on modern hardware such as GPUs with massive parallel processing capabilities. We thus seek strategies to parallelize DeltaNet across sequence length to enable hardware-efficient training. In this post, we first discuss parallel scan as a interesting-but-impractical strategy for parallelizing DeltaNet. We then give another algorithm for parallelization that is more efficient in practice. Let’s start with De

DeltaNet Explained (Part II) | Songlin Yang DeltaNet Explained (Part II) An algorithm that parallelizes DeltaNet computation across the sequence length dimension This blog post series accompanies our NeurIPS ‘24 paper - Parallelizing Linear Transformers with the Delta Rule over Sequence Length (w/ Bailin Wang , Yu Zhang , Yikang Shen and Yoon Kim ). You can find the implementation here and the presentation slides here . Part I - The Model Part II - The Algorithm Part III - The Neural Architecture Parallel Scan for DeltaNet: A Failed Attempt Ok, we’ve seen in the previous section that DeltaNet

related reading