Build a Transformer in JAX from scratch: how to write and train your own models | AI Summer

How to develop and train a Transformer with JAX, Haiku and Optax. Learn by example how to code Deep Learning models in JAX

In this tutorial, we will explore how to develop a Neural Network (NN) with JAX. And what better model to choose than the Transformer . As JAX is growing in popularity, more and more developer teams are starting to experiment with it and incorporating it into their projects. Despite the fact that it lacks the maturity of Tensorflow or Pytorch, it provides some great features for building and training Deep Learning models. For a solid understanding of JAX basics, check my previous article if you haven’t already. Also you can find the full code in our Github repository . One of the common proble

saved by

• Soham Shah
• Alex Shaw

related reading

• Transformer (deep learning) - Wikipediaen.wikipedia.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Why You Should (or Shouldn't) be Using Google's JAX in 2023assemblyai.com
• Why You Should (or Shouldn't) be Using Google's JAX in 2023assemblyai.com
• A guide to JAX for PyTorch developers | Google Cloud Blogcloud.google.com
• Transformers from Scratche2eml.school
• The Annotated Transformernlp.seas.harvard.edu
• GitHub - jax-ml/jax: Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more · GitHubgithub.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformers from scratch | peterbloem.nlpeterbloem.nl