✳flâneur — a map of the web's best reading

Training Deep Networks with Data Parallelism in Jax

mishalaskin.com · 1,590 words · saved by 1 readers

Misha Laskin personal website. Includes a blog and projects focused on artifical intelligence.

Training Deep Networks with Data Parallelism in Jax You , Sun Feb 19 2023 • large model engineering Back One of the main challenges in training large neural networks, whether they are LLMs or VLMs, is that they are too large to fit on a single GPU. To address this issue, their training can be parallelized across multiple GPUs. This means either parallelizing the data or model to distribute computation across several devices. In this post, we'll cover batch splitting, also known as data parallelism, and show how to use JAX's pmap function to parallelize computations across multiple devices. Par

Explore this link on the map →

related reading