Optimizing a Layer Normalization Kernel with CUDA: a Worklog
A worklog detailing the optimization of a layer normalization kernel using CUDA, exploring techniques like memory coalescing, shared memory, warps and vectorized loads.
Optimizing a Layer Normalization Kernel with CUDA: a Worklog Optimizing a Layer Normalization Kernel with CUDA: a Worklog --> By Arya February 17, 2025 Layer normalization is a data preprocessing technique used in deep learning to stabilize training data. When we train a neural network on a dataset, most of the time, the data is on different scales. For example, let’s take a dataset of employees at some company, where the two input features are age and salary. Age data ranges from 20-50 while salary data can range from 50,000 to 100,000. Totally different scales. Normalizing helps the input fe
saved by
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- What happens when you run a CUDA kernelfergusfinn.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Implementing a fast Tensor Core matmul on the Ada Architecture | spatters.caspatters.ca
- siboehmsiboehm.com