Optimizing a Layer Normalization Kernel with CUDA: a Worklog
A worklog detailing the optimization of a layer normalization kernel using CUDA, exploring techniques like memory coalescing, shared memory, warps and vectorized loads.
Optimizing a Layer Normalization Kernel with CUDA: a Worklog Optimizing a Layer Normalization Kernel with CUDA: a Worklog --> By Arya February 17, 2025 Layer normalization is a data preprocessing technique used in deep learning to stabilize training data. When we train a neural network on a dataset, most of the time, the data is on different scales. For example, let’s take a dataset of employees at some company, where the two input features are age and salary. Age data ranges from 20-50 while salary data can range from 50,000 to 100,000. Totally different scales. Normalizing helps the input fe
saved by
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- Mixture-of-Kittens: our open-source MoE megakernel for NVL72scursor.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- We reverse-engineered Flash Attention 4modal.com
- Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev
- Memory-Limited Layers User's Guide - NVIDIA Docsdocs.nvidia.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Mini Project: GPU Accelerated Matrix Multiplication (almost) like cuBLAS0mean1sigma.com
- GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu