Optimizing a Layer Normalization Kernel with CUDA: a Worklog
A worklog detailing the optimization of a layer normalization kernel using CUDA, exploring techniques like memory coalescing, shared memory, warps and vectorized loads.
Optimizing a Layer Normalization Kernel with CUDA: a Worklog Optimizing a Layer Normalization Kernel with CUDA: a Worklog --> By Arya February 17, 2025 Layer normalization is a data preprocessing technique used in deep learning to stabilize training data. When we train a neural network on a dataset, most of the time, the data is on different scales. For example, let’s take a dataset of employees at some company, where the two input features are age and salary. Age data ranges from 20-50 while salary data can range from 50,000 to 100,000. Totally different scales. Normalizing helps the input fe
Explore this link on the map →saved by
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- Worklog: Optimising GEMM on NVIDIA H100 for cuBLAS-like Performance (WIP) – Hamza's Bloghamzaelshafie.bearblog.dev
- Memory-Limited Layers User's Guide - NVIDIA Docsdocs.nvidia.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Mini Project: GPU Accelerated Matrix Multiplication (almost) like cuBLAS0mean1sigma.com
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- Matrix Multiplication CUDA - ECA - GPU 2018-2019ecatue.gitlab.io
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- Learning CUDA by optimizing matrix-vector multiplication (SGEMV) for cuBLAS-like performance - A worklog – Maharshi's blogmaharshi.bearblog.dev
- Outperforming cuBLAS on H100: a Worklogcudaforfun.substack.com