LIME YAO

0 followers · 759 views

on the atlas — 24

highlights — 37

30 TFLOPs that the GPU can provide
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
High occupancy is useful because it allows us to hide the high latency of our operations, by having a bigger pool of issue-able instructions available
How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
onfiguration for the build cache. cache_root
API Reference — TensorRT LLM
-llama/Llama-2-7b-chat-hf")
KV cache strategies
However, for small requests, TMA loads have higher latency than regular async data copies because of the address generation overhead. Thus, NVIDIA recommends programmers to use TMAs for large data copies to amortize the overhead. For example, in LLM inference, TMA is not suitable for workloads that load KV cache in small chunks, but works well when each chunk is a multiple of 16 bytes
NVIDIA Tensor Core Evolution: From Volta To Blackwell
up to 64 “resident warps” per S
How to Think About GPUs | How To Scale Your Model
TPUs have a lot more fast cache memory. TPUs also have a lot more VMEM than GPUs have SMEM (+TMEM), and this memory can be used for storing weights and activations in a way that lets them be loaded and used extremely fast. This can make them faster for LLM inference if you can consistently store or prefetch model weights into VMEM.
How to Think About GPUs | How To Scale Your Model
A GPU programmer can just launch dozens of different kernels, each running on a totally independent SM. On the other hand, those kernels might get horrible performance because they are thrashing the L2 cache or failing to coalesce memory loads; because the hardware controls so much of the runtime, it becomes hard to reason about what’s going on behind the scenes. As a result, TPUs can often get closer to peak roofline performance with less work.
How to Think About GPUs | How To Scale Your Model
SMEM is used for storing activations and inputs to TC matmuls.
How to Think About GPUs | How To Scale Your Model
the critical threshhold now depends on and not
All About Rooflines | How To Scale Your Model
we need to keep on learning and we need to pay special attention to how we represent things in our memory to ensure we understand the implications of the things we learn.
Formula for Human Genius and Creativity
Intuition is an inability to explicitly express knowledge that is already wired in the neural network of the brain.
Formula for Human Genius and Creativity
what new powers our brains can acquire with relatively little effort if the new knowledge is selected in the right way
Formula for Human Genius and Creativity
This person is really intelligent! Look at his record! To use a computer metaphor, the scientist is endowed with the best hardware and software money can buy. He or she is optimally equipped for problem solving
Formula for Human Genius and Creativity
This kid is really intelligent! The sky is the limit for him. The kid is equipped with high quality extensible hardware infrastructure
Formula for Human Genius and Creativity
This guy is really intelligent! See how fast his brain reacts! The student has a fast processor installed and his RAM has a lightning access time
Formula for Human Genius and Creativity
one key disadvantage of the Vera Rubin NVL144 CPX is that it has a fixed number and ratio of Rubin and Rubin CPX chips, which makes it less flexible should one wish to change the PD ratio
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
PP will tend to deliver higher token throughput per GPU than Expert Parallelism (EP), but the tradeoff is that PP suffers from a higher time to first token (TTFT) than EP
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
Rubin CPX’s off-chip I/O is limited to 16 lanes of PCIe Gen6 which is approximately 1Tbit/s of unidirectional bandwidth vs 14.4Tbit/s of NVLink for R200. This is enough I/O for performing prefill even for modern MoE frontier LLMs.
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
This complete disaggregation only seems to provide excellent results under certain input/output sequence length ratios and for long decode lengths, with other scenarios seeing underwhelming benefits
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
Since prefill is inherently going to underutilize memory bandwidth resources one approach to reduce wastage is to reduce the quantity and cost of the memory
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
when sequence length exceeds 32k, the FLOPS utilization reaches 100% while the memory bandwidth utilization drops
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
Once specialized inference hardware becomes ubiquitous, using uniform hardware will feel like buying a jackhammer to squash a bug
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
gains are likely from reducing higher precision compute units to have more FP4 ALUs
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
Dense PFLOPs for the Rubin CPX follow the same 3:2 sparse to dense ratio as the R200 because it inherits a similar tensor core architecture as Rubin R200
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
generation of KVCache is much less bandwidth intensive due to the parallel nature of prefill, therefore the HBM is underutilized during this step
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
unning prefill on a chip with lots of expensive HBM featuring very high memory bandwidth is a waste
Another Giant Leap: The Rubin CPX Specialized Accelerator & Rack – SemiAnalysis
However, the way the device is stimulated can provide some information about the type of fault. For example, different test patterns could isolate the fault to a specific input pin
BIST for Analog Weenies | Analog Devices
The process of compressing the response introduces the possibility of enabling a faulty CUT to produce a correct signature, but the probability of a fault going undetected becomes vanishingly small as the pattern length increases.
BIST for Analog Weenies | Analog Devices
Since an SM can execute multiple thread blocks concurrently, typically one wants the number of thread blocks to be several times higher than the number of SMs
GPU Performance Background User's Guide - NVIDIA Docs
term wave to refer to a set of thread blocks that run concurrently
GPU Performance Background User's Guide - NVIDIA Docs
To train to convergence using 15T tokens, Meta would require an amount of energy equal to the annual consumption of an entire neighborhood of 3,400 US households.
H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysis
Amdahl’s Law, which describes the speedup that can be achieved by parallelizing computing steps, can be used to quantify the speedup of strong scaling
H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysis
Extending this calculation further, training 300B tokens on GPT3 175B would require 19 annual US households’ worth of energy consumption for FP8 and 28 households’ worth of annual energy consumption for BF16.
H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysis
The average annual US household in 2022 consumed 10,791kWh of energy or approximately 38,847,600,000 Joules. Dividing this 10,791 kWh by 8,760 hours per year gives us 1,232 W of power on average over the year – a little more than the 1,200W used by a single GB200 GPU!
H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysis
This improvement results from NVIDIA CuDNN/CuBLAS engineers writing more optimized fused wgmma kernels, NCCL engineers writing more optimized collectives that use fewer SMs for communication among other improvements. At the end of the day, it is the full software stack optimization that matters.
H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysis
GB200 chip consumes 1200W per chip vs 700W for the H100
H100 vs GB200 NVL72 Training Benchmarks – Power, TCO, and Reliability Analysis, Software Improvement Over Time – SemiAnalysis