GPU Architecture and the Execution Model

The Problem

Production LLM serving is full of claims that sound like hardware trivia until they suddenly determine your latency bill: "prefill is compute-bound, decode is memory-bound," "KV cache pressure," "low occupancy on small batches," "tensor-core throughput." Without a first-principles model of what the GPU is doing, those phrases remain slogans.

The gap shows up in three places.

You cannot debug a kernel you cannot model. A Triton flash-attention kernel fails a benchmark. Is the issue too few warps per SM, too much shared memory per block, or arithmetic intensity below the ridge point? Each diagnosis points to a different fix, and guessing wastes days.

You cannot size a serving deployment without roofline arithmetic. A 70B model at FP16 needs ~140 GB of weights. At decode, every new token reads all of those weights once. The memory wall, not the FLOP count, sets your tokens-per-second per GPU until batch size grows large enough to amortize the weight read.

You cannot read vLLM or SGLang source without this vocabulary. PagedAttention, RadixAttention, chunked prefill, and speculative decoding are all engineering responses to the same underlying facts: warps execute in SIMT fashion, memory hierarchy matters, and occupancy plus arithmetic intensity determine whether you are compute- or memory-bound.

This lesson builds the model from the hardware up and ships a simulator that turns a kernel configuration into occupancy, arithmetic intensity, roofline position, and estimated runtime.

The Concept

CPU throughput vs GPU throughput

A modern CPU is optimized for latency. It has large caches, branch prediction, out-of-order execution, and a comparatively small number of powerful cores. It wins when a single thread must complete a task quickly and when the working set fits in cache.

A GPU is optimized for throughput. It has hundreds of times more arithmetic units, but they are simpler, in-order, and share control logic. A GPU hides latency by keeping thousands of threads in flight at once: while one warp waits for memory, another warp with ready operands is scheduled. Big caches would waste die area that is better spent on more arithmetic. The design bet is that parallelism, not cache, will cover latency.

The same instruction runs across many threads in a SIMD/vector style, but CUDA calls the model Single Instruction, Multiple Threads (SIMT): each thread has its own registers and can take independent control paths, but the hardware executes one instruction for the whole warp at a time.

Streaming Multiprocessors and the H100 example

The GPU is organized into Streaming Multiprocessors (SMs). Each SM contains arithmetic units, schedulers, registers, and shared memory. An NVIDIA H100 SXM5 GPU contains roughly 132 SMs (values vary slightly by SKU). To make the numbers concrete for this lesson, we use the following approximate H100 specifications:

These are rounded, SKU-dependent numbers; treat them as educational anchors, not datasheets.

A warp scheduler on an SM can issue an instruction from a different warp every cycle. With four schedulers and 32 threads per warp, a single SM can keep over 2,000 threads resident. Across 132 SMs the chip can hold tens of thousands of warps. The scheduler picks a warp whose operands are ready, so the more independent warps resident, the more latency the hardware can hide.

CUDA execution hierarchy: thread → warp → block → grid

A CUDA kernel launches as a grid of blocks, and each block contains threads.

Copythread ⊂ warp ⊂ block ⊂ grid

• A thread is the smallest unit. It has private registers and a thread ID.
• A warp is 32 threads. The SM schedules whole warps, not individual threads.
• A block is a group of threads that run on the same SM and can cooperate through shared memory and block-level synchronization. Threads in a block are divided into warps.
• A grid is the collection of all blocks launched by the kernel. Blocks are distributed across SMs by the hardware.

Thread IDs matter for indexing data: each thread computes a portion of the output. When threads access consecutive memory addresses, the hardware coalesces those loads into fewer cache lines, which is the single most important optimization for global-memory bandwidth.

SIMT execution and warp divergence

In SIMT, all 32 threads in a warp execute the same instruction on different data. If the threads take different branches of an if statement, the hardware cannot execute both branches simultaneously. It runs one branch, masking off the threads that did not take it, then runs the other branch, masking off the threads that did. This is warp divergence, and it serializes the warp:

In the worst case a divergent if/else doubles the time for that warp. Conditional logic on thread ID should be avoided in inner loops. This is why kernels like flash attention work hard to keep all threads in a warp on the same control path.

Occupancy

Occupancy is the ratio of active warps on an SM to the maximum warps the SM can hold:

Higher occupancy generally improves latency hiding because the scheduler has more warps to choose from. However, occupancy is not the only metric that matters — a kernel with high occupancy but terrible memory access patterns can still be slow. The hardware limits occupancy through:

• Threads per block. A block with 1,024 threads occupies 32 warps. If an SM supports 2,048 threads, at most two such blocks can reside on it.
• Registers per thread. Each SM has a fixed register file. A block that uses many registers per thread leaves fewer warps resident. The compiler assigns registers; the simulator models this with a per-thread register budget.
• Shared memory per block. Shared memory is fast but scarce. A block that claims most of an SM's shared memory prevents additional blocks from launching on that SM.
• Max blocks per SM. Even if thread and register limits allow more, the hardware caps the number of concurrent blocks per SM.

The limiting resource is whichever produces the smallest number of resident blocks.

Memory hierarchy and latency

GPU memory is a hierarchy of capacity, bandwidth, and latency:

The farther a load travels, the more cycles it costs. A memory-bound kernel is one whose loads cannot be served by registers or shared memory and must repeatedly go to HBM. Tiling is the standard response: load a small tile of data into shared memory once, then reuse it for many arithmetic operations before writing results back.

For matrix multiplication, a naive implementation reads each input element once per output element it contributes to. A tiled implementation loads a tile into shared memory and reuses each element for an entire row or column of the output tile. The traffic reduction is proportional to the tile size.

Roofline model

The roofline model plots achievable performance as the minimum of peak compute and the memory-bound ceiling:

where P is peak compute, B is memory bandwidth, and I is arithmetic intensity in FLOPs per byte:

The ridge point is where the two roofs meet:

For the H100 numbers above:

Any kernel with I < 295 is memory-bound on HBM bandwidth; any kernel with I > 295 is compute-bound. This is why:

• A naive FP32 matmul has arithmetic intensity 1/4 = 0.25 FLOPs/Byte — deeply memory-bound.
• A tiled FP32 matmul with tile size T has intensity roughly T/4 — better, but still memory-bound for modest tiles.
• Decoder generation for a 70B model at FP16 reads every parameter once for about 2 FLOPs per parameter, giving I = 1 FLOP/Byte — strongly memory-bound until batching raises reuse.

Batching helps because the same weight is read once and used for every token in the batch. With batch size B, arithmetic intensity for decode scales roughly with B, eventually crossing the ridge point and shifting the workload from memory-bound to compute-bound.

Build It

The lesson ships a pure-NumPy "GPU execution financial model." You provide a kernel configuration — grid and block shape, FLOPs per thread, global bytes per thread, shared memory per block, and register budget — and the simulator prints occupancy, arithmetic intensity, roofline bound, and estimated runtime.

It also compares a naive matmul against a shared-memory-tiled matmul and prints a text roofline table for the H100.

Run it:

consoleCopypython3 main.py

You will see the H100 ridge point (~295 FLOPs/Byte), the dramatic traffic reduction from tiling, the occupancy hit from a 120 KB shared-memory allocation, and the arithmetic intensity of decoder generation for a 70B model. Every constant in the output maps to a named constant in main.py and a concept in this doc.

Verify It

code/tests/test_main.py locks down the hardware and simulator invariants:

• the H100 ridge point is approximately 295 FLOPs/Byte;
• a CUDA warp contains 32 threads;
• naive matmul is memory-bound with arithmetic intensity below 1 FLOP/Byte;
• tiling reduces global memory traffic by the tile size factor;
• tiling raises arithmetic intensity relative to the naive version;
• excessive shared memory reduces occupancy;
• high register usage reduces occupancy;
• decoder-generation arithmetic intensity is far below the ridge point.

Why This Matters for Engineers

• The roofline model turns arguments into arithmetic. Before you blame the framework, compute arithmetic intensity. If you are below the ridge point, quantization, tiling, or batching will help; more FLOPs will not.
• Decoder is memory-bound by design. Autoregressive generation reads the full weight matrix for each token. That is why batching, speculative decoding, and quantization all target memory bandwidth, not raw compute.
• Occupancy is a latency-hiding budget, not a speed dial. A kernel with 100% occupancy can still be slow if its memory access pattern is scattered. A kernel with 50% occupancy can be fine if every warp is compute-heavy and well-coalesced.
• Tiling is the first optimization for memory-bound kernels. The simulator shows a 32x reduction in HBM traffic for a 32x32 tile. Real cuBLAS and CUTLASS add many more tricks, but tiling is the foundation.
• Batch size is a roofline knob. Larger batches amortize weight reads and push decode from memory-bound toward compute-bound. That is the economic reason serving engines batch requests, even at the cost of latency tail.
• vLLM, SGLang, and Triton all speak this language. PagedAttention reduces KV-cache HBM traffic; RadixAttention reuses prompt prefixes; chunked prefill balances compute and memory; Triton kernels tune block size and tile size for occupancy. The same model explains all of them.

Hints

• If occupancy is unexpectedly low, check the limiting resource in this order: block size, shared memory per block, registers per thread, max blocks per SM.
• Memory-bound kernels benefit from coalesced access, tiling into shared memory, and increasing arithmetic intensity through batching — not from more SMs.
• A divergent branch inside a hot loop can cost more than a few extra FLOPs; prefer predication or data layout changes when possible.
• Decoder generation is strongly memory-bound for small batches; expect throughput to scale nearly linearly with batch size until the ridge point is crossed.

Key Terms