Arithmetic Intensity and Roofline Modeling

Fetch Operands: The GPU reads the weights from HBM (counts toward Bytes).

Execute Math: The Tensor Cores multiply the inputs (counts toward FLOPS).

Write Output: The GPU writes the activations back to HBM (counts toward Bytes).

Evaluation: Profilers divide the FLOP count by the Byte count to place the kernel on the Roofline chart.

Because the compute capabilities of GPUs (20 PFLOPS) are scaling vastly faster than HBM bandwidth (8 TB/s), the ridge point is moving further to the right. This means it is becoming mathematically harder to keep the Tensor Cores saturated.

FlashAttention was invented purely as a Roofline Optimization. Standard attention reads/writes the attention matrix to HBM repeatedly. FlashAttention computes it dynamically in SRAM, drastically reducing "Bytes" and artificially inflating the Arithmetic Intensity past the ridge point, turning a memory-bound problem into a compute-bound problem.

Operator Fusion: Combine multiple element-wise operations (e.g., Matmul + Bias + ReLU) into a single kernel to prevent intermediate HBM writes.

Quantization: Move from FP16 (2 bytes) to FP4 (0.5 bytes) to shrink the denominator of the Arithmetic Intensity equation.

Nsight Compute: Automatically generates a Roofline Analysis chart, plotting your exact kernel execution against the physical GPU boundaries.

System design interviews for AI infrastructure rely heavily on Roofline math. "Calculate the time to generate 1 token on a 70B parameter model." You must know that decoding is memory bound, so you divide 70GB (if INT8) by the HBM bandwidth (e.g., 2 TB/s) to find the absolute minimum latency (35ms).