Memory-Bound vs Compute-Bound Diagnostics

To isolate the operational boundary, an engineer first monitors macro-metrics via nvidia-smi dmon or DCGM exporters. High compute utilization coupled with near-maximum power draw reliably indicates heavy compute saturation. If sm__efficiency is low but memory bandwidth is pegged at peak physical limits, the diagnosis is confirmed as a memory bandwidth bottleneck. Profiling via Nsight Compute subsequently confirms the exact warp stall reasons, definitively proving whether threads are paused on LDG (load global) instructions or stalled by complex mathematical pipeline dependencies.

Failing to diagnose the boundary accurately leads directly to catastrophic optimization failures. For example, explicitly vectorizing a loop to increase arithmetic throughput will yield absolutely zero latency reduction if the workload is currently stalled on global memory reads. Conversely, spending weeks writing complex shared-memory caching algorithms is wasted engineering effort if the underlying matrix math already perfectly saturates the available Tensor Cores.

In LLM inference serving, the prefill phase (processing the input prompt) involves large matrix multiplications and is heavily compute-bound. Conversely, the decoding phase (generating output tokens one-by-one) relies on reading the massive KV cache for every single token generation, rendering it severely memory-bandwidth bound. Top AI labs deploy hybrid infrastructure routing strategies, dynamically shifting inference traffic to specific nodes based on these highly distinct bounded profiles to maximize cluster throughput.

Optimization must be strictly tailored to the diagnosis. Memory-Bound Fixes involve ensuring coalesced memory accesses, utilizing shared memory block tiling, and quantizing weights and activations to lower precision formats to reduce traffic. Compute-Bound Fixes involve enabling Tensor Float 32 (TF32), leveraging sparsity, and optimizing algorithmic mathematical complexity. Latency-Bound Fixes involve drastically increasing batch sizes to spawn more warps, allowing the hardware scheduler to seamlessly hide latency via extreme thread concurrency.

Nsight Compute, nvidia-smi dmon, DCGM Exporter, DeepSpeed, and FlashAttention.

A classic, filtering interview question asks: "Why is Batch Size 1 inference on an LLM so incredibly inefficient?" The strong candidate will immediately pivot the conversation to memory-bound constraints, explaining that loading the entire massive model weight matrix from HBM to SM registers for a single token generation yields a devastatingly low arithmetic intensity, permanently stranding the vast compute resources of the GPU.