LLM Quantization Formats Compared: GGUF, EXL2/EXL3, and MLX Architectures

What Is LLM Quantization?

LLM quantization reduces the numerical precision of a neural network's parameters to cut memory footprint and accelerate inference. Converting high-precision types like FP16 or BF16 into lower-precision integer representations, 8-bit, 4-bit, or lower, lets massive models run on consumer and edge hardware constrained by limited memory capacity and bandwidth.

Making good infrastructure decisions here means keeping four distinct structural layers straight. Confusing them is the most common error in local deployment:

1. Quantization algorithm: the mathematical method used to compress the model, AWQ, GPTQ, SmoothQuant, or the calibration processes behind K-quants.
2. Quantized representation / model format: the file format that stores compressed weights, scales, and architecture metadata, GGUF, EXL2/EXL3, Safetensors.
3. Inference runtime: the software engine that loads the format, manages memory, and executes the matrix multiplications, llama.cpp, ExLlamaV2/V3, vLLM, MLX-LM.
4. Hardware architecture: the physical environment dictating memory topology and compute, NVIDIA CUDA GPUs, Apple Silicon, x86 CPUs.

When practitioners search "GGUF vs EXL2 vs MLX," they're rarely comparing three interchangeable algorithms. They're comparing deployment ecosystems. GGUF is a model-file container tightly coupled to the llama.cpp runtime. EXL2/EXL3 are quantized formats built specifically for the ExLlama GPU inference engines. MLX is an overarching machine learning framework from Apple, with its own model representations optimized exclusively for Apple Silicon's unified memory. All three ultimately rest on the same underlying discipline of low-precision matrix multiplication.

---

Weight-Only vs Activation Quantization

Weight-only quantization compresses only the model's static parameters. Activation quantization also compresses the dynamic, intermediate states calculated as data flows through the network's layers.

For local inference, weight memory is usually the dominant concern, because model weights have to move from memory into the compute cores for every single token generated during batch-1 autoregressive decoding. Weight-only quantization substantially reduces the baseline RAM/VRAM required and can directly speed up decoding by cutting the bytes moved over the memory bus.

Activations still matter heavily for compute and for the KV cache, though. Quantizing weights does not automatically quantize the KV cache, the precision of weights and the precision of the KV cache are configured independently in most runtimes. Context length stays a major memory constraint even when weights are aggressively compressed to 3-bit or 4-bit.

The approximate baseline relationship for weight-only quantization:

Theoretical Weight Memory ≈ Number of Parameters × Effective Bits per Parameter ÷ 8

That's only the theoretical minimum. Metadata, quantization scales, zero-points, unquantized embeddings, output layers, and allocator overhead push the actual runtime footprint higher than this formula suggests.

---

GGUF: The Flexible Local-LLM Container

GGUF (GPT-Generated Unified Format) is a binary container format for storing models for inference, primarily within the llama.cpp ecosystem and compatible runtimes. It provides an extensible file structure for quantized tensors, architecture definitions, tokenizer data, and runtime metadata, all in a single file.

GGUF became the standard for local CPU/GPU hybrid inference because it embeds architecture metadata directly in the file. The inference engine reads these key-value pairs to construct the computational graph, making the format highly adaptable to new transformer architectures without separate configuration files.

GGUF's most prominent deployment advantage is its established support for CPU/GPU layer offloading. When a model's weights exceed available GPU VRAM, llama.cpp can offload a user-defined number of layers to the GPU while keeping the rest in system RAM. The CPU processes the initial layers, hands the intermediate tensor off over the PCIe bus to the GPU for the offloaded layers, and gets the result back. Traversing PCIe and relying on system RAM introduces latency and bandwidth bottlenecks, but this hybrid approach trades token throughput for memory capacity, letting users run models that fundamentally don't fit on their discrete GPU.

K-Quants and Mixed Precision

K-quants are a family of block-wise quantization schemes inside the llama.cpp ecosystem. Instead of uniformly compressing every weight to a flat bit-width, K-quants group weights into blocks (often 256 weights, further subdivided for scale calculation) and use mixed precision across the model.

Naming conventions like Q4_K_M encode structural information, but they don't mean every parameter is exactly 4 bits:

• Q4: an overarching target of approximately 4 bits per weight.
• K: the block-wise K-quantization scheme.
• M: "Medium," a predefined heuristic mix of precisions across tensor types to balance quality and size.

In a Q4_K_M representation, tensor types historically more sensitive to quantization error may be stored at 6-bit, while less sensitive tensors sit at 4-bit. Block metadata and scales (often 16-bit or 8-bit) are stored alongside. The effective bits per weight ends up higher than the nominal 4.0 label suggests.

Other common variants: Q5_K_M / Q6_K offer conservative compression with a strong quality-to-size trade-off when memory allows; Q2_K / Q3_K are aggressive compression schemes, typically reserved for severe memory constraints and increasingly sensitive to model architecture and calibration data.

Importance Matrices (imatrix) in the Quantization Pipeline

An importance matrix (imatrix) contains sensitivity information derived from running calibration data, a representative text dataset, through the model. It maps how sensitive the model's output is to changes in specific tensors, based on activation statistics.

Without importance information, a quantizer generally minimizes the numerical error of the weights themselves. But a small numerical error in a highly interconnected layer can cascade into significant output degradation. An imatrix-aware process supplies this activation-derived sensitivity data to the GGUF quantization tool, which then prioritizes preserving the weights most critical to output accuracy and shifts quantization error toward less important regions.

An imatrix isn't synonymous with GGUF itself, GGUF is just the container; it can hold naively quantized models or imatrix-guided ones. Imatrix-guided calibration matters most for aggressive compression like Q2_K or Q3_K, where it often measurably reduces perplexity degradation versus standard quantization.

---

EXL2 and EXL3: The ExLlama Ecosystem

EXL2 was a quantized model format built specifically for the ExLlamaV2 inference engine, an ecosystem heavily optimized for NVIDIA GPUs.

While GGUF K-quants use predefined, heuristic-based mixed precision, EXL2 used a data-driven, variable-bitrate approach instead. During quantization, calibration data was evaluated and the algorithm solved a Hessian-based error estimation to measure the sensitivity of individual layers. Based on that layer-wise measurement, EXL2 allocated varying bit-widths (2-bit through 8-bit) to different layers to hit a user-defined target average size. A model quantized to 4.0 bpw (bits per weight) in EXL2 averages 4.0 bits per parameter, but the physical file is a patchwork of different precisions dictated by the calibration data.

Q4_K_M vs EXL 4.0 bpw

EXL 4.0 bpw is not structurally equivalent to GGUF Q4_K_M, whether the EXL side is EXL2 or EXL3. They're entirely different quantization schemas, calibration workflows, and internal data structures. Both target roughly 4 bits per parameter, but the dynamic layer-wise allocation and the ExLlama kernel implementations produce different runtime characteristics, memory layouts, and quality profiles.

EXL3: Trellis-Based Quantization

ExLlamaV3's EXL3 format is a genuine architectural shift from EXL2, not just a version bump. It's built on trellis-coded vector quantization, using procedural codebooks and incoherence processing (ideas drawn from the QTIP quantization method) to preserve more information at low bitrates than EXL2's per-layer bit allocation could. EXL3 also retains more of the original Hugging Face tensor structure, which makes integration with other tooling easier.

The practical result: EXL3 holds up meaningfully better under aggressive compression. Community testing has shown Llama-3.1-70B quantized to EXL3 remaining coherent at around 1.6 bpw with a 3-bit output layer, fitting inference under 16GB of VRAM for a model that needs roughly 140GB at full precision. That kind of extreme compression was never realistic with EXL2's allocation scheme.

GPU-Resident Inference and Effective Precision

Both EXL2 and EXL3 are highly competitive for GPU-resident NVIDIA inference when the ExLlama execution path matches the workload. The runtime uses CUDA kernels designed to maximize GPU memory bandwidth and minimize overhead.

EXL terminology uses granular decimals (3.0 bpw, 4.25 bpw, 5.0 bpw, 6.0 bpw), which lets infrastructure engineers target their hardware limits precisely. If a 32B model at 4.0 bpw leaves 4GB of VRAM unused on a 24GB GPU, an engineer can select a 4.5 bpw version instead, fully using the VRAM to extract the highest mathematical precision the hardware allows.

Note on CPU/system-RAM capability: the ExLlama family is primarily designed around GPU execution. Rudimentary CPU fallback exists in some backend implementations, but relying on system RAM generally defeats the architectural point of ExLlamaV2/V3. Its advantage shows up when the model and KV cache reside entirely in GPU memory. For heavily memory-constrained deployments that need significant offloading, GGUF remains the more established and flexible hybrid path.

---

MLX: The Apple Silicon Ecosystem

MLX is a machine learning array framework from Apple's machine learning research group, designed specifically for the hardware/software environment of Apple Silicon (M-series SoCs).

A common misconception treats MLX as "Apple's file format for local LLMs." MLX is a comprehensive, PyTorch-like framework for array operations, model training, and inference. Its ecosystem includes tools to convert and quantize models into MLX-specific representations (often saved as safetensors with MLX configuration files) that run natively on Apple's Metal API.

Unified Memory Architecture in Practice

"MLX vs GGUF on Mac" is a distinct comparison from the NVIDIA side precisely because of unified memory. In a traditional discrete GPU architecture, an x86 CPU paired with an NVIDIA PCIe GPU, system RAM and GPU VRAM are physically separate domains, and moving data between them relies on the PCIe bus, a severe bottleneck compared to internal GPU memory bandwidth.

On Apple Silicon, the CPU and GPU share a unified memory system with no discrete CPU-RAM-to-GPU-VRAM copy boundary. If an M-series Mac has 128GB of unified memory, both the CPU and GPU can address that memory pool directly. This lets a Mac load a 100B+ parameter model entirely into memory the GPU cores can reach, a capacity that would otherwise require multiple discrete GPUs.

Unified memory makes large models capacity-feasible, though, not necessarily fast. The memory still has finite bandwidth, and the CPU and GPU contend for it. Token generation speed stays bounded by the SoC's memory bandwidth: current-generation Apple Silicon spans roughly 120 GB/s on base M-series chips up to 460-614 GB/s on M5 Max (depending on GPU core count) and around 1.2 TB/s on M5 Ultra. Running a model that uses 95% of a Mac's unified memory can also starve the OS of working memory, causing swapping and severe performance degradation.

---

Ecosystem Comparison

---

Quantization Levels and Reasoning Quality

Quantization alters model behavior and introduces mathematical error, but there's no universal bit threshold at which reasoning "breaks." Degradation depends on model architecture, parameter count, training quality, calibration dataset, and workload. Larger models (70B+) are generally more resilient to aggressive quantization than smaller ones (7B-8B).

Benchmark evidence varies by implementation, but practitioners generally observe these heuristics:

• 8-bit and above: usually closely mirrors unquantized (FP16/BF16) baseline behavior. Often preferred when memory capacity and bandwidth are abundant, though the delta from 6-bit is sometimes hard to measure outside strict academic benchmarks.
• 5-6 bit: often a strong quality/size compromise. For many models, perplexity degradation is minimal, and instruction following, coding, and mathematical reasoning stay robust.
• ~4 bit: extremely common, since it provides substantial memory savings while letting large models fit on consumer hardware. Standard summarization and creative tasks remain highly coherent, but minor degradation in zero-shot coding accuracy and multi-step logic can show up depending on the model and task.
• ~2-3 bit: more aggressive and increasingly sensitive to the quantizer, calibration data, and task. Measurable perplexity degradation, increased hallucination rates, and reduced nuance are common here, though an aggressively quantized 70B model often still outperforms a higher-precision 7B model.

Quality has to be measured against the specific task. A model quantized to 4-bit might score identically to its FP16 counterpart on general factual recall, but drift slightly on a highly specific long-context tool-use task.

---

How Much Memory Does a Quantized LLM Need?

Estimating memory requires separating theoretical weight memory from actual file size, and actual file size from total runtime footprint.

Baseline formula for weights alone: Theoretical Size in GB ≈ (Parameter Count in Billions × Effective Bits per Weight) / 8

The actual downloaded file size is rarely identical to the theoretical number. Quantization formats include block scales, metadata, tokenizer configuration, and higher-precision tensors (like unquantized embeddings and the output language modeling head). Depending on format and architecture, this adds anywhere from a few hundred megabytes to several gigabytes of overhead.

The runtime memory requirement is larger still. The inference runtime needs temporary memory buffers, context allocation, allocator overhead, and, most significantly, the KV cache.

---

KV Cache, Architecture, and Context Length

One of the most destructive misconceptions in local AI infrastructure is assuming that if the model file size is smaller than available RAM, the model will run successfully.

During autoregressive decoding, transformers maintain a record of past tokens to avoid recalculating the entire sequence, this is the KV cache. It resides in RAM/VRAM and grows linearly with context length and batch size.

Calculating KV cache size accurately means accounting for the model's specific attention architecture. Modern LLMs frequently use Grouped-Query Attention (GQA) or Multi-Query Attention (MQA), which use fewer KV heads than standard Multi-Head Attention, dramatically reducing context memory.

The general relationship for KV-cache memory per token (batch size 1):

Bytes per token = 2 × Layers × KV_Heads × Head_Dimension × Bytes_per_Element

Example architecture (Llama 3 70B): 80 layers, 8 KV heads (GQA), head dimension 128. Using FP16 (2 bytes per element):

If a 70B model's weights consume roughly 39GB of VRAM and you try to run a 128K context prompt on a 48GB setup, the total requirement (~81GB) results in an out-of-memory error, even though the weights alone fit easily.

Note on KV-cache precision: weight quantization and KV-cache quantization are independent decisions. Even with 4-bit weights, the KV cache defaults to FP16 in most runtimes. Newer implementations support 8-bit or 4-bit KV caches (Q8_0 or Q4_0 in llama.cpp, or an 8-bit cache in ExLlama), trading a potential reduction in long-context accuracy for substantial VRAM savings.

---

Inference Speed: Prefill vs Decode

"Tokens per second" isn't a single universal metric. Inference splits into two phases with different hardware dependencies:

1. Prompt processing (prefill / time to first token): the runtime processes the input prompt in parallel. This phase leans on matrix-matrix multiplications and is largely compute-bound. High GPU core counts and kernel efficiency accelerate prefill significantly.
2. Token generation (decode): the runtime generates output sequentially, one token at a time. In batch-size-1 autoregressive decode, the entire model weight has to move from memory to the compute cores for each generated token. Single-user generation speed is therefore often strongly memory-bandwidth bound.

Memory capacity dictates how large a model you can load. Memory bandwidth is a primary governor of how fast that model generates tokens. Bandwidth isn't the only factor, though, kernel efficiency, attention implementation, and CPU execution bottlenecks also affect final decode speeds.

---

Multi-GPU Architecture Considerations

When deploying on multiple GPUs, "I have two 24GB GPUs" doesn't mean the system behaves identically to "one monolithic 48GB GPU." Running a model across multiple GPUs requires the runtime to split the computation.

• Tensor parallelism: splits individual layers across GPUs, requiring constant, high-bandwidth communication, often suited to enterprise interconnects like NVLink.
• Pipeline parallelism / layer splitting: places the first portion of the model's layers on GPU 1, the remainder on GPU 2.

Without a high-speed interconnect, data has to travel over the host system's PCIe bus, adding synchronization latency. The KV cache is typically distributed across the GPUs too, and some runtimes need extra memory for duplicate activations or allocator overhead. So 24 + 24 GB yields usable capacity slightly lower than a monolithic 48GB block. Performance depends heavily on the runtime implementation, PCIe topology (x16 vs x8 lanes), and GPU balance.

---

Practical Worked Examples

Example 1: Fitting a 32B Model on a 24GB GPU

• Model: a generic 32B dense model using GQA.
• Hardware: 1× NVIDIA RTX 3090/4090 (24GB VRAM).
• Weight memory (4-bit target): theoretical baseline ~16GB. Actual file size with metadata, tokenizer, and unquantized embeddings runs approximately 16.5 to 18GB depending on the exact quantizer.
• Remaining VRAM: ~6 to 7.5GB.
• Context analysis: an 8K context at FP16 might consume roughly 1.5GB. Runtime buffers and OS overhead take another ~1.5GB. Total footprint: ~20.5GB, fits comfortably.
• What if 32K context? That might need ~6GB, pushing the total to ~25GB, which OOMs on a 24GB card. The engineer has to either quantize the KV cache to 8-bit (roughly halving cache memory) or drop weight precision to a ~3.5 bpw format to free up VRAM.
• Format choice: since the model fits entirely in VRAM for standard context lengths, EXL3 (the actively maintained ExLlama format) is an excellent fit for maximizing CUDA inference speed. If the engineer needs massive context and has to offload some layers to system RAM, GGUF is the appropriate fallback.

Example 2: Running a Large Model on a 128GB Apple Silicon Mac

• Model: ~104B parameters.
• Hardware: Mac Studio, M5 Max, 128GB unified memory, roughly 460-614 GB/s bandwidth depending on GPU core count.
• Weight memory (4-bit target): ~52GB theoretical, ~56GB actual.
• Remaining memory: ~72GB.
• Context analysis: the capacity is massive. The user can allocate 15-20GB for an extensive context window and still leave ample memory for macOS.
• Speed reality: unified memory makes loading this model capacity-feasible without enterprise GPUs, but decode speed is heavily influenced by memory bandwidth. Moving ~56GB of weights for every token generated yields a theoretical bandwidth ceiling in roughly the 8-11 tokens/sec range at the low end of that bandwidth window. This is strictly a mathematical upper bound, real-world speed will be lower due to CPU/GPU contention, kernel execution overhead, and background system processes. See our M5 Ultra vs M5 Max comparison for the fuller capacity-vs-bandwidth tradeoff on current Apple Silicon.
• Format choice: MLX provides deep native integration and Python-based ecosystem benefits for Apple developers. GGUF (via a Metal-compiled runtime like llama.cpp) offers a highly accessible alternative with ubiquitous pre-quantized model availability.

---

Choosing an Inference Stack for Your Hardware

The goal isn't finding a universal winner, it's matching the deployment format to the hardware constraints.

EXL3 (ExLlamaV3) is often the best fit when:

• You're running NVIDIA CUDA GPUs.
• The entire model and required KV cache fit appropriately into available VRAM.
• Your priority is maximizing batch-1 throughput via highly optimized CUDA kernels.
• You want fine-grained bpw control to extract maximum precision from a specific VRAM limit, including very aggressive sub-2-bit compression if needed.

GGUF is often the best fit when:

• Your desired model exceeds GPU VRAM, requiring hybrid CPU/GPU layer offloading.
• You're running inference entirely on CPU.
• You need a single-file format with broad, immediate availability across community repositories and third-party UIs (LM Studio, Ollama).
• You want highly tested K-quant and imatrix workflows for aggressive compression scenarios.

MLX is often the best fit when:

• You're deploying on Apple Silicon (M-series SoCs).
• You want to use Apple's Metal API directly via an Apple-native array framework.
• You're building AI applications within the Apple ecosystem and want PyTorch-like semantics optimized for unified memory.

---

Common LLM Quantization Myths

• "4-bit means every parameter is exactly 4 bits." False. Most 4-bit formats use mixed-precision blocks (GGUF Q4_K_M) or data-driven variable bitrates (EXL2/EXL3), alongside higher-precision scales, metadata, and unquantized embeddings.
• "EXL2 is simply a faster quantization algorithm than GGUF." False. They operate at different layers. EXL2/EXL3 are model formats for a specific runtime family (ExLlama). GGUF is a flexible container standard.
• "MLX is just Apple's version of a GGUF file." False. MLX is a comprehensive machine learning framework that includes model conversion, training, and representation tooling natively optimized for Apple Silicon.
• "If the model fits in RAM, it will run fast." False. Capacity dictates whether it loads; memory bandwidth dictates how fast it generates tokens during autoregressive decode. System RAM has significantly lower bandwidth than GPU VRAM.
• "Quantizing weights also quantizes the KV cache." False. They're configured separately. Weight quantization reduces the model footprint, but context memory needs its own precision configuration.

---

Conclusion

There's no universally "best" quantization format. The right inference stack is determined by the intersection of model architecture, effective bits per weight, available memory capacity, memory bandwidth, runtime support, context requirements, and hardware topology.

Before downloading a model, run through this checklist:

• Hardware topology: a discrete NVIDIA GPU, a CPU-only server, or an Apple Silicon Mac?
• Memory capacity: do the quantized weights, allocator overhead, and target KV cache fit within your highest-bandwidth memory tier, VRAM or unified memory?
• Bandwidth boundaries: are you relying heavily on PCIe transfer or system RAM, and is your application tolerant of the resulting drop in decode tokens/sec?
• Runtime suitability: does your chosen runtime, llama.cpp, ExLlamaV3, MLX, support your required hardware and context scaling features, and is it still actively maintained?

Aligning the model representation with the physical hardware constraints is how engineers deploy local LLMs that balance a practical memory footprint against robust, defensible reasoning capability.

---

Frequently Asked Questions

What is the difference between GGUF and EXL2?

GGUF is a flexible model container format heavily associated with llama.cpp, excelling at hardware compatibility and CPU/GPU layer offloading. EXL2 was a variable-precision model format for the ExLlamaV2 runtime, heavily optimized for GPU-resident execution on NVIDIA hardware. ExLlamaV2 is now a legacy, archived project; its successor ExLlamaV3 uses a new format called EXL3.

Is ExLlamaV2 (EXL2) still maintained?

No. As of 2026, the ExLlamaV2 repository is archived and read-only. Development continues on ExLlamaV3, and TabbyAPI's main branch dropped ExLlamaV2 support, keeping it only on a legacy branch. Existing EXL2 quants remain installable and functional, but the format will not receive further updates.

Is GGUF better than EXL2 or EXL3?

This depends on hardware topology and workload. GGUF provides superior flexibility for CPU offloading, Apple Silicon, and mixed-hardware environments. EXL3 is often preferred by engineers running purely GPU-resident inference on NVIDIA hardware who want granular bits-per-weight control and strong quality at low bitrates.

Is MLX better than GGUF on Mac?

Both perform well on Apple Silicon. MLX provides an Apple-native array framework ideal for developers building directly within the ecosystem. GGUF, via Metal-accelerated runtimes like llama.cpp, offers massive out-of-the-box compatibility with existing user interfaces and pre-quantized model repositories.

What does Q4_K_M mean?

It's a GGUF naming convention targeting roughly 4 bits per weight. "K" indicates it uses block-wise K-quants, and "M" (Medium) indicates a predefined heuristic mix of precisions across different tensor types to balance quality and size.

What does 4.0 bpw mean?

It stands for bits per weight. In EXL2 or EXL3, it means the quantization algorithm calibrated the model and assigned varying precisions, such as 3-bit, 4-bit, or 6-bit, to different layers, achieving a mathematical average of 4.0 bits per parameter across the model.

Which quantization should I use for a 24GB GPU?

This depends on model size and context requirement. For an 8B model, 6-bit or 8-bit precision fits easily with large context. For a 32B model, a roughly 4.0 to 4.5 bpw format fits comfortably with standard context. A 70B model cannot fit entirely in VRAM at any usable precision on 24GB, requiring GGUF layer offloading to system RAM or multi-GPU expansion.

How much VRAM does a 70B model need?

A conventional dense 70B model generally cannot fit comfortably within 24GB of VRAM at commonly used quality-preserving quantization levels. At 4-bit, a 70B model's actual weight footprint is typically 38-40GB. Depending on architecture, context length, and KV-cache precision, total runtime memory typically falls between 42GB and 50+GB, making 48GB the practical baseline for GPU-resident execution.

Does context length affect memory usage?

Yes, significantly. Every token processed or generated requires memory in the KV cache. Depending on the model's attention architecture (MHA vs GQA) and precision settings, expanding context from 8K to 128K can increase memory consumption by several gigabytes to tens of gigabytes.

Does quantizing the weights also quantize the KV cache?

No. Weight precision and KV-cache precision are configured independently in most runtimes. A model with 4-bit weights defaults to an FP16 KV cache unless it's explicitly configured otherwise, so context memory still needs its own precision setting to shrink.