Running Local LLMs: VRAM Requirements, Quantization (GGUF/EXL2), and the Definitive Hardware Guide

[!KEY TAKEAWAYS]

• VRAM is King, Not Compute: Local LLM inference speed is almost strictly memory-bandwidth bound. If a model fits entirely in VRAM, generation is blazing fast; if it spills into system RAM, token generation drops by 80–95%.
• The Sweet Spot Quant: Q4_K_M (4.5 bits/weight) remains the undisputed gold standard, slashing memory footprint by ~70% compared to 16-bit float with negligible real-world perplexity degradation.
• The Hidden KV Cache Trap: Running an 8B model at 8k context requires ~1GB for KV cache; running that same model at 64k or 128k context can consume 8GB–16GB in attention buffers alone, triggering out-of-memory crashes.
• The Best Bang-for-Buck GPUs: A used NVIDIA RTX 3090 24GB remains the undisputed champion for hobbyists and developers; Apple Silicon Mac Studio (64GB–192GB unified memory) dominates massive 70B+ models without enterprise server rack costs.
• Interactive Calculator: Use our interactive VRAM calculator below to dial in your exact parameter size, quantization format, and context window requirements.

Running open-weights language models entirely on your own local silicon has shifted from an eccentric hobbyist experiment into a crucial pillar of modern privacy and developer workflows. Whether you want an uncensored coding assistant, a zero-leakage enterprise document summarizer, or simply want to escape OpenAI's rate limits and usage subscriptions, running models like Llama 3.1, Qwen 2.5, or DeepSeek locally is now practical on consumer hardware.

However, the hardware realities of LLM inference remain deeply misunderstood. Many developers buy top-tier graphics cards with blistering compute specs—such as the RTX 4070 Ti Super or RTX 4080—only to discover they cannot run the models they want because they hit a brick wall: Video RAM (VRAM) capacity.

In this guide, we break down the exact mathematics of local LLM memory requirements, demystify quantization formats, explain the memory dynamics of the attention KV cache, and provide an interactive calculator to size your rig before buying hardware.

---

The Core Law of Local Inference: Memory Bandwidth vs Compute

When you train an AI model, compute (FLOPS and Tensor Cores) is the primary bottleneck. Backpropagation requires trillions of floating-point matrix multiplications.

During autoregressive token generation (inference), however, the workload changes dramatically:

1. To predict a single new token, the GPU must stream every single weight parameter of the entire neural network from VRAM across the memory bus and into the GPU compute cores.
2. If you are running an 8-billion parameter model in 4-bit precision, that means ~4.5 to 5.5 Gigabytes of data must be shuttled across the memory bus for every single word generated.
3. Therefore, your generation speed in tokens per second (tok/s) is fundamentally governed by this formula:

$$\text{Tokens per Second} \approx \frac{\text{Memory Bandwidth (GB/s)}}{\text{Model File Size in VRAM (GB)}}$$

For example, an RTX 3090 possesses a 384-bit memory bus delivering 936 GB/s of memory bandwidth. When hosting a 5 GB quantized 8B model, the GPU can theoretically shuttle the weights upwards of 100+ times per second, yielding blistering real-world outputs of 90–120 tokens/second.

Conversely, if the model is just 1 GB too large to fit in VRAM, llama.cpp or Ollama must offload the remaining layers to system DDR4/DDR5 RAM via the PCIe bus. Standard dual-channel DDR5 delivers only 50–80 GB/s—a tenfold reduction in throughput. The moment your model spills over VRAM into system memory, your generation speed plummets from 80 tokens/second down to a crawl of 3–5 tokens/second.

---

Interactive Local LLM VRAM Calculator

Use the interactive tool below to adjust parameter sizes, choose quantization tiers, and test context buffer sizes to see exact VRAM consumption and recommended hardware configurations:

[[widget:vram-calculator]]

---

Understanding Quantization: Q4_K_M vs Q5_K_M vs Q8 vs FP16

Original models released by Meta, Mistral AI, or Alibaba are trained and published in Bfloat16 or FP16 precision (16 bits, or 2 bytes, per parameter). At FP16:

• A 7B parameter model requires $7 \times 2 = 14\text{ GB}$ of raw memory.
• An 8B parameter model requires $8 \times 2 = 16\text{ GB}$ of raw memory.
• A 70B parameter model requires $70 \times 2 = 140\text{ GB}$ of raw memory.

Running 70B unquantized requires two to four $10,000+ data center GPUs (like the NVIDIA A100 or H100). This is where quantization enters.

Quantization compresses the floating-point weights into lower-bit representations (integers like INT4 or INT8). The modern standard format for CPU/GPU unified inference is GGUF (developed for llama.cpp and utilized under the hood by Ollama).

Here is how the common quantization levels compare:

Quantization Type	Average Bits Per Weight	File Size Relative to FP16	Perplexity Impact (Quality Loss)	Recommended Use Case
FP16 / BF16	16.0	100% (Baseline)	0.00 (Zero loss)	Fine-tuning, synthetic dataset generation, maximum academic fidelity
Q8_0	8.5	~53%	Negligible (< 0.01%)	When you have excess VRAM and want mathematically near-perfect output
Q6_K	6.56	~41%	Minimal (< 0.05%)	High-fidelity coding and mathematical logic tasks
Q5_K_M	5.5	~34%	Barely perceptible (< 0.1%)	The enthusiast sweet spot; slightly better reasoning than Q4
Q4_K_M	4.5	~28%	Very low (< 0.2%)	The universal standard. Best speed, memory, and accuracy balance
Q3_K_M	3.5	~22%	Noticeable degradation	Only when shoehorning a larger model onto constrained VRAM
Q2_K	2.5	~16%	Severe degradation	Avoid; coherent syntax often breaks down

What Does the _K_M Suffix Mean?

Older quantization methods (like legacy Q4_0) converted all weight tensors uniformly to 4 bits. Modern "K-quants" (k-means) use hybrid block quantization:

• Critical attention and gate tensors that disproportionately affect reasoning are kept at higher precision (e.g. 5 or 6 bits).
• Less critical feed-forward matrix tensors are quantized to 4 bits.
• The M denotes "Medium" (balanced performance), whereas S is "Small" and L is "Large". In 99% of cases, Q4_K_M or Q5_K_M is the exact variant you should download.

---

The Silent Killer: The Attention KV Cache

A common mistake is calculating VRAM solely by multiplying parameter count by bit precision:

$$\text{Estimated VRAM} = \frac{\text{Parameters} \times \text{Bits}}{8}$$

If you use this naive formula, your application will crash with an out-of-memory (CUDA OOM) error the moment you feed it a large prompt.

Why the KV Cache Grows

During multi-turn chat or document summarization, the Transformer architecture must store the Key and Value vectors for every previous token across every attention head and hidden layer.

The formula for the KV Cache size is:

$$\text{KV Cache (Bytes)} = 2 \times \text{Layers} \times \text{Hidden Dimension} \times \text{Context Length} \times \text{Bytes per Element}$$

Consider the practical implications on an 8B model (such as Llama 3.1 8B, with 32 layers and 4096 hidden size):

• At 4,096 tokens (4k context): The KV cache occupies $\approx 1.07\text{ GB}$.
• At 16,384 tokens (16k context): The KV cache occupies $\approx 4.29\text{ GB}$.
• At 65,536 tokens (64k context): The KV cache occupies $\approx 17.17\text{ GB}$.
• At 131,072 tokens (128k context): The KV cache occupies $\approx 34.35\text{ GB}$!

Notice what happened: The attention context buffer at 64k tokens is more than three times larger than the entire model itself!

Solutions for KV Cache Bloat

1. KV Cache Quantization (--cache-type-k q8_0 --cache-type-v q8_0): llama.cpp and vLLM allow you to quantize the KV cache to 8-bit or 4-bit. Enabling 8-bit KV cuts context memory in half with virtually zero loss in needle-in-a-haystack retrieval accuracy.
2. FlashAttention: Always ensure FlashAttention is enabled (-fa in llama.cpp), which calculates self-attention in fast GPU SRAM chunks without materializing massive intermediate memory matrices.

---

Hardware Hierarchy: The Best Rigs for Every Budget

Tier 1: The $300–$500 Starter Rig (8B Models at Full Speed)

• GPU: NVIDIA GeForce RTX 3060 12GB (Used: ~$220–$260) or RTX 4060 8GB / 16GB
• What it runs: Llama 3.1 8B Q4_K_M, Mistral 7B, Gemma 2 9B, Qwen 2.5 7B.
• Performance: 60–90 tokens/second.
• Why it matters: The 12GB RTX 3060 has been the king of entry-level AI for three years because its 12GB VRAM buffer leaves comfortable headroom for the model plus an 8k context window.

Tier 2: The $800–$1,200 Sweet Spot (14B–32B Models)

• GPU: NVIDIA GeForce RTX 3090 24GB (Used: ~$650–$750) or RTX 4090 24GB
• What it runs: Qwen 2.5 14B Q8_0, DeepSeek R1 Distill 32B Q4_K_M, Command R 35B.
• Performance: 40–80 tokens/second.
• Why it matters: 24GB of VRAM on a 384-bit memory bus is the single most versatile consumer hardware platform in existence. A single RTX 3090 handles virtually any modern 8B–32B coding or reasoning model with large context buffers.

Tier 3: The 70B Heavyweight Champion (Apple Silicon Unified Memory)

• Hardware: Apple Mac Studio (M2/M3 Max or Ultra) with 64GB, 128GB, or 192GB Unified Memory
• What it runs: Llama 3.1 70B Q4_K_M, Qwen 2.5 72B, DeepSeek 67B.
• Performance: 20–35 tokens/second (Ultra) or 12–18 tokens/second (Max).
• Why it matters: Building a PC with >70GB of VRAM requires at least three to four discrete RTX 3090 GPUs, specialized server motherboards with multiple PCIe x16 slots, high-wattage 1600W power supplies, and dedicated cooling. An Apple Mac Studio sits silently on a desk, sips 120 Watts of power, and allows the GPU direct access to up to 192GB of unified system RAM at 800 GB/s bandwidth.

---

Quick Reference CLI Commands for Immediate Testing

To run these models with optimal GPU layer offloading and FlashAttention right now, use the following terminal commands:

# 1. Run Llama 3.1 8B with Ollama (auto-detects VRAM)
ollama run llama3.1:8b

# 2. Run Qwen 2.5 14B with custom context window (16k) in Ollama
ollama run qwen2.5:14b-instruct-q4_K_M --context 16384

# 3. Maximum-efficiency llama.cpp server with FlashAttention and 8-bit KV cache
./llama-server -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 16384 \
  --flash-attn \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080

By sizing your VRAM accurately before choosing your model weights, you eliminate memory swapping, avoid costly hardware buyer's remorse, and achieve commercial-grade inference speeds entirely on your own local machines.