NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 5090
VRAM: 32 GB
BW: 1792 GB/s
$1999 / £1899
7.4 / 32 GB (Q4_K_M)
~265.0 tok/s
US & UK AI Infrastructure Standard
GGUF • EXL2 • AWQ • vLLM
LLM Quantization & Multi-GPU VRAM Calculator
Precision hardware sizing for AI engineers across the US & UK. Estimate GGUF, EXL2, AWQ, and FP16 memory footprints, KV cache overhead, and real-world tokens/sec across 30+ GPUs.
ollama run llama3.1:8b
Loaded calculator for an 8.03B Parameter LLM on NVIDIA RTX 4090. Recommended 4-bit Q4_K_M requires 7.4 GB VRAM.
1. Model & Hardware Configuration
2. Hardware Compatibility & Speed Telemetry
Evaluating an 8.03B Parameter LLM against 1x NVIDIA RTX 4090 (24 GB VRAM).
Q4_K_M Total VRAM
7.37 GB
Weights: 4.9 GB
KV Cache Memory
1.00 GB
8,192 ctx (FP16)
Est. Speed (Q4_K_M)
~48.5 tok/s
1008 GB/s BW
Best Quantization
Q6_K / Q8_0
Max quality fit
VRAM Allocation Breakdown (Q4_K_M 4-bit Recommended)
7.37 GB used of 24 GB VRAM
Model Weights (4.87 GB)
GQA KV Cache (1.00 GB)
Compute Buffer (1.50 GB)
3. Complete Quantization Formats Breakdown (GGUF, EXL2, AWQ, FP16)
| Format & Engine | Bitrate | Weights | KV Cache | Total VRAM | Est. Speed | Quality / Perplexity | Hardware Status |
|---|---|---|---|---|---|---|---|
|
FP16 / BF16
Full Precision • Original Weights
|
16.00 bpw | 16.06 GB | 1.00 GB | 18.56 GB | ~45.2 tok/s | Lossless 100% (+0.000 (Baseline)) | Fits in VRAM (100% GPU) |
|
FP8 / Q8_0
GGUF / vLLM • Production Grade
|
8.50 bpw | 8.53 GB | 1.00 GB | 11.03 GB | ~85.1 tok/s | 99.9% Near-Lossless (+0.004) | Fits in VRAM (100% GPU) |
|
Q6_K (6-bit)
GGUF k-quant • High Precision
|
6.56 bpw | 6.58 GB | 1.00 GB | 9.08 GB | ~110.2 tok/s | 99.5% Excellent (+0.021) | Fits in VRAM (100% GPU) |
|
Q5_K_M (5-bit)
GGUF k-quant • Balanced Plus
|
5.69 bpw | 5.71 GB | 1.00 GB | 8.21 GB | ~127.1 tok/s | 99.0% Very High (+0.056) | Fits in VRAM (100% GPU) |
|
Q4_K_M (4-bit)
GGUF k-quant • Recommended
|
4.85 bpw | 4.87 GB | 1.00 GB | 7.37 GB | ~149.1 tok/s | 98.2% Sweet Spot (+0.152) | Fits in VRAM (100% GPU) |
|
AWQ / GPTQ 4-bit
vLLM / TensorRT • High Throughput
|
4.50 bpw | 4.52 GB | 1.00 GB | 7.02 GB | ~160.7 tok/s | 98.0% Fast Serving (+0.160) | Fits in VRAM (100% GPU) |
|
EXL2 (4.0 bpw)
ExLlamaV2 • Max Speed
|
4.00 bpw | 4.01 GB | 1.00 GB | 6.51 GB | ~180.8 tok/s | 97.6% Ultra Fast (+0.195) | Fits in VRAM (100% GPU) |
|
IQ4_XS (i-Matrix)
GGUF i-quant • VRAM Saver
|
4.25 bpw | 4.27 GB | 1.00 GB | 6.77 GB | ~170.1 tok/s | 97.8% Compact 4-bit (+0.180) | Fits in VRAM (100% GPU) |
|
Q3_K_M (3-bit)
GGUF k-quant • Tight VRAM
|
3.91 bpw | 3.92 GB | 1.00 GB | 6.42 GB | ~184.9 tok/s | 94.5% Noticeable Loss (+0.480) | Fits in VRAM (100% GPU) |
|
IQ3_XXS (3-bit i-quant)
GGUF i-quant • Extreme Fit
|
3.06 bpw | 3.07 GB | 1.00 GB | 5.57 GB | ~236.3 tok/s | 92.0% Degraded (+0.710) | Fits in VRAM (100% GPU) |
|
Q2_K (2-bit)
GGUF k-quant • Last Resort
|
2.63 bpw | 2.64 GB | 1.00 GB | 5.14 GB | ~274.9 tok/s | 85.0% Experimental (+1.650) | Fits in VRAM (100% GPU) |
|
IQ1_S (1.58-bit)
GGUF Sub-2-bit • Ultra Extreme
|
1.56 bpw | 1.57 GB | 1.00 GB | 4.07 GB | ~350.0 tok/s | 72.0% Heavy Loss (+3.800) | Fits in VRAM (100% GPU) |
Multi-Vendor GPU & Accelerator Compatibility Cards
Live comparison across 31 consumer, workstation, Apple Unified Memory, and data-center GPUs for your active model configuration (shown for Q4_K_M 4-bit quantization). Click any card for its dedicated hardware benchmark page.
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 5080
VRAM: 16 GB
BW: 960 GB/s
$999 / £949
7.4 / 16 GB (Q4_K_M)
~142.0 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 5070 Ti
VRAM: 16 GB
BW: 896 GB/s
$749 / £719
7.4 / 16 GB (Q4_K_M)
~132.5 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 5070
VRAM: 12 GB
BW: 672 GB/s
$549 / £529
7.4 / 12 GB (Q4_K_M)
~99.4 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 4090
VRAM: 24 GB
BW: 1008 GB/s
$1599 / £1549
7.4 / 24 GB (Q4_K_M)
~149.1 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 4080 Super
VRAM: 16 GB
BW: 736 GB/s
$999 / £959
7.4 / 16 GB (Q4_K_M)
~108.9 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 4070 Ti Super
VRAM: 16 GB
BW: 672 GB/s
$799 / £769
7.4 / 16 GB (Q4_K_M)
~99.4 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 4070 Super
VRAM: 12 GB
BW: 504 GB/s
$599 / £579
7.4 / 12 GB (Q4_K_M)
~74.5 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 4060 Ti 16GB
VRAM: 16 GB
BW: 288 GB/s
$449 / £429
7.4 / 16 GB (Q4_K_M)
~42.6 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 4060
VRAM: 8 GB
BW: 272 GB/s
$299 / £289
7.4 / 8 GB (Q4_K_M)
~40.2 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 3090 / 3090 Ti
VRAM: 24 GB
BW: 936 GB/s
$799 / £749
7.4 / 24 GB (Q4_K_M)
~138.4 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 3080 12GB
VRAM: 12 GB
BW: 912 GB/s
$499 / £469
7.4 / 12 GB (Q4_K_M)
~134.9 tok/s
NVIDIA • Consumer GPU
Runs on 1x Card
NVIDIA RTX 3060 12GB
VRAM: 12 GB
BW: 360 GB/s
$279 / £259
7.4 / 12 GB (Q4_K_M)
~53.2 tok/s
Apple • Unified Memory
Runs on 1x Card
Apple M3 Ultra (Mac Studio)
VRAM: 192 GB
BW: 800 GB/s
$5999 / £5799
7.4 / 192 GB (Q4_K_M)
~118.3 tok/s
Apple • Unified Memory
Runs on 1x Card
Apple M4 Max 128GB
VRAM: 128 GB
BW: 546 GB/s
$4499 / £4299
7.4 / 128 GB (Q4_K_M)
~80.8 tok/s
Apple • Unified Memory
Runs on 1x Card
Apple M4 Max 64GB
VRAM: 64 GB
BW: 546 GB/s
$3699 / £3499
7.4 / 64 GB (Q4_K_M)
~80.8 tok/s
Apple • Unified Memory
Runs on 1x Card
Apple M4 Pro 64GB (Mac mini)
VRAM: 64 GB
BW: 273 GB/s
$1999 / £1899
7.4 / 64 GB (Q4_K_M)
~40.4 tok/s
Apple • Unified Memory
Runs on 1x Card
Apple M4 24GB (Mac mini)
VRAM: 24 GB
BW: 120 GB/s
$799 / £769
7.4 / 24 GB (Q4_K_M)
~17.7 tok/s
AMD • Consumer GPU
Runs on 1x Card
AMD Radeon RX 7900 XTX
VRAM: 24 GB
BW: 960 GB/s
$899 / £849
7.4 / 24 GB (Q4_K_M)
~142.0 tok/s
AMD • Consumer GPU
Runs on 1x Card
AMD Radeon RX 7900 XT
VRAM: 20 GB
BW: 800 GB/s
$699 / £669
7.4 / 20 GB (Q4_K_M)
~118.3 tok/s
AMD • Workstation GPU
Runs on 1x Card
AMD Radeon PRO W7900
VRAM: 48 GB
BW: 864 GB/s
$3499 / £3299
7.4 / 48 GB (Q4_K_M)
~127.8 tok/s
AMD • Data Center GPU
Runs on 1x Card
AMD Instinct MI300X
VRAM: 192 GB
BW: 5300 GB/s
$15000 / £14200
7.4 / 192 GB (Q4_K_M)
~350.0 tok/s
NVIDIA • Workstation GPU
Runs on 1x Card
NVIDIA RTX 6000 Ada
VRAM: 48 GB
BW: 960 GB/s
$6800 / £6400
7.4 / 48 GB (Q4_K_M)
~142.0 tok/s
NVIDIA • Workstation GPU
Runs on 1x Card
NVIDIA RTX A6000
VRAM: 48 GB
BW: 768 GB/s
$4500 / £4200
7.4 / 48 GB (Q4_K_M)
~113.6 tok/s
NVIDIA • Data Center GPU
Runs on 1x Card
NVIDIA L40S 48GB
VRAM: 48 GB
BW: 864 GB/s
$8500 / £8000
7.4 / 48 GB (Q4_K_M)
~127.8 tok/s
NVIDIA • Data Center GPU
Runs on 1x Card
NVIDIA A100 80GB SXM
VRAM: 80 GB
BW: 2039 GB/s
$14000 / £13200
7.4 / 80 GB (Q4_K_M)
~301.6 tok/s
NVIDIA • Data Center GPU
Runs on 1x Card
NVIDIA H100 80GB SXM5
VRAM: 80 GB
BW: 3350 GB/s
$27000 / £25500
7.4 / 80 GB (Q4_K_M)
~350.0 tok/s
NVIDIA • Data Center GPU
Runs on 1x Card
NVIDIA H200 141GB
VRAM: 141 GB
BW: 4800 GB/s
$32000 / £30000
7.4 / 141 GB (Q4_K_M)
~350.0 tok/s
NVIDIA • AI Workstation
Runs on 1x Card
NVIDIA DGX Spark (GB10)
VRAM: 128 GB
BW: 500 GB/s
$3000 / £2850
7.4 / 128 GB (Q4_K_M)
~73.9 tok/s
Intel • Consumer GPU
Runs on 1x Card
Intel Arc A770 16GB
VRAM: 16 GB
BW: 560 GB/s
$289 / £269
7.4 / 16 GB (Q4_K_M)
~82.8 tok/s
Intel • Consumer GPU
Runs on 1x Card
Intel Arc B580 12GB
VRAM: 12 GB
BW: 456 GB/s
$249 / £239
7.4 / 12 GB (Q4_K_M)
~67.4 tok/s
Programmatic LLM Hardware & Quantization Directory
Explore pre-calculated VRAM profiles, GQA KV cache tables, and multi-GPU deployment guides for top open-weights LLMs. Our edge router supports any custom model URL pattern (e.g., /model/your-model-70b).
Meta AI • Dense
70.6B Params
Meta Llama 3.3 70B Instruct
Meta Llama 3.3 70B Instruct delivers flagship 405B-class reasoning and coding performance in a dense 70.6-billion p...
Q4_K_M: 44.3 GB
FP16: 143.2 GB
Meta AI • Dense
8.03B Params
Meta Llama 3.1 8B Instruct
Meta Llama 3.1 8B is the industry-standard edge LLM for consumer GPUs, capable of running at 100+ tokens/sec in Q4_...
Q4_K_M: 6.4 GB
FP16: 18.1 GB
Meta AI • Dense
405B Params
Meta Llama 3.1 405B Instruct
Meta Llama 3.1 405B is one of the largest dense open-weights foundation models, requiring multi-GPU tensor parallel...
Q4_K_M: 247.0 GB
FP16: 812.0 GB
DeepSeek • MoE
671B Params
DeepSeek-R1 671B (MoE)
DeepSeek-R1 671B is a Mixture-of-Experts (MoE) reasoning model with 671B total parameters and 37B active parameters...
Q4_K_M: 408.3 GB
FP16: 1344.0 GB
DeepSeek • Dense
32.8B Params
DeepSeek-R1 Distill Qwen 32B
DeepSeek-R1 Distill Qwen 32B packs frontier chain-of-thought math and coding capability into a 32.8B dense model th...
Q4_K_M: 21.4 GB
FP16: 67.6 GB
DeepSeek • Dense
70.6B Params
DeepSeek-R1 Distill Llama 70B
Distilled from DeepSeek-R1 onto the Llama 3.3 70B backbone, offering elite reasoning accuracy for dual-24GB GPU set...
Q4_K_M: 44.3 GB
FP16: 143.2 GB
Alibaba Cloud • Dense
72.7B Params
Qwen 2.5 72B Instruct
Qwen 2.5 72B Instruct is a top-tier multilingual and coding foundation model with 80 transformer layers and 128K co...
Q4_K_M: 45.6 GB
FP16: 147.4 GB
Alibaba Cloud • Dense
32.5B Params
Qwen 2.5 Coder 32B Instruct
Qwen 2.5 Coder 32B is the premier open-weights code generation and refactoring model designed to fit on a single 24...
Q4_K_M: 21.2 GB
FP16: 67.0 GB
Alibaba Cloud • Dense
14.7B Params
Qwen 2.5 14B Instruct
Qwen 2.5 14B bridges the gap between 8B and 32B models, running effortlessly on 12GB and 16GB GPUs with room to spa...
Q4_K_M: 10.4 GB
FP16: 31.4 GB
Mistral AI • Dense
24B Params
Mistral Small 3 24B Instruct
Mistral Small 3 24B is engineered for low-latency local inference on 16GB and 24GB GPUs with only 40 layers for rap...
Q4_K_M: 16.0 GB
FP16: 50.0 GB
Mistral AI • Dense
123B Params
Mistral Large 2 123B
Mistral Large 2 123B is a flagship dense model ideal for 96GB+ Mac Studio configurations or dual-48GB workstation G...
Q4_K_M: 76.1 GB
FP16: 248.0 GB
Mistral AI • MoE
46.7B Params
Mixtral 8x7B Instruct (MoE)
Mixtral 8x7B uses a sparse Mixture-of-Experts architecture where 46.7B parameters reside in VRAM while only 12.9B a...
Q4_K_M: 29.8 GB
FP16: 95.4 GB
Mistral AI • MoE
141B Params
Mixtral 8x22B Instruct (MoE)
Mixtral 8x22B combines 141B total weights with a fast 39B active parameter path per token....
Q4_K_M: 87.0 GB
FP16: 284.0 GB
Google DeepMind • Dense
27.2B Params
Google Gemma 3 27B Instruct
Google Gemma 3 27B delivers exceptional performance-per-watt on single 24GB GPUs such as the RTX 3090, RTX 4090, an...
Q4_K_M: 18.0 GB
FP16: 56.4 GB
Microsoft • Dense
14.7B Params
Microsoft Phi-4 14B
Microsoft Phi-4 14B is trained on high-density synthetic textbook data, achieving STEM reasoning comparable to mode...
Q4_K_M: 10.4 GB
FP16: 31.4 GB
Cohere For AI • Dense
104B Params
Cohere Command R+ 104B
Cohere Command R+ 104B specializes in Retrieval-Augmented Generation (RAG), citations, and multi-step tool use....
Q4_K_M: 64.5 GB
FP16: 210.0 GB
Popular Model + GPU Hardware Pairing Guides (US & UK Benchmarks)
Exact VRAM & KV Cache Engineering Formulas
Unlike rough rule-of-thumb estimators, quantizationcalculator.com models the exact memory layout used by modern inference runtimes (llama.cpp, ExLlamaV2, vLLM, and TensorRT-LLM):
1. Quantized Model Weights (GB)
Weights_GB = (Parameters_Billions × Bits_Per_Weight) / 8
2. GQA Key-Value (KV) Cache (GB)
KV_GB = (2 × Layers × KV_Heads × Head_Dim × Context × (KV_Bits / 8)) / 1024³
3. Bandwidth-Bound Token Speed (tok/s)
Speed = (Memory_Bandwidth_GBs × 0.72) / Active_Weights_GB
Frequently Asked Questions
How much VRAM do I need to run an 8.03B Parameter LLM locally?
In full FP16/BF16 precision, an 8.03B Parameter LLM requires approximately 18.6 GB of VRAM including an 8,192-token KV cache and CUDA runtime buffer. Using 4-bit GGUF quantization (Q4_K_M at 4.85 bits per weight), total VRAM drops to 7.4 GB, while 8-bit quantization (Q8_0) requires 11.0 GB.
What is the difference between GGUF (Q4_K_M), EXL2, and AWQ quantization?
GGUF (used by llama.cpp, Ollama, and LM Studio) supports both pure GPU execution and hybrid CPU+GPU RAM offloading, making Q4_K_M the most versatile format. EXL2 (ExLlamaV2) runs strictly inside GPU VRAM and offers fine-grained bitrates (e.g., 3.5, 4.0, 4.25, 6.0 bpw) with maximum single-user tokens/sec. AWQ (Activation-aware Weight Quantization) is optimized for high-concurrency production inference engines like vLLM and TensorRT-LLM.
How does Context Window (KV Cache) affect GPU VRAM consumption?
Every token in your active context window stores Key and Value tensors across all transformer layers. With Grouped-Query Attention (GQA), KV cache memory equals 2 * Layers * KV_Heads * Head_Dim * Context_Tokens * Bytes_Per_Element. Quantizing the KV cache from FP16 (16-bit) to Q8_0 (8-bit) or Q4_0 (4-bit) cuts context memory overhead by 50% to 75% with minimal perplexity impact.
How are local LLM Tokens Per Second (tok/s) calculated across different GPU cards?
Autoregressive token generation (batch size 1) is memory-bandwidth bound rather than compute bound. For each generated token, the GPU must read all active model weights from VRAM into Tensor Cores. Therefore, theoretical peak speed equals Memory Bandwidth (GB/s) divided by Active Model Size (GB), typically achieving 65% to 78% efficiency in llama.cpp and ExLlamaV2.