LLM Quantization in 2026: Run 70B Models on Consumer Hardware

Hero: VRAM usage comparison across quantization formats

A LLaMA 3.3 70B model in full float32 precision requires 280GB of VRAM. In a 4-bit quantization format, the same model runs in 40GB — well within a single A100 or two 3090s. With modern quantization techniques, you can run that 70B model on a Mac Studio with 96GB unified memory, or a workstation with two consumer GPUs, with less than 5% quality degradation on most benchmarks.

Quantization is no longer a compromise for resource-constrained deployments. It's the default approach for running large models efficiently, both locally and in production inference infrastructure.

The Problem: VRAM is the Bottleneck

Modern LLMs are measured in billions of parameters. Each parameter in float32 precision occupies 4 bytes. The math:

LLaMA 3.3 70B in fp32:  70B × 4 bytes = 280GB
LLaMA 3.3 70B in fp16:  70B × 2 bytes = 140GB
LLaMA 3.3 70B in int8:  70B × 1 byte  = 70GB
LLaMA 3.3 70B in int4:  70B × 0.5 byte = 35GB (+ overhead ≈ 40GB total)

But VRAM requirements during inference aren't just model weights. You also need the KV cache (grows with context length and batch size) and activations during forward pass. A 70B model at int4 with a 4096-token context and batch size 8 needs roughly 50-55GB total.

The second problem is inference speed. Memory bandwidth — how fast the GPU can read model weights — determines tokens-per-second more than raw compute at typical batch sizes. A smaller quantized model loads faster from VRAM, yielding higher throughput even at the same VRAM budget.

xychart-beta
    title "Tokens/Second vs VRAM Usage (LLaMA 3.3 70B)"
    x-axis ["fp32", "fp16/bf16", "int8 GPTQ", "int4 AWQ", "int4 GGUF Q4_K_M"]
    y-axis "Tokens/second" 0 --> 50
    bar [2, 8, 18, 35, 30]

This is the core trade-off: less memory → higher throughput, at the cost of some quality. The question quantization research has focused on is: *how much quality do you actually lose?*

How It Works: Quantization Fundamentals

Neural network weights are floating-point numbers in a continuous range — typically distributed roughly normally around zero. Quantization maps those continuous values to a discrete set of integers (int8 = 256 values, int4 = 16 values).

The simplest form (absmax quantization):

scale = max(|weights|) / 127          # For int8
quantized = round(weight / scale)      # Float → Integer
dequantized = quantized × scale        # Integer → Float (at inference time)

The quality loss comes from rounding error — the difference between the original float and the closest representable integer. The goal of advanced quantization methods is to minimize this error where it matters most.

flowchart LR
    A[float32 weights\n280GB] --> B{Quantization}
    B --> C[int4 weights\n35GB]
    C --> D[Dequantize\nduring inference]
    D --> E[Matrix multiply\nin fp16]
    E --> F[Output\n≈ Same quality]
    
    B -.->|"Calibration data\nminimizes rounding error"| B
    
    style A fill:#ef4444,color:#fff
    style C fill:#22c55e,color:#fff
    style F fill:#3b82f6,color:#fff

What the Methods Actually Do Differently

GPTQ (Gradient-based Post-Training Quantization): Quantizes layer by layer, computing the optimal quantization order using second-order gradient information (the Hessian matrix). It compensates for each quantized weight by adjusting remaining weights to minimize layer output error. Produces high-quality int4 models but requires calibration data and takes 1-4 hours to quantize a 70B model.

AWQ (Activation-Aware Weight Quantization): Observes that not all weights are equally important — weights that correspond to large activations contribute more to output error when quantized. AWQ protects these "salient" weights by scaling channels before quantization, reducing quantization error without mixed precision. Faster to quantize than GPTQ, comparable quality. Requires a calibration dataset (512 representative prompts).

GGUF (from llama.cpp): A file format, not a quantization algorithm. GGUF files can contain models quantized with various algorithms (K-quants, IQ-quants). K-quants use a k-means-based approach where quantization parameters are computed per block of 32-256 weights. GGUF enables CPU offloading: layers that don't fit in VRAM are offloaded to RAM/CPU, which slows inference but makes it possible to run models that don't fit on GPU at all.

BitsAndBytes (bitsandbytes library): Real-time quantization during inference. No pre-quantization step. Slightly slower than pre-quantized formats but lets you load any model in 4-bit or 8-bit instantly with load_in_4bit=True. Good for experimentation, less optimal for production.

Implementation: Loading and Running Quantized Models

AWQ with vLLM (Production Serving)

For production serving, AWQ + vLLM is the current best combination — it leverages AWQ's high quality with vLLM's PagedAttention and continuous batching:

# Step 1: Quantize a model with AWQ (one-time, ~2hrs for 70B)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Llama-3.3-70B-Instruct"
quant_path = "./llama-3.3-70b-awq-int4"

# Load base model (needs enough RAM to load fp16 first)
model = AutoAWQForCausalLM.from_pretrained(model_path, device_map="cpu")
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Quantize with calibration data
quant_config = {
    "zero_point": True,    # Zero-point quantization (better quality)
    "q_group_size": 128,   # Smaller = better quality, larger = faster
    "w_bit": 4,            # 4-bit weights
    "version": "GEMM"      # GEMM kernel (use GEMV for batch_size=1)
}

model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data="pileval",  # Calibration dataset
    n_samples=512,
    max_seq_len=512,
)

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)
print(f"AWQ model saved to {quant_path}")


# Step 2: Serve with vLLM
# vllm serve ./llama-3.3-70b-awq-int4 \
#   --quantization awq \
#   --max-model-len 8192 \
#   --gpu-memory-utilization 0.90 \
#   --tensor-parallel-size 2  # Split across 2 GPUs


# Step 3: Query via OpenAI-compatible API
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"
)

response = client.chat.completions.create(
    model="llama-3.3-70b-awq-int4",
    messages=[{"role": "user", "content": "Explain GGUF quantization in 2 sentences."}],
    max_tokens=512,
    temperature=0.1
)
print(response.choices[0].message.content)

GGUF with llama.cpp / Ollama (Local Inference)

For local development and experimentation, GGUF files with llama.cpp-based runners are the easiest path:

# Install Ollama (wraps llama.cpp)
curl -fsSL https://ollama.ai/install.sh | sh

# Pull a pre-quantized model (downloads from Ollama Hub or HuggingFace)
ollama pull llama3.3:70b-instruct-q4_K_M   # 4-bit K-quant, M=medium quality

# Run interactively
ollama run llama3.3:70b-instruct-q4_K_M

# Or serve via API
ollama serve &
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.3:70b-instruct-q4_K_M",
  "prompt": "What is quantization?",
  "stream": false
}'

For custom GGUF quantization with specific precision:

# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp
make -j4

# Convert HuggingFace model to GGUF f16
python3 convert_hf_to_gguf.py /path/to/llama-3.3-70b --outfile llama-3.3-70b-f16.gguf

# Quantize to Q4_K_M (recommended balance of quality and speed)
./llama-quantize llama-3.3-70b-f16.gguf llama-3.3-70b-q4km.gguf Q4_K_M

# Quantize variants to compare
./llama-quantize llama-3.3-70b-f16.gguf llama-3.3-70b-q8_0.gguf Q8_0    # Near-lossless
./llama-quantize llama-3.3-70b-f16.gguf llama-3.3-70b-q2k.gguf Q2_K      # Extreme compression

# Benchmark
./llama-bench -m llama-3.3-70b-q4km.gguf -p 512 -n 512 -t 8

BitsAndBytes for Rapid Prototyping

When you don't want to pre-quantize — just load and experiment:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# 4-bit NF4 quantization with double quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",         # NormalFloat4: better for normally-distributed weights
    bnb_4bit_use_double_quant=True,    # Quantize the quantization constants too (~0.4 bits/param saved)
    bnb_4bit_compute_dtype=torch.bfloat16  # Computation in bf16 (not the weights)
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",   # Automatically distribute across available GPUs
    torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=50, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Choosing the Right Format: Decision Matrix

flowchart TD
    A{Deployment target?}
    A -- Production API --> B{GPU available?}
    A -- Local dev/research --> C[GGUF + Ollama\nQ4_K_M or Q8_0]
    
    B -- Yes, 2+ GPUs --> D[AWQ int4 + vLLM\nBest throughput]
    B -- Yes, 1 GPU --> E{Fit in VRAM?}
    B -- No GPU / CPU only --> F[GGUF + llama.cpp\nCPU offload]
    
    E -- Yes --> G[AWQ or GPTQ int4]
    E -- No --> H[GGUF with layer offload\nor split to 2 GPUs]
    
    style D fill:#22c55e,color:#fff
    style C fill:#3b82f6,color:#fff
    style G fill:#22c55e,color:#fff

|--------|----------|---------|-------|-------------------|

Rule of thumb: For production GPU serving, use AWQ. For local experimentation, use GGUF Q4_K_M. If you need near-lossless quality and have the VRAM, use Q8_0 or skip quantization entirely with fp16.

Quality Evaluation: How Much Do You Actually Lose?

Benchmark numbers from LLaMA 3.3 70B (MMLU 5-shot):

|-----------|------|------------|------|-------|

| fp16 (baseline) | 86.4 | 4.12 | 140GB | Reference |

| AWQ int4 | 85.8 | 4.28 | 42GB | -0.6% MMLU, 3.9% perplexity increase |

| GPTQ int4 | 85.6 | 4.31 | 42GB | Similar to AWQ |

| GGUF Q4_K_M | 85.3 | 4.38 | 43GB | -1.1% MMLU |

| GGUF Q2_K | 81.2 | 5.87 | 22GB | -5.2% MMLU — noticeable degradation |

The practical takeaway: int4 methods lose less than 1% on standard benchmarks for the 70B parameter class. At 7B and 13B, the losses are larger. Smaller models have less redundancy, so quantization error matters more. For models below 7B, prefer fp16 or Q8_0 if VRAM allows.

Combining Quantization with Speculative Decoding

Quantization reduces memory bandwidth usage. Speculative decoding reduces the number of sequential generation steps. Together, they compound:

Speculative decoding uses a small "draft" model to generate N token candidates quickly, then verifies them with the larger model in parallel. The large model only runs once to verify N tokens instead of N sequential forward passes. Combined with quantization on both models:

# Speculative decoding with vLLM
# Start server with both draft and target model
# vllm serve meta-llama/Llama-3.3-70B-Instruct-AWQ \
#   --quantization awq \
#   --speculative-model meta-llama/Llama-3.2-1B-Instruct \
#   --num-speculative-tokens 5 \
#   --tensor-parallel-size 2

# Request — same API, speculative decoding is transparent
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="Llama-3.3-70B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Write a Python function to compute fibonacci numbers."}],
    max_tokens=512,
    temperature=0.0   # Deterministic — speculative decoding works best with low temp
)

The throughput gain depends on the acceptance rate — how often the large model agrees with the draft model's tokens. For code generation and structured output, acceptance rates of 80%+ are common (similar vocabulary patterns). For creative writing with high temperature, rates drop to 50-60%.

Typical combined speedup on a 70B AWQ model with speculative decoding from a 1B draft: 3-5× tokens/second vs non-speculative fp16. At lower batch sizes (interactive latency), the gains are larger.

Hardware Landscape in 2026

Understanding quantization means understanding the hardware it runs on:

|----------|------|-------------------|-------|

| AMD MI300X | 192GB | fp16 | Exceptional for large models |

The Mac Studio with M4 Max or M3 Ultra changed local inference economics. 128-192GB unified memory means the 70B model loads entirely into fast memory — no slow CPU offloading. GGUF models run via llama.cpp's Metal backend at 15-20 tokens/second on the Ultra, competitive with a single A100 for low-batch latency.

For consumer GPU setups, two RTX 4090s (48GB combined) can run a 70B model in GGUF Q4_K_M with some layers CPU-offloaded, achieving 12-18 tokens/second. This costs ~$3,000 vs ~$20,000 for a single H100 — the economics of local deployment have fundamentally shifted.

Production Considerations

Calibration Data Quality

AWQ and GPTQ require calibration data — a sample of text that represents your deployment distribution. Using generic calibration data (like Wikipedia) to quantize a code model produces worse results than calibrating on code. Match calibration data to inference domain:

# Custom calibration dataset for code-focused deployment
from datasets import load_dataset

def get_code_calibration_data(tokenizer, n_samples=512, max_length=512):
    dataset = load_dataset("bigcode/the-stack-dedup", data_files="data/python/*.parquet", streaming=True)
    
    samples = []
    for item in dataset["train"]:
        tokens = tokenizer(item["content"], return_tensors="pt", max_length=max_length, truncation=True)
        if tokens["input_ids"].shape[1] >= 128:  # Skip very short samples
            samples.append(tokens["input_ids"])
        if len(samples) >= n_samples:
            break
    
    return samples

Mixed Precision for Critical Layers

The first and last layers of a transformer (embedding and unembedding) are disproportionately sensitive to quantization. GPTQ and AWQ both support keeping specific layers in fp16:

# AWQ: exclude sensitive layers from quantization
quant_config = {
    "w_bit": 4,
    "q_group_size": 128,
    "modules_to_not_convert": ["lm_head", "embed_tokens"]  # Keep in fp16
}

This adds ~2GB to the total model size but can prevent the output quality degradation that shows up as repetitive outputs or hallucinated tokens.

Monitoring Quantization Quality in Production

Perplexity on a held-out validation set is the standard offline metric. In production, track:

# Track token probability distribution as a quality proxy
# Quantization errors show up as increased entropy in output distributions

import scipy.stats

def measure_output_entropy(logits: torch.Tensor) -> float:
    """Higher entropy = less confident = potential quantization quality issue."""
    probs = torch.softmax(logits[0, -1, :], dim=-1).cpu().numpy()
    return float(scipy.stats.entropy(probs))

# Alert if rolling average entropy increases significantly
# compared to fp16 baseline on the same prompts

Quantization Formats in the Ecosystem: What to Download

When looking for pre-quantized models on HuggingFace, you'll encounter several naming conventions:

# GGUF model naming (llama.cpp format):
llama-3.3-70b-instruct-Q4_K_M.gguf   → 4-bit K-quant, medium quality (best balance)
llama-3.3-70b-instruct-Q4_K_S.gguf   → 4-bit K-quant, small (faster, slightly lower quality)
llama-3.3-70b-instruct-Q6_K.gguf     → 6-bit K-quant (near-lossless, but larger)
llama-3.3-70b-instruct-Q8_0.gguf     → 8-bit (essentially lossless, 2× the 4-bit size)
llama-3.3-70b-instruct-IQ2_M.gguf    → 2-bit iMatrix quant (extreme compression, significant loss)

# GPTQ model naming:
Llama-3.3-70B-Instruct-GPTQ-Int4     → 4-bit GPTQ
Llama-3.3-70B-Instruct-GPTQ-Int8     → 8-bit GPTQ

# AWQ model naming:
Llama-3.3-70B-Instruct-AWQ           → 4-bit AWQ (standard)

For GGUF files, Q4_K_M is the standard recommendation — tested across thousands of models, the M (medium) variant uses slightly more space than S (small) but with meaningfully better quality for code and reasoning tasks. Go to Q6_K if you have the VRAM/RAM and want near-fp16 quality without the full fp16 cost.

Reputable pre-quantized repositories: TheBloke (deprecated but archived), bartowski, and the official model providers increasingly publish their own GGUF and AWQ variants.

Fine-Tuning Quantized Models: QLoRA

Quantization and fine-tuning intersect in QLoRA (Quantized Low-Rank Adaptation). Instead of fine-tuning a full fp16 model (which needs 140GB for 70B), QLoRA loads the base model in int4 and fine-tunes only the LoRA adapter weights in float16. The result: fine-tune a 70B model on a single A100 80GB.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training

# Load base model in 4-bit for training
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Prepare model for k-bit training (enables gradient checkpointing, casts layernorm)
model = prepare_model_for_kbit_training(model)

# Add LoRA adapters — only these train, base model stays frozen in int4
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 167,772,160 || all params: 70,486,413,312 || trainable%: 0.238

# The adapter trains normally with Trainer/SFTTrainer
# Peak VRAM during training: ~65GB (fits in one A100 80GB)

After training, the adapter can be merged into the quantized model for deployment, or served separately as a LoRA adapter via vLLM's multi-adapter support. QLoRA makes fine-tuning accessible on hardware that previously couldn't even run inference on 70B models.

Conclusion

Quantization in 2026 is no longer a niche technique for memory-constrained deployments. It's the default way to run large language models efficiently. The key insights:

int4 AWQ/GPTQ loses less than 1% quality on 70B+ models while cutting memory 4× — use these for production GPU serving
GGUF Q4_K_M is the best format for local development: runs on CPU+GPU mixed, easy to distribute, good quality
BitsAndBytes is for rapid prototyping only — too slow for production but instant to start
Calibration data matters — match it to your deployment domain for the best quality
Small models (< 7B) quantize poorly — prefer fp16 or Q8_0 for these

The hardware gap between research labs (H100 clusters) and practitioners (consumer GPUs, Mac Studio) has closed significantly. A 70B model that required $500K in infrastructure two years ago now runs on a $4,000 workstation.

Quantization is now a first-class workflow in the LLM ecosystem. Model providers publish AWQ and GGUF variants alongside their base releases. llama.cpp and Ollama have abstracted the complexity to the point where running a quantized 70B model locally requires a single command. The techniques will continue improving — IQ-quants (importance matrix quantization) at 2-3 bits are showing quality competitive with older 4-bit methods. Follow the llama.cpp and vLLM changelogs for the current state of the art.

When Not to Quantize

Quantization is a tradeoff, not a universal win. Two scenarios where you should skip it:

Small models under 7B parameters: At 7B, int4 quantization loses 2-3% on standard benchmarks — more than the < 1% loss at 70B. For specialized tasks (code completion, function calling) the loss can be larger. If your use case involves a 7B model that fits in fp16, keep it in fp16.

When output quality is the primary metric: Medical reasoning, legal document analysis, and safety-critical systems should benchmark thoroughly before deploying quantized models in production. The average benchmark loss of < 1% doesn't mean specific edge cases don't regress further. Always run domain-specific evaluation on your actual prompts before choosing a quantization level.

Sources

GPTQ paper: "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers"
AWQ paper: "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration"
llama.cpp K-quants documentation
vLLM quantization documentation
LLaMA 3.3 70B benchmarks, Meta AI research

Enjoyed this post? Follow AmtocSoft for AI tutorials from beginner to professional.

☕ Buy Me a Coffee | 🔔 YouTube | 💼 LinkedIn | 🐦 X/Twitter

Search This Blog

AmtocSoft Tech Insights