Understand why massive context windows crash local AI models.
Simulating 4-bit quantization (e.g., Q4_K_M).
Dense GQA assumed — 8 kv-heads × 128 head-dim. MoE, MLA and sliding-window models differ.
llama.cpp defaults to fp16. Quantising the cache is the main lever for long contexts.
The core parameters of the AI. These files must be loaded into memory to execute inference. At 4-bit compression, this requires roughly 0.62 GB per 1 Billion parameters.
The model's active working memory. Its size is driven by layers × kv-heads × head-dim, not by parameter count — so it grows far faster than the weights as context expands.
This models the mechanism, for exact figures on a specific model, do your own math.