GGUF — AI Glossary

Reference details

Related terms

• Quantization
• Ollama
• Open Weights

How it works

1. The file opens with a magic number and version, so a loader can reject an incompatible file before reading anything large.
2. A key-value metadata block follows, carrying the architecture name, the hyperparameters needed to construct the model, and the full tokenizer — vocabulary, merges and special token ids.
3. Because the tokenizer travels inside the file, there is no separate vocabulary artifact to mismatch, which removes the most common cause of a model loading and then producing nonsense.
4. Tensor descriptors come next: for each tensor a name, a shape, a quantisation type and a byte offset into the data section.
5. Tensor data is stored contiguously and aligned, which lets the loader memory-map the file so weights are paged in by the operating system on demand rather than read into RAM up front.
6. That mapping is why a model larger than available RAM can still run, and why startup is near-instant on a second load — the pages are already in the filesystem cache.

Example: One file, and the two numbers that decide whether it runs

A GGUF file carries weights, tokenizer and metadata together, so running a model is opening a file rather than assembling a directory of shards, configs and vocab files that must all match. That self-containment is why local runtimes standardised on it — the format removes an entire class of setup failure. The quantisation written into that file is what decides whether it runs at all on a given machine. A 7B model at full half precision is 7e9 x 2 = 14 GB of weights; the same model at a common four-bit scheme is roughly 4.1 GB, a factor of about 3.4. On a machine with 8 GB available only the second fits, which is the entire reason consumer-grade local inference exists. Because the loader memory-maps the file rather than reading it, the resident set grows to what has actually been touched rather than to the file size — so a model can begin responding before all of it has been paged in.

Commonly confused with

GGUF is a container format, not a quantization method. The quantization scheme (Q4_K_M, Q5_K_S and so on) describes how weights were compressed; GGUF describes how the result is stored. Two GGUF files of the same model can differ enormously in quality and size. The practical consequence is that saying which model you are running conveys almost nothing on its own: the quantisation is the variable that moved, and a benchmark result from one file does not transfer to another file of the same model.

When to use it

Reach for it when:

• Running models locally through llama.cpp, Ollama or LM Studio
• CPU or mixed CPU/GPU inference on consumer hardware
• You want a single artifact to distribute
• Where memory is the binding constraint, since the quantisation in the file is what decides whether it loads

Reach for something else when:

• Server-class GPU serving, where vLLM-style formats are better suited
• You need to fine-tune — train first, convert afterwards
• Picking a quant level without checking quality on your own task
• Quoting a benchmark without naming the quantisation, which is the variable that moved

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GGUF — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path