ModelRefs / llama.cpp — AI Glossary

llama.cpp — AI Glossary

A CPU/GPU inference engine for quantized LLMs in pure C/C++, enabling local inference without Python or ML frameworks.

Overview

llama.cpp (Gerganov 2023) implements efficient GGML tensor operations for quantized LLM inference. Supports GGUF model format with INT4/INT8/BF16 quantization, partial GPU offloading (layers on GPU, rest on CPU RAM), Metal (Apple Silicon), CUDA, and Vulkan backends. The foundation of LM Studio, Ollama, and Jan.

Reference details

Topic	infrastructure
Last reviewed	2026-06-24

How it works

The engine is implemented in C and C++ with no Python or framework dependency, so a build runs wherever a compiler does.
Models are loaded from a single-file quantised format that carries weights and metadata together, which is what makes distribution a file copy.
Quantisation is per block with its own scale, so 4-bit and 5-bit variants trade memory against quality at a granularity the user picks at download time.
Layers can be split between accelerator and host memory, so a model larger than available video memory still runs — slowly, at the speed of the slowest tier.
Backends cover several platforms, which is why it is the common path for local inference on consumer hardware.

Example: What partial offload actually costs

A 13B model at 4-bit is about 13e9 x 0.5 ≈ 6.5GB. On an 8GB card it fits and runs at accelerator speed. A 34B model at 4-bit is about 17GB, so on the same card perhaps 40% of layers offload and 60% run against host memory — which has roughly an order of magnitude less bandwidth. Since decoding reads every weight each token, throughput falls toward the slow tier's rate rather than averaging the two. Offload converts 'will not run' into 'runs slowly', which is often the right trade and is not a small one.

Commonly confused with

Treated as an alternative to production serving engines, which it is not aimed at. Server-class engines optimise many concurrent requests on accelerators with paged cache and continuous batching; this optimises one or a few requests on whatever hardware is present, including CPUs. That makes it the right tool for local and embedded use and the wrong one for a multi-tenant endpoint. It is also distinct from the quantisation methods themselves — it consumes a quantised file, and the format is not the algorithm that produced it.

When to use it

Reach for it when:

Local or offline inference on consumer hardware, including CPU-only machines.
A dependency-free binary is needed, or the target is embedded.
A model must run on hardware that cannot hold it, accepting the offload penalty.

Reach for something else when:

Serving many concurrent users, where a server-class engine is far more efficient.
Throughput per device is the metric being optimised.
Full-precision quality is required, since the format is quantised by design.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to llama.cpp — AI Glossary.

Frequently asked questions

What is llama.cpp?