MLC-LLM — AI Glossary

Reference details

Related terms

• llama.cpp
• Apple MLX
• Edge AI

How it works

1. The model's compute graph is captured and then compiled for a specific hardware target rather than interpreted at runtime, so operator fusion, memory layout and scheduling decisions are made ahead of time.
2. A compiler stack handles the target-specific code generation, which is what lets one source model produce artefacts for several accelerator families without hand-written kernels for each.
3. Quantisation is applied as part of compilation, so the weight format and the generated kernels are chosen together rather than a quantised model being handed to a generic runtime.
4. The output is a self-contained artefact per target — compiled kernels plus packed weights — which is the operational cost of the approach: one model times several targets is several builds to produce and test.
5. Targets include browser execution through a web graphics API, which makes in-page inference possible with no server and no installation at all.
6. On mobile the binding constraint is the per-application memory limit rather than total device memory, so the quantisation choice is what decides whether a model is deployable.

Example: Why mobile deployment is a memory-limit problem

A phone with 8GB of RAM does not give an application 8GB; the operating system typically allows a fraction before terminating it, so assume roughly 3GB usable. A 7B model at 4-bit is about 7e9 x 0.5 = 3.5e9 bytes, 3.5GB, which exceeds that budget before any context allocation — so it does not deploy, regardless of the device's total memory. A 3B model at 4-bit is 3e9 x 0.5 = 1.5GB, leaving roughly 3 - 1.5 = 1.5GB for the KV cache, activations and the rest of the application. Push the 7B to 3-bit and it is 7e9 x 0.375 = 2.6GB, which fits the weights and leaves only 0.4GB for everything else — technically loadable, practically fragile. The decision is therefore made by the memory ceiling and not by which model is better: at a 3GB budget the real choice is a 3B model with comfortable headroom or a heavily quantised 7B with none, and the second option trades the quality gain of more parameters against the quality loss of coarser quantisation, which is frequently a wash.

Commonly confused with

Compilation and quantisation are different levers that this approach applies together, and attributing its speed to the wrong one leads to wasted effort. Quantisation reduces memory traffic by shrinking the weights, which raises the bandwidth-bound generation ceiling. Compilation reduces overhead and improves scheduling for the specific hardware, which helps most where kernel launch cost and memory layout dominate. A compiled 4-bit model is faster than an interpreted 4-bit model and faster still than an interpreted 16-bit one, and knowing which contributed what is the difference between recompiling and requantising when the result is too slow.

When to use it

Reach for it when:

• On-device inference where no server is acceptable — privacy constraints, offline operation, zero marginal cost per request.
• Hardware without a mature native stack, where the compiler generates what would otherwise be hand-written kernels.
• In-browser inference, where the model must run in a page with no installation.
• Fixed, long-lived model-and-target combinations, so the compilation cost is paid once and amortised.

Reach for something else when:

• Server-side deployment on mainstream accelerators, where established engines are faster to adopt and better supported.
• Frequently changing models, where each change means recompiling and retesting every target.
• Frontier-size models, which do not fit the device classes this approach targets.
• Teams without capacity to maintain a build matrix, since the artefact count is models times targets.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to MLC-LLM — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path