Apple MLX — AI Glossary

Reference details

Related terms

• llama.cpp
• Edge AI
• Quantization

How it works

1. The framework targets a unified memory architecture in which processor and accelerator address the same physical memory, so a tensor is not copied between host and device — the transfer that dominates many workloads simply does not exist.
2. Arrays are lazily evaluated: operations build a graph and compute only when a result is needed, which lets the framework fuse operations and avoid materialising intermediates.
3. Kernels are generated for the platform's own compute API rather than a vendor GPU toolkit, which is why the performance is competitive on this hardware and unavailable anywhere else.
4. Quantised weight formats are first-class, since on this hardware the binding constraint is memory bandwidth and smaller weights raise the achievable token rate directly.
5. Low-rank fine-tuning is practical because the unified pool gives the accelerator access to far more memory than a consumer discrete card exposes, so the limiting factor becomes time rather than capacity.
6. Throughput is still bandwidth-bound in the same way as any other generation workload: tokens per second is at most memory bandwidth divided by the resident weight size.

Example: Capacity against bandwidth, which is the whole trade

A 70B model at 4-bit is about 70e9 x 0.5 = 35GB. On a machine with 36GB of unified memory that fits in one device's addressable pool; on consumer discrete cards with 24GB each it needs 2 cards and the sharding that implies. So capacity favours unified memory decisively. Bandwidth does not. Taking 400 GB/s for the unified machine and 1,000 GB/s for the discrete card, the generation ceiling is bandwidth over resident size: 400 / 35 = 11.4 tokens per second against 1,000 / 35 = 28.6 — the discrete setup is 28.6 / 11.4 = 2.5x faster on the same model. That is the trade stated plainly: one machine runs the model at all and runs it slowly, the other runs it quickly and needs two cards to hold it. Which is correct depends entirely on whether 11 tokens per second is usable for the task, and for interactive single-user work it generally is, while for serving several concurrent users it is not.

Commonly confused with

This is a general array and training framework, not an inference engine, and the distinction matters when comparing it to the tools people usually set it against. A dedicated inference runtime does one thing — run a quantised model fast — and nothing else. This framework provides the array operations, automatic differentiation and training loop as well, which is why fine-tuning is possible within it and why benchmarks against an inference-only tool compare one of its capabilities rather than its purpose. Reports that it is faster or slower are specific to a model, a quantisation and a machine generation.

When to use it

Reach for it when:

• Local inference and fine-tuning on this hardware specifically, where the unified pool and native kernels are the advantage.
• Models in the range where capacity matters more than speed — sizes a discrete consumer card cannot hold at all.
• Low-rank fine-tuning on a single machine, where the memory available to the accelerator is what makes it feasible.
• Research and prototyping where the array API and automatic differentiation are needed alongside inference.

Reach for something else when:

• Serving concurrent users, where the bandwidth ceiling limits aggregate throughput regardless of capacity.
• Cross-platform deployment, since the framework targets one hardware family by design.
• Full pretraining or large fine-tunes, which need cluster-scale interconnect this hardware does not provide.
• Any comparison of frameworks without naming the model, quantisation and machine generation, since the ordering changes with all three.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Apple MLX — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path