LM Studio — AI Glossary

Reference details

Related terms

• Ollama
• llama.cpp
• OpenAI-Compatible API

How it works

1. Models are downloaded in a quantised, self-describing container format whose file embeds the architecture, the tokenizer and the quantisation scheme, so the runtime needs no separate configuration.
2. A quantisation variant is chosen at download time, trading size against quality. This is the one decision that determines whether the model runs at all on the machine.
3. Weights are memory-mapped and layers are assigned between accelerator and CPU. A model larger than available accelerator memory can still run with some layers on CPU, at a throughput cost proportional to how many.
4. Generation speed is bounded by memory bandwidth, not arithmetic: each token requires reading the whole weight set, so the ceiling is bandwidth divided by model size and no amount of spare compute raises it.
5. An OpenAI-compatible local server exposes the loaded model on localhost, which is what lets existing client code point at a local model by changing a base URL.
6. Context length is configured per load and consumes memory on top of the weights, so a long context can be the thing that fails rather than the model size.

Example: The one formula that predicts local generation speed

Generation is bandwidth-bound: producing one token requires reading every weight once, so tokens per second is at most memory bandwidth divided by model size. On a machine with 100 GB/s of memory bandwidth running a 7B model quantised to roughly 4.4GB, the ceiling is 100 / 4.4 = 22.7 tokens per second. The same machine with a 13B model at about 7.9GB gives 100 / 7.9 = 12.7, and a 3B model at 2GB gives 100 / 2 = 50. The relationship is exactly inverse in model size, which is why the practical question on a laptop is always which size fits the speed you need rather than which model is best. It also explains why machines with unified memory outperform their specifications suggest: a device with 400 GB/s running the same 4.4GB model has a ceiling of 400 / 4.4 = 90.9 tokens per second, four times faster with no more arithmetic capability. And it sets the floor on expectations for CPU offload — once layers move to system memory at a fraction of the bandwidth, the achievable rate follows the slowest path the weights travel.

Commonly confused with

LM Studio is a desktop application wrapping an inference engine, not an engine itself, and the distinction matters when diagnosing performance. The engine beneath it determines speed, quantisation support and hardware acceleration; the application provides model discovery, a chat interface and the local server. So a throughput problem is an engine-and-hardware question, while a model-not-loading problem is usually the application's configuration. The related confusion is with hosted inference: the local server speaks the same API shape as a cloud provider, which makes client code portable and does not make the models or the throughput comparable.

When to use it

Reach for it when:

• Local experimentation where data must not leave the machine, which is the strongest reason to run locally at all.
• Evaluating quantisation trade-offs by hand, where downloading several variants and comparing is faster than reasoning about it.
• Offline or intermittent-connectivity work, where an API is not available.
• Pointing existing OpenAI-compatible client code at a local model by changing a base URL, for development against a free endpoint.

Reach for something else when:

• Serving other users or applications in production, which needs a real serving stack rather than a desktop application.
• Frontier capability, since the models that fit on consumer hardware are not the largest available.
• Any workload where throughput matters and the machine's memory bandwidth sets a ceiling below your requirement.
• Long-context work on a memory-constrained machine, where the context allocation rather than the weights is what fails.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LM Studio — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path