Ollama — AI Glossary

Reference details

Related terms

• GGUF
• Open Weights
• Inference Endpoint

How it works

1. A model entry is described by a manifest naming a base weight file plus a prompt template, a system message, and default sampling parameters — so the same weights can be registered twice with different behaviour.
2. Pulling a model fetches content-addressed layers and caches them by digest, which is why two entries sharing a base download it once.
3. The server loads the weight file through an embedded inference engine and memory-maps it, so start-up is dominated by page-in rather than by reading the whole file.
4. The context length the server allocates is its own parameter, set at load time and independent of what the weights support — this is the configuration that silently truncates long prompts when it is lower than expected.
5. Models are unloaded after an idle period and reloaded on the next request, which is why the first call after a pause is slow and subsequent ones are not.
6. The HTTP surface mirrors a chat completions API, so the same client code reaches a local model and a hosted one — while the template and sampling defaults underneath are the runtime's rather than the provider's, which is where behaviour diverges.

Example: Check the configured context, not the model's maximum

A model card advertising a very long context does not mean your local server is using it. The runtime has its own context setting, and if it is lower than the model's maximum, prompts are quietly truncated to fit — no error, no warning, just a model that appears to forget the start of a long document. This is the most common local-inference confusion, and it looks like a quality problem rather than a configuration one. Before concluding a local model is weak at long inputs, print the context length the server actually loaded it with.

Commonly confused with

Ollama is a local runtime and model manager, not a model and not a serving stack for production. It wraps a CPU/GPU inference engine and exposes a convenient CLI and an OpenAI-shaped API. For concurrent multi-user serving, engines built for batching and paged attention are the appropriate comparison; Ollama optimises for a single developer's machine.

When to use it

Reach for it when:

• Local development and prototyping against open-weights models without an API key
• Data that must not leave the machine, where any hosted call is disqualifying
• Quickly comparing quantizations of a model on your own hardware

Reach for something else when:

• Production multi-user serving, where a batching inference server is the right tool
• Benchmarking throughput and citing it as the model's performance
• Assuming default context, quantization and sampling settings match the hosted version of the same model

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Ollama — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path