Edge AI — AI Glossary

Reference details

Related terms

• Quantization
• Model Distillation
• GGUF
• Ollama

How it works

1. The model is converted from its training format into the device runtime's own format, which fixes the architecture and the precision at conversion time rather than at load time.
2. Quantisation is chosen to match what the device's accelerator supports in hardware, because a precision the silicon cannot execute natively is emulated and loses the speed the conversion was for.
3. The runtime dispatches to whichever units the device exposes — a neural accelerator, the integrated GPU, or the CPU — and a layer the accelerator cannot handle falls back silently to a slower path.
4. Memory is shared with the operating system rather than dedicated, so the usable budget is what is free at that moment and a model that loads on an idle device can fail to load on a busy one.
5. Sustained throughput sits below the burst figure because the device throttles on temperature, which means a benchmark measuring the first few seconds overstates what a long generation will do.
6. Updates are the part with no server-side equivalent: a new model version has to be distributed to every device, so a fleet runs several versions at once and evaluation has to account for that.

Example: What you trade for privacy and latency

On-device, the audio never leaves the phone and there is no network round trip. The cost is capability: you are running a few-billion-parameter quantised model against a frontier model in the cloud. Edge wins where the data must not move or the latency budget is tens of milliseconds — not where the task is hard.

Commonly confused with

Edge AI is not the same as a small model. Edge is where inference runs; a small model is what runs there. You can serve a small model in the cloud, and a well-provisioned edge device can run a surprisingly large one. The constraint is the device, not the label.

When to use it

Reach for it when:

• Data must not leave the device for legal or contractual reasons
• The network is unreliable or absent
• Per-request cloud cost at your volume is prohibitive

Reach for something else when:

• The task genuinely needs frontier capability
• You cannot update models across a fleet you do not control
• Device thermal and battery limits would make it unusable in practice

Referenced by

This term is used by the following ModelRefs references:

• on device stack
• edge runtime stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Edge AI — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path