ModelRefs / Groq — AI Glossary

Groq — AI Glossary

A silicon company building the LPU (Language Processing Unit), an ASIC delivering extremely low-latency LLM token generation.

Overview

Groq's LPU achieves 300–500 tokens/second per user for 70B models—10–20× faster than GPU-based inference—by using a deterministic dataflow architecture that eliminates memory bandwidth bottlenecks. GroqCloud offers API access to LLaMA 3, Mixtral, and Gemma at sub-50ms TTFT. Targets latency-critical real-time applications.

Reference details

Topic	infrastructure
Last reviewed	2026-06-24

How it works

The accelerator is a deterministic dataflow design: the compiler schedules every operation and data movement ahead of time, so there is no dynamic caching to miss.
Weights are held in large on-chip SRAM rather than streamed from external memory, which removes the memory-bandwidth wall that limits token-by-token decoding.
Because decode is bandwidth-bound rather than arithmetic-bound on conventional accelerators, moving weights on-chip is what produces the per-user speed rather than raw arithmetic throughput.
On-chip memory is small, so a large model is distributed across many chips and the deterministic schedule coordinates them.
The service exists to run that silicon, so the model catalogue is whatever has been compiled for it — a narrower selection than a GPU host offers.

Example: Why per-user speed and throughput are different products

Decoding one token requires reading every weight once. For a 70B model at 2 bytes per parameter that is about 140GB of reads per token. An accelerator with 2TB/s of external bandwidth therefore caps out near 2,000/140 ≈ 14 tokens per second for a single stream, however much arithmetic it has. Batching amortises those reads across many users, which raises aggregate throughput while leaving one user's rate roughly where it was. Holding the weights on-chip attacks the per-stream number directly, which is the metric batching cannot improve.

Commonly confused with

Unlike Together, Fireworks or Replicate, this is a chip company whose inference service exists to run its own silicon, which is why its distinguishing claim is token generation speed rather than model breadth. The comparison is therefore not like-for-like: the trade is exceptional decode throughput against a narrower catalogue of supported models. Do not confuse it with Grok, an unrelated model family.

When to use it

Reach for it when:

Single-stream token speed is the product — live voice, interactive agents, anything where a user waits on generation.
The model you need is in the compiled catalogue.
Low latency matters more than breadth of model choice.

Reach for something else when:

Aggregate throughput per pound is the goal, which batching on conventional accelerators addresses.
You need a model or a custom fine-tune outside the supported set.
Self-hosting the same weights later is a requirement.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Groq — AI Glossary.

Frequently asked questions

What is Groq?