Ray Serve — AI Glossary

Reference details

Related terms

• vLLM
• BentoML
• Model Routing

How it works

1. A deployment is a Python class annotated as a Ray actor. Ray places replicas of it across the cluster's nodes, and the number of replicas is the scaling unit.
2. An HTTP proxy runs on each node and routes an incoming request to a replica, so there is no single ingress bottleneck as the cluster grows.
3. Requests are batched dynamically: the proxy holds arriving requests for a configured window or until a maximum batch size, then hands the batch to one replica call. The window trades tail latency for throughput directly.
4. Fractional GPU allocation lets several deployments share one device by declaring resource fractions, which is what makes multi-model serving on a single GPU possible rather than merely desirable.
5. Deployments compose into a graph: one deployment's handle is called from another, so a retrieval step, a reranker and a generator are separate scaling units in one application and each scales on its own load.
6. For LLM workloads Ray Serve typically fronts an inference engine rather than running the model loop itself, delegating continuous batching and paged attention to that engine while it handles routing, autoscaling and composition.

Example: Where the composition pays for itself

A RAG endpoint has three stages: an embedding call at 10ms, a reranker at 40ms and a generator at 800ms. Deployed as one process, scaling for generator load scales all three, and every replica carries all three models' memory. As separate deployments, replica counts follow the actual bottleneck. At 50 requests per second, concurrency needed is load x latency: the generator needs 50 x 0.8 = 40 concurrent slots, the reranker 50 x 0.04 = 2, and the embedder 50 x 0.01 = 0.5, so one replica each covers the two cheap stages. Combined, the monolith needs 40 replicas each holding all three models; split, it needs 40 generator replicas plus 2 small ones. If the reranker occupies 2GB, that is 40 x 2 = 80GB of duplicated weights the split deployment does not pay for, against 2 x 2 = 4GB that it does.

Commonly confused with

Ray Serve is not an inference engine, and this is the most common misreading of its role. vLLM, TGI and TensorRT-LLM implement the token loop — paged attention, continuous batching, kernel selection. Ray Serve implements the serving layer around it — HTTP, routing, autoscaling, multi-model composition, canary traffic. They are complements, and the usual production shape is Ray Serve deployments wrapping vLLM engines. Comparing the two as alternatives leads teams to expect throughput improvements from Ray Serve that only an engine change can deliver.

When to use it

Reach for it when:

• Multi-stage pipelines where stages have different compute profiles and should scale independently.
• Multi-model serving on shared GPUs, where fractional allocation avoids dedicating a device per model.
• Clusters that already run Ray for training or batch work, so scheduling and observability are shared.
• Traffic splitting for canary and A/B evaluation of model versions behind one endpoint.

Reach for something else when:

• One model, one container, fixed traffic. A plain engine behind a load balancer is less to operate and fails in fewer ways.
• Teams with no Ray experience and no other Ray workload: the cluster is a second distributed system to debug alongside your application.
• Strict hard-real-time latency budgets, where dynamic batching windows and actor hops add variance you cannot remove.
• Serverless-only environments. Ray Serve expects a long-lived cluster, which is the opposite of a scale-to-zero cost model.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Ray Serve — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path