BentoML — AI Glossary

Reference details

Related terms

• Ray Serve
• Triton Inference Server
• Serverless Inference

How it works

1. A service wraps one or more models behind declared endpoints, so serving concerns live in code next to the model rather than in deployment configuration.
2. Dependencies, model artefacts and the service definition are packaged together into a single versioned bundle.
3. That bundle builds into a container, so what runs in production is the artefact that was tested rather than a rebuilt approximation.
4. Adaptive batching groups concurrent requests at the serving layer, which raises throughput without the caller knowing.
5. Because the abstraction is framework-agnostic, the same packaging covers a classical model, a transformer, or a composed pipeline.

Example: What packaging the dependencies together prevents

A model works locally and fails in production on a tokeniser version difference — the classic reproducibility failure, and the one that costs an afternoon to find because the error surfaces as wrong output rather than an import error. Bundling the model artefact with its pinned dependencies makes the deployed unit the tested unit, so that class of difference cannot arise between environments. The cost is a build step; the alternative is debugging environment drift at the point of deployment.

Commonly confused with

Grouped with LLM inference engines, which solve a different problem. An engine like vLLM optimises transformer decoding — paged cache, continuous batching, optimised kernels — and is what you want behind a large language model. This is a packaging and serving framework: it wraps any model with endpoints, dependencies and scaling policy, and for an LLM it typically wraps an engine rather than replacing one. It is also not a model registry; it packages for deployment rather than governing an approval lifecycle.

When to use it

Reach for it when:

• Models of several kinds need consistent packaging and serving.
• The deployed artefact should contain its own pinned dependencies.
• Serving concerns — batching, endpoints, scaling — belong in code with the model.

Reach for something else when:

• Serving a large language model where a specialised inference engine is the core need.
• Only a hosted API is called, where there is nothing to package.
• An approval lifecycle is required, which is a registry concern.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to BentoML — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path