Modal — AI Glossary

Reference details

Related terms

• Replicate
• RunPod
• Serverless Inference

How it works

1. Python functions are decorated and the platform builds, ships and runs them in containers, so there is no Dockerfile or cluster to manage.
2. The container image is defined in the same Python code as the function, which keeps the environment and the logic versioned together.
3. Cold starts are reduced by caching prepared container snapshots, which matters because loading model weights, not starting Python, is the slow part.
4. Persistent volumes hold weights and datasets between invocations, so a large download happens once rather than per request.
5. Scaling is per invocation and billing is per second of execution, which suits bursty work and penalises idling.

Example: Where serverless GPU economics turn over

A dedicated accelerator at a notional £2 per hour costs about £1,440 a month whether used or not. The same work serverless at £0.000X per second is cheaper precisely while utilisation is low: at 2 hours of actual compute a day, that is roughly 60 hours a month against 720 — about 8% utilisation, so a reserved instance costs roughly twelve times the compute consumed. Above roughly 50-60% utilisation the reservation wins. The crossover is the whole decision, and it moves with cold-start overhead.

Commonly confused with

General serverless compute for Python, not an LLM API — which is the distinction against Together, Fireworks or Bedrock. You bring the code and the model and it supplies autoscaling GPUs, so it is the right tool for custom inference, batch jobs and training, and the wrong comparison for a hosted endpoint you call with a model name. Compare it with RunPod and other GPU platforms, not with model providers.

When to use it

Reach for it when:

• Bursty or scheduled GPU work — batch inference, fine-tuning jobs, evaluation runs.
• You want to run arbitrary Python on accelerators without operating infrastructure.
• Utilisation is low enough that per-second billing beats a reservation.

Reach for something else when:

• You want a hosted model endpoint rather than a compute platform — that is an inference provider.
• Utilisation is high and sustained, where reserved capacity is cheaper.
• Cold-start latency is unacceptable on the critical path.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Modal — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path