What Are World Models? The AI Approach Beyond Language Models

New to the agent side of this? See what an AI agent is and what tool calling is for the pieces a world model would eventually plug into.

What is a world model?

The term comes from a 2018 paper by David Ha and Jürgen Schmidhuber titled simply World Models. It framed an agent as three parts. A vision component compresses raw observations into a compact representation, and a memory component tracks how that representation changes over time. A controller then chooses actions based on what the memory predicts will happen next.

That framing still holds up. In plain terms, a world model is trained to answer one question: given what I am seeing now, and given an action I might take, what would I see next?

It is a simulator the system builds for itself, rather than one a human programs by hand. That makes it distinct from a traditional physics engine, which is hand-coded rather than learned, and from a large language model, which predicts tokens rather than physical state.

World models vs. large language models

This is a real architectural disagreement, not a marketing frame. Yann LeCun, formerly Meta's chief AI scientist, has argued for years that language-only training has a structural ceiling.

Text describes the world, but a great deal of physical competence is rarely written down. How objects move, what happens when you push something, basic spatial reasoning: people do not narrate any of it. A model trained only on text has no direct exposure to that information.

	Large language models	World models
Trained on	Text, increasingly images and audio	Primarily video and sensor or action data
Predicts	The next token in a sequence	The next state of the world, given an action
Output space	Language	Representations of physical state, sometimes rendered as video or 3D
Strong at	Reasoning in language, broad knowledge from text	Physical plausibility, short-horizon action planning
Weak at	Physical consistency, spatial and causal reasoning not described in text	Language reasoning, abstract and symbolic tasks

The two approaches are not mutually exclusive. Meta's own V-JEPA 2 work pairs its video-trained encoder with a language model for some benchmarks. Most researchers on both sides describe these as complementary architectures rather than a replacement fight, even where the public framing is sharper than that.

How a world model actually works

Most explainers stop at the phrase it predicts video. The part that makes a world model useful is the planning loop built on top of that prediction, and it has four steps.

• Encode. Raw video or sensor input is compressed into a representation rather than kept as pixels. V-JEPA 2's encoder outputs embeddings that capture the state of a scene and discard irrelevant detail.
• Predict. Given the current representation and a candidate action, a predictor estimates the representation of the resulting state. V-JEPA 2's predictor outputs predicted embeddings, so prediction happens in that compressed space. It is cheaper and more robust to predict that the cup is now closer to the edge than to predict every pixel of a frame.
• Score. Several candidate actions are predicted forward, and each resulting state is scored against a goal. That goal might be an image showing where an object should end up.
• Act. The best-scoring action is taken, the world updates, and the loop repeats. Meta describes exactly this: at each time step the robot re-plans and executes the top-rated next action via model-predictive control.

That loop is what zero-shot planning means in practice. The robot was never trained on the specific task, yet it can still estimate the consequences of candidate actions well enough to pick a reasonable one.

Choosing between those candidate actions is the other half of the problem, and it has a field of its own: see what reinforcement learning is for how an agent learns which action is worth taking.

What is real right now: V-JEPA 2, Genie 3 and Cosmos

Three organisations are building world models for three different jobs. None of this is vaporware, since all three have published technical detail rather than only announcements.

Meta's V-JEPA 2 is a 1.2-billion-parameter model built on the Joint Embedding Predictive Architecture. It trains in two stages. The first uses more than 1 million hours of video and 1 million images with no human labels, and the second uses a much smaller set of robot video paired with the control actions executed.

Meta reports that the second stage needs about 62 hours of robot data to produce a model usable for planning and control. Deployed on real robots for pick-and-place with new objects in unseen environments, it achieved 65 to 80 percent success.

Goals are specified as images: short-horizon tasks use a single goal image, while longer pick-and-place tasks are given a sequence of visual subgoals to reach in order. Meta has released the code and checkpoints for commercial and research use.

Google DeepMind's Genie 3 targets a different problem: generating a playable, interactive environment from a text prompt. DeepMind states it generates worlds navigable in real time at 24 frames per second at 720p.

Those worlds stay largely consistent for several minutes, with visual memory reaching about one minute back. It is a limited research preview available to a small cohort of academics and creators, with a constrained action space and no solution yet for multiple independent agents interacting in one generated world.

NVIDIA's Cosmos targets a third job. It generates synthetic, physics-aware video to train robots and autonomous vehicles where real data is too expensive or dangerous to collect.

NVIDIA has released it under the NVIDIA Open Model License, which it describes as a permissive open model licence allowing commercial use. The diffusion and autoregressive models run between 4 and 14 billion parameters across three tiers. Nano targets real-time and edge use, Super is the performant baseline, and Ultra is for maximum fidelity and for distilling custom models.

Developers fine-tune on their own footage with NVIDIA's NeMo framework. Named users include Hillbot, generating 3D environments for robot training data, and Waabi, which NVIDIA describes as evaluating Cosmos for curating autonomous-vehicle data and simulation.

Two further names appear constantly in coverage of this space. The first is Yann LeCun's AMI Labs. LeCun left Meta in November 2025 after twelve years, having founded its FAIR lab. He is AMI Labs' executive chairman rather than its chief executive, a role held by Alexandre LeBrun, and the company is pursuing the JEPA approach LeCun first proposed in 2022.

The second is Fei-Fei Li's World Labs, which shipped a commercial product called Marble in November 2025. Marble generates persistent, downloadable 3D environments from text, images or video.

Both have raised substantial outside funding, World Labs through 2025 and AMI Labs in a large seed round reported in 2026. We are not attaching a precise valuation or round figure to either without a primary disclosure to point at. Treat funding figures you see elsewhere as a sign of investor conviction, not technical evidence. As of this writing AMI Labs has announced no public product.

What is proven vs. what is still promised

This is the part most coverage skips, and it is the part that matters if you are deciding whether to pay attention to this category yet.

Proven, with published numbers. V-JEPA 2's 65 to 80 percent pick-and-place success on unseen objects and environments is a reported result, not a projection. Genie 3's frame rate, resolution and several-minutes consistency are DeepMind's own stated specifications for a shipping, if limited-access, preview. The Cosmos open-weight release and its NeMo fine-tuning path are usable today.

Explicitly not proven, by the labs' own admission. Meta introduced three new benchmarks alongside V-JEPA 2: IntPhys 2, Minimal Video Pairs and CausalVQA. It did so because current models, V-JEPA 2 included, score well below human accuracy on physical plausibility and counterfactual reasoning.

Meta puts human performance on those tests at 85 to 95 percent. It states plainly that there is a notable gap between human performance and that of top models including V-JEPA 2.

DeepMind states that Genie 3 cannot reliably model multiple independent agents, and that it cannot simulate real-world locations with perfect geographic accuracy.

Nobody in this space claims robust long-horizon planning. Meta states that V-JEPA 2 currently learns and predicts at a single time scale, and names hierarchical models planning across multiple temporal and spatial scales as future work.

The honest summary is narrow. Short-horizon, narrow-task physical prediction works today, on real robots and with real numbers, while general physical common sense does not. That second capability is the one that would let a model reason reliably about novel situations, and the labs building these systems say they are not there yet.

Why this matters if you are building with AI

If you are building agents, APIs or products today, world models are not yet something you reach for the way you reach for a language model API. There is no world-model endpoint with the maturity of a chat completions call. Here is what is worth tracking.

• Robotics and physical automation. If your roadmap touches the physical world, V-JEPA 2's approach is the most concretely useful result in this space today. It learns from video and deploys with little task-specific data.
• Synthetic training data. Cosmos is usable now for generating physics-plausible synthetic video to supplement data collection. That is a practical near-term use even for teams not building a world model themselves.
• Simulation and testing environments. Genie 3-style generation points toward on-demand test environments for agents. The few-minutes consistency window and preview access mean it is not a production tool yet.
• A reason to keep agents in perspective. The limitations these labs publish are a useful corrective if you have assumed an AI agent understands the physical consequences of what it does. Mostly it does not, and the people building the systems meant to fix that say they are not there yet either.

For the adjacent architectural question of how much context to give a model, and how, see the guide to fine-tuning vs. RAG vs. long context.

Sources

Ha and Schmidhuber, World Models (2018). Origin of the term and the vision, memory and controller framing, accessed 17 Oct 2026.

Meta AI, V-JEPA 2: a world model trained primarily on video. Architecture, training-data scale, robot success rates, the three new benchmarks, model-predictive control, the single-time-scale limitation, and Meta's own stated gap to human performance, accessed 17 Oct 2026.

Google DeepMind, Genie 3: A new frontier for world models. Frame rate, resolution, consistency window, preview status, and DeepMind's own stated limitations, accessed 17 Oct 2026.

NVIDIA, NVIDIA Makes Cosmos World Foundation Models Openly Available. Purpose, licence, model sizes, tier names, fine-tuning path, and named users, accessed 17 Oct 2026.

Understanding World or Predicting Future? A Comprehensive Survey of World Models. Cross-check for general terminology and category framing, accessed 17 Oct 2026.

Methodology: every specific figure here is drawn from the building lab's own announcement rather than secondary coverage. That covers parameter counts, success rates, frame rates, resolution, data scale and licence terms, all re-verified against those pages on 18 Oct 2026. Details about AMI Labs and World Labs, including LeCun's role and the timing of funding rounds, come from press reporting. They are described without precise valuation or round figures, which we could not verify against a primary disclosure.