By Christian Rupprecht, Chief Research Officer, Third Dimension AI
World models have become one of the most discussed (and invested in) technologies in AI in recent years, yet the term means something different to almost everyone. Depending on who you ask, it might mean a video game generator, a physics engine, a planning module for robotics, or something else. In this blog post, we will take a closer look at what world models mean for physical AI, and how that shapes the way Third Dimension AI is building its own world model.
Defining World Models
The term "world model" used to be a precise scientific term for a model that maps states plus actions to future states. Naturally, researchers tend to get a bit grumpy when scientific terms are being used broadly and without following the original meaning. But this is the small price we need to pay to have real-world impact. So rather than dwelling on the precise meaning, we should look at why we are excited about world models to understand what the term tries to convey.
We know that training models requires a lot of data. But not only that, we also know that the distribution of the data matters: we want to cover as many scenarios as possible and ideally roughly even out the occurrence. This is because deep learning has always been orders of magnitude better at interpolating between things it has seen during training than at extrapolating outside the training data.
So over the years, the field has gone through different stages of sourcing training data. Collecting real-world data and curating/annotating works very well but is expensive and time-consuming, thus limited in scale. Synthetic data via video-game-style simulations allows us to model rare situations and give much finer control for interventions (for instance, what to do when faced with ten or even a hundred different road signs at an intersection). But creating these scenarios requires an expert to model the scene, the objects, and the behaviour of all participants in the scene, which is again costly and does not scale well. Moreover, with synthetic data of this kind, there is always a gap between it and the real world in terms of realism and appearance. It is thus very difficult to verify if an agent that behaves a certain way in the simulation would behave the same way in the real world.
The term world model is now being used to describe a new way of generating data: we learn from real data how the real world behaves, and we can then prompt the model to generate scenarios in which the agent can act and the world reacts accordingly.
This is appealing in two ways:
- It can be trained on real data and thus does not depend on manually annotated or synthetically created data, producing real-looking results as input to the downstream agent.
- We can make these models conditional on some form of control: we might, for example, describe the scene in text or give a rough layout as input. This allows us to easily generate new scenarios and to deal with the “what if” interventions discussed before.
This is the promise of world models:
A steerable, realistic data generator and simulator that can be used to train and evaluate downstream models.
It moves data from a passive role into an active role: we can dynamically generate the most useful and balanced training data for our agent, and we can then probe the agent with hypotheticals and analyze its behaviour.
Challenges with World Modeling
Of course, there are several problems we need to work out along the way. Most importantly, there is almost a chicken-and-egg problem here. Our hope is that the agent can generate diverse and balanced data, making it easier to train the downstream task agent (since interpolation is better than extrapolation). However, our world model has the same restriction: it needs balanced and diverse data to train itself. So if we had that, could we just use it to train the agent directly?
There are two aspects that, to some extent, alleviate this concern. World models accept actions as input and then simulate how the world behaves. This is a clear advantage over static data, as the agent can explore what different actions do during training, rather than being forced into an exact replay of the original data. This means that even if generalization in a world model is still challenging, it turns static data into interactable data, which is a large advantage for model training.
Second, world models are controllable probabilistic generators. This means that we can sample more data than we need and perform sanity checks and filtering to deal with situations where the generalization does not work well. Often, generative artefacts are less problematic than they initially seem. If my world model occasionally generates a person with 11 fingers, this only means that my robot will also work when it sees someone with an unusual number of fingers. More scientifically: as long as the real world is a subset of what we are sampling from the model, things should work out fine.
World Models in Applications
World models are reshaping how systems render, simulate and even decide what to do next. In gaming, they can be used to generate playable environments and predict how a scene should evolve based on player actions, reducing the need to hand-author the world and every interaction. In media and entertainment, world models can power faster and more cost-effective video generation and visual effects. In robotics, world models can construct a photorealistic world and let an agent simulate the likely outcome of an action before it’s executed on real hardware, helping teams better validate and scale their operations. For something like architectural and city planning, world models can support the simulation of how spaces will be used, how pedestrians and traffic might move through a design, and how a structure will perform before it’s built.
In addition to the various real-world applications of world models, there are also different underlying technologies for building world models: 3D vs. 2D, video models vs geometry, etc., which we will explore in another blog post in the future. At Third Dimension, we see the term world model not as a precise scientific definition, but as a description of a capability: data as an active controllable component for physical AI.
This matters because we believe data is the bottleneck now holding physical AI back. Real-world collection doesn’t scale, as it’s slow, expensive and most of the time you don’t encounter the unusual or dangerous scenarios you most need to test against. Synthetic data scales, but the gap between simulated and real means you’re never quite sure an agent’s behavior in simulation will hold up on real hardware.
World models are our answer to this data problem: we start from high-fidelity, multimodal reconstruction of the real world, then use it as a foundation for unlimited, programmable generation. Rather than relying on traditional synthetic data or real-world collection, we ground our reconstruction in reality and produce new variations, scenarios and worlds on demand to enable validation and training at scale for autonomous robots.
