Li Feifei: Functional Classification of World Models and Prospects for Spatial Intelligence

(Source: MIT Technology Review)

Of the three, she determined, the renderer is the most mature commercialized but has a ceiling (good looks doesn't mean physically correct), the planner is the most exciting but furthest from actual deployment (the gap between lab demonstration and actual use is still huge), and the simulator is a critical hub that has been seriously underestimated. Because the simulator works at the level of geometry, physics, and dynamics, it can not only project pixels upward for human consumption, but also derive action consequences downward for robots to use. Once you master simulation, you have the foundation for rendering and planning at the same time; not the other way around.

This post is, of course, a World Labs product declaration. Their Marble is already outputting both Gaussian spatter and collision meshes in an attempt to unify the renderer and simulator into a single model. The end story depicted at the end of the article is a unified world basic model that can freely switch between rendering, simulation, and planning according to downstream requirements. Whether this vision can be realized is another story, but as an analytical framework, the renderer/simulator/planner's rule of three may indeed help penetrate some of the noise of the current “world model” concept.

The full text is translated below.

“The world is the sum of everything that happened.”

—Wittgenstein, Philosophical Theory of Logic, 1921

The world is not made of words.

In an earlier article, we proposed that spatial intelligence is the next frontier of AI, and the world model is the path to it. Now, the World Labs team and I wanted to go one step further: Of the many things that are now called “world models,” which functional modules actually make up this capability? What are their respective uses?

Language models give machines strong control over concepts, vocabulary, and reasoning, but the physical world, whether virtual or real, operates on a completely different basis. The language model learns the statistical structure of text, and the world model learns the statistical structure of space and time: how light falls on a surface, what a garden looks like from an angle never captured by a camera, and how responsive objects are and follow the laws of physics.

This makes “world model” one of the most important and most misused terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI all claim to be modeling the world, but they each refer to very different things. A video model that can generate gorgeous but physically impossible flames, a language model that improvises playable games, and a physical engine that faithfully simulates the combustion process are all called by the same name.

The ancient Greeks were never able to agree on what constituted the world, whether it was fire, water, or indivisible atoms, because the “world” was never a single thing. It's always an alternative word used by a certain thinker to reason about some kind of generality. AI inherits the same problem, and it just happened at a time when accuracy was most needed in this field.

The closed loop behind taxonomy

To clear up this confusion, you can start with a picture that is older than all of the above techniques. All reinforcement learning materials, including the classic Sutton and Barto, have used variants of the same picture to describe how agents interact with the world for decades. The official name of this map is Partial Observable Markov Decision Process (POMDP), and the term “world model” was originally defined in this tradition.

An agent (which can be a human, robot, or software system) performs an action. These actions change the state of the world. However, the smart body can never directly see the state itself; what it receives is observation: photons falling on the retina, sensor readings, and pixels in video frames. New observations guide new actions, and the cycle goes back and forth.

The word “state” needs to be taken apart because its meaning shifts in different fields. What I'm talking about here is not the chemist's state; it's not the difference between a solid state, a liquid state, and a gas state. Here is the state of physicists and roboticists: a complete description of everything happening in the world at a given moment, including every object, every position, every speed, and every property. The state is the underlying reality of the world. In principle, it is complete, but it can never be directly observed by any intelligent entity in it. Observation is an agent's partial view of this reality. Actions are responses made by the intelligent body based on this.

This closed loop (smart body → action → state → observation → agent) is the structure that gives the term “world model” its technical meaning. The phrase itself is even older, and dates back to Kenneth Craik's proposal in 1943, where he believed that the mind makes inferences by running “small-scale models” of reality, and in the late 1980s and early 1990s, the concept was introduced into the field of neural networks. This closed loop also explains what people mean when they use the term today. The various things now known as world models are actually different projections of the same closed loop, and each output is a different component of the closed loop.

The three functions of the world model

The first world model is the renderer. The renderer outputs observations, specifically pixels for the human eye, and the most important quality indicator is visual fidelity. A video model that turns text prompts into cinematic aerial footage is a renderer; interactive systems like Google's Genie 3 or World Labs' own RTFM are also renderers, which generate images in real time based on user input. Such models do not have an explicit understanding of three-dimensional structures. It produces an image that the viewer sees, not what the thing itself looks like. The buildings in the aerial footage may be flawless from the sky, but try walking through the city below and they'll collapse.

The second type is an emulator.

The simulator outputs a state: a geometric, physical, or dynamically-faithful representation of the world, on which humans and computer programs can calculate and interact. The renderer's contract is purely visual, while the simulator's contract is structural. It requires geometry to stand up to test, physics to follow Newton's laws, and dynamic behavior to meet the expectations of the laws of physics. The emulator serves both types of users simultaneously. Professionals such as architects, designers, filmmakers, and game developers need accuracy beyond visual credibility. Computer programs such as reinforcement learning agents, robot controllers, and autonomous vehicles use simulators as training grounds, where they interact with the world on a large scale to test scenarios that are either dangerous, expensive, or simply impossible to execute in reality.

The third type is the planner.

The planner outputs actions. Given an observation and a goal, the planner answers the question: what should the agent do next. In many ways, the planner is the inverse process of the renderer. The renderer uses action as input and output observation, and the planner uses observation as input and output action, thus closing the perception-action circuit. Vision-language-motion models (VLA), model-based systems, and a new wave of world action models (World Action Models) are all different attempts by planners: to enable systems to decide what robots should do in an unstructured world.

The above three categories cover most of the work currently being implemented, and the distinction between them is very useful in practice. But these three categories are not fundamentally separate from each other. They share the same basic set of knowledge about how the world works: geometry, physics, dynamics. A model that can render a cup from any angle should, in principle, also simulate what happens when the cup is pushed, and plan to pick it up with one hand. More and more interesting research is deliberately blurring the boundaries between these three.

Why simulation is a key hub

Of the three categories, emulators received the least amount of public attention, yet were the most important of the three. This post wants to rectify this asymmetry.

The renderer is currently the most commercialized. Numerous image or text-to-video products are rapidly expanding in the consumer and business markets. Google's Nano Banana model puts renderer-level image generation capabilities into the hands of potentially hundreds of millions of users. The technology is real, and so is the market. However, the goal of renderer optimization is visual credibility rather than physical accuracy, and this ceiling is important. Their output is beautiful, but you can't use them to design a building or train a robot.

The planner is both the most exciting and the least mature, and it's closely related to the rapidly evolving field of robotics learning. Over the past two years, this field has produced quite a few robot demonstrations that look impressive in the video, but we need to be honest about what these demonstrations actually showed. Almost all of the demonstrations were confined to a highly restricted laboratory environment, with limited object types and very short task times. None have proven the complexity, variety, and length of time required for real-world deployments. From an amazing demo video to a robot that works reliably in a kitchen, warehouse, or operating room, the gap is still huge.

Despite this, the size of the commercial bet is still considerable. A wave of well-funded new entrants are scrambling to launch generic planning systems, while large infrastructure players are basing their planning capabilities on a broader simulation stack.

The simulation is a bridge connecting the two. If language is an abstraction of the world, and a pixel is a projection of the world, then geometry, physics, and dynamics are the world itself. The simulator must work at this level: it is a structural framework from which both visual performance (for use by the renderer) and behavioral consequences (for the planner) can be deduced.

A model that has mastered simulation can project its understanding into pixels for human consumption, as well as motion predictions for use by physical intelligence. And a model that only masters rendering or planning can't do either. The commercial space here is extremely large. NVIDIA's Omniverse alone has a target market size of over trillion dollars, which the company estimates, covering factories, warehouses, supply chains, and digital twins. Robotic training, autonomous driving testing, architectural visualization, engineering design, and drug discovery all rely on some form of simulation.

The most difficult open questions in this field are also concentrated here. 3D data with explicit geometry, material properties, and physical labeling is several orders of magnitude scarce than the internet video used for renderer training. The sim-to-real gap (the difference between object behavior in simulation and behavior in the real world) still exists. Generative simulators also introduce a new risk based on this: AI-generated geometry may look correct, but actually contain self-intersecting or misscale issues, causing physical simulations to produce ridiculous results. The computational cost of large-scale multiphysics simulations (rigid bodies, deformable objects, fluids, and cloth all interact simultaneously) is still several orders of magnitude higher than simulation in a single field.

At World Labs, Marble is our first step in this direction. It takes multi-modal input (text, image, video, or spatial sketch) to generate an explorable 3D environment while outputting Gaussian splats (Gaussian splats) for visual exploration and collision meshes for the physics engine to operate. But Marble is just the first chapter in a long arc. As the boundaries between rendering, simulation, and planning begin to melt, an entire field is writing the story.

The borders are melting, and what will happen next

Currently, the most important trend in this field is that the three categories are beginning to merge. The consensus behind it is that the knowledge required to render a world, simulate it, and act in it is largely the same. Using the previous example, a model that actually understands how the cup is placed on the table (its geometry, material properties, response to force, etc.) should be able to render the cup from any angle, simulate what happens when the cup is pushed, and plan to pick it up with one hand. The three categories are three projections of the same underlying understanding.

For example, there has been a small but growing amount of recent work from various robotics laboratories, which have shown a possibility, at least conceptually, that a pre-trained video renderer can act as a backbone network that combines world prediction and motion prediction, allowing a single model to simultaneously imagine “what will happen” and “what to do”, thus building a bridge between renderers and planners. World Labs' Marble can now simultaneously output Gaussian spills and collision meshes from a single model, breaking the boundary between renderers and simulators. Every level is moving from passive output to interactive systems: renderers become responsive to motion conditions, simulator-generated worlds become more controllable and editable, and planners begin to reason carefully rather than just react.

The logical end point is a unified world model: a basic model that can render photo-realistic views, generate physically accurate structures, plan action sequences, and switch between different output modes according to the needs of downstream users. We will still face a series of serious challenges. The data landscape is extremely unbalanced, with renderers hosting massive amounts of internet video, while simulators and planners face a serious lack of 3D assets and robot demonstration data. Optimization for visual aesthetics may sacrifice the accuracy required for robotics or high-fidelity simulations. Reconciling these tensions within a single architecture is a core open problem in today's world model research, and it is also something World Labs is committed to solving as it continues to evolve Marble.