Li Feifei: Functional Classification of World Models and Prospects for Spatial Intelligence
Author: Ga Yang Original title: Li Feifei Latest Long Article: When video generation, robots, and NVIDIA all call themselves world models, we need a taxonomy “world model” which is probably the hottest and most confusing concept in the AI field since 2025. When Sora came out, OpenAI called it a world simulator; Genie lets you walk around in the generated images, also called a world model; the robotics company said it was making a world model; NVIDIA said Omniverse was the infrastructure for the world model, and even the game engine was pulled into this story. Everyone is using the same words, but they aren't saying the same thing at all. Today, Li Feifei published a new article on his personal Substack clarifying this concept. She first went back to the most classic diagram in the reinforcement learning textbook (POMDP closed loop: intelligence → action → state → observation → smart body), then pointed out that what is now called a “world model” is actually three different projections of this closed loop. The output pixel (observation) is the renderer, the output state is the emulator, and the output action is the planner. The classification criteria are very simple, depending on which part of the closed loop you are outputting. (Source: MIT Technology Review) Of the three, she determined that of the three, the renderer is the most commercialized but has a ceiling (good looking doesn't mean physically correct), the planner is the most exciting but furthest from actual deployment (the gap between lab demonstration and actual use is still huge), and the emulator is a critical hub that is seriously underestimated. Because the simulator works at the level of geometry, physics, and dynamics, it can not only project pixels upward for human consumption, but also derive action consequences downward for robots to use. Once you master simulation, you have the foundation for rendering and planning at the same time; not the other way around. This post is, of course, a World Labs product declaration. Their Marble is already outputting both Gaussian spatter and collision meshes in an attempt to unify the renderer and simulator into a single model. The end story depicted at the end of the article is a unified world basic model that can freely switch between rendering, simulation, and planning according to downstream requirements. Whether this vision can be realized is another story, but as an analytical framework, the renderer/simulator/planner's rule of three may indeed help penetrate some of the noise of the current “world model” concept. The full text is translated below. “The world is the sum of everything that happened.” ——Wittgenstein, “Philosophy of Logic,” 1921 The world is not composed of words. In an earlier article, we proposed that spatial intelligence is the next frontier of AI, and the world model is the path to it. Now, the World Labs team and I wanted to go one step further: Of the many things that are now called “world models,” which functional modules actually make up this capability? What are their respective uses? Language models give machines strong control over concepts, vocabulary, and reasoning, but the physical world, whether virtual or real, operates on a completely different basis. The language model learns the statistical structure of text, and the world model learns the statistical structure of space and time: how light falls on a surface, what a garden looks like from an angle never captured by a camera, and how responsive objects are and follow the laws of physics. This makes “world model” one of the most important and most misused terms in AI today. Computer vision, robotics, reinforcement learning, and generative AI all claim to be modeling the world, but they each refer to very different things. A video model that can generate gorgeous but physically impossible flames, a language model that improvises playable games, and a physical engine that faithfully simulates the combustion process are all called by the same name. The ancient Greeks were never able to agree on what constituted the world, whether it was fire, water, or inseparable atoms, because the “world” was never a single thing. It's always an alternative word used by a certain thinker to reason about a certain generality. AI inherits the same problem, and it just happened at a time when accuracy was most needed in this field. To clear this confusion behind the closed loop of taxonomy, we can start with a map that is older than all of the techniques described above. All reinforcement learning materials, including the classic Sutton and Barto, have used variants of the same picture to describe how agents interact with the world for decades. The official name of this map is Partial Observable Markov Decision Process (POMDP...
