This Week in Robotic Learning (TWIRL) #5: The General Purpose World Model Labs
World Labs, AMI, and Google DeepMind are all working on world models. What does it mean for robotics?
Warning: this post contains spoilers for Marty Supreme for some reason. You can avoid them pretty easily though.
World models, generally, are any digital environment in which visuals (or simulacra of visuals) are created. There’s a wide range of them that we’ll get into shortly, but the reason they matter for robotics is because they represent a simpler path to cheap and abundant data. Collecting sufficient data from the real world is a combination of impossible and expensive, while, theoretically, digital representations ought to be cheaper and more voluminous. Pretend, for example, you needed to teach your robot how to do some goofy TikTok trick shots. You could have some guys spend six hours trying to get it right, or, if you have an accurate physics engine, you can run it in simulation and transfer that knowledge into your robot’s brain.
World models are already a crucial part of both robotics’ present and future. It’s simply too expensive for a robotics company to rely solely on teleoperated data, and so there needs to be something digital to fill that gap. But what counts as a world model can be confusing. Here’s a quick breakdown:
Type 1: Simulators. These are hand-crafted, artisanally engineered environments, like a flight simulator. MuJoCo is a prominent example in robotics. It’s a matter of dispute if these are really world models since they’re not built via neural net predictions.
Type 2: Static scene representations. These are 3D pixelated worlds that a camera (but not an agent) can move through.Think of, say, Google Street View, but generated by a neural net rather than a pastiche of images.
Type 3: Generative Video. These are AI-generated non-interactive videos. Think of Sora 2 (RIP king, you’ll be survived by that video of a woman jumping through a glass bridge with a huge rock).
Type 4: Action-controlled generative world models. These are the same as type 3 except they are in fact interactive. These are useful for letting a robot or AI agent basically go into an AI-generated video and start doing things and dealing with the consequences thereof. NVIDIA’s Cosmos-Predict is an example, as is Bagel.
Type 5: Latent dynamics model. All the above world models rely in some respect on creating pixelated spaces to understand movements. These do not. Instead, they take the relevant details of a scene or movement and compress it into a “concept-space.” So, if you picture scoring a goal in a soccer game, you don’t need the visual information from every blade of grass. Meta/Yann LeCun’s JEPA family is an effort in this direction.
In robotics, simulators are the most widely deployed at this time because for the most part, the learned models just aren’t good enough to actually be useful.
There are going to be limits to how quickly and widely simulators can scale. The real world has so many edge cases and weird situations that something that is generative intuitively seems like a more flexible and valuable environment to train robots on how to take action.
When I look at some of the most well-funded world model builders, their statements lay out a very clear and inspiring argument that general purpose, AI-generated world models are not just a nice to have, but are a critical element in the journey to achieving superhuman intelligence.
The next generation of world models will enable machines to achieve spatial intelligence on an entirely new level—an achievement that will unlock essential capabilities still largely absent from today’s AI systems.
I speculate that common sense may emerge from learning world models that capture the self-consistency and mutual dependencies of observations in the world, allowing an agent to fill in missing information and detect violations of its world model.
Learned world models like Genie, and text-to-video generation models like Sora, demonstrate that foundation video models can be used as learned simulators, including in real-world settings like robotics and autonomous driving. If these works combine with learned multi-modal reward models, they could be used to generate an open-ended curriculum of tasks, scaling to task spaces far larger and more photorealistic than can currently be achieved. At sufficient scale, this may provide a path to generating AI agents with superhuman adaptability across a wide range of previously unseen tasks, which can be deployed in the real world across the rapidly closing Sim-to-Real gap.
This is, in some respects, self-evidently true. A genuinely perfect, comprehensive world model, delivered locally and instantaneously to the existing levels of digital intelligence we have already created would solve every material problem. It would decompose the actual physical world into semantically consistent bits of information upon which calculations can run and run and run until all the machines do what we want them to.
Consider the demos we tend to look at right now. Making a grilled cheese sandwich. Folding laundry. Making a bed. Plugging in a plug. What’s hard about these things? It’s not the capability of the machines. When given proper operating instructions and the right circumstances (lighting, placement, etc.), the machines can successfully do these things.
But the real world is made almost entirely of the wrong circumstances.
And this is where, as our imaginations get really excited, we must come back to reality (actual reality, not world model reality) in a somewhat depressing sort of way. The actual state of play is limited–general purpose world model labs are, by their own admission, quite far away from creating anything essential.
The general world model bet is dependent on the notion that there simply is not and never can be enough real-world or even video data to make general purpose intelligent robots work, and also that they’ll have many years to figure it out before then. It’s not a crazy bet at all, but it’s a pessimistic both because it implies that existing robotics-focused models and teleoperated data will be insufficient, and because none of these labs seem particularly close to landing the plane.
SPLAT
I’m going to say the phrase “Gaussian splat” as infrequently as possible in this section because it is gross and I don’t want to allow it to spend any more time in my mind than absolutely required. However, it is the fundamental approach underlying World Labs’ model. A Gaussian splat is a process for taking a visual scan and translating it into a 3d image rendering. These splats are described as ellipsoids, and if you pile a bunch of them together into a specific space, the output is an interactive 3d visual space. These are type 2 models–static scenes. I made one. Take a look if you wish!
Ah, well, it’s not exactly placing you in a whimsical, cartoonish robotic version of Paris the way I hoped. But then, I only supplied them with one image, used the cheapest model, and gave a pretty basic prompt. The preset ones are definitely more fleshed out.
This is a weightless world. You float around and through things. To get to the real world, the World Labs team will need physics.
They get this! CEO and AI pioneer Fei-Fei Li’s manifesto cites the “physical consistency” as among key principles of the company:
World models that unlock spatial understanding and reasoning must also generate simulated worlds of their own. They must be capable of spawning endlessly varied and diverse simulated worlds that follow semantic or perceptual instructions—while remaining geometrically, physically, and dynamically consistent—whether representing real or virtual spaces.
The real value of world models as a class of product is narrowing or even eliminating the sim-to-real gap that synthetic data creates. A future in which we have commoditized robotic machinery with robotics policies and world models layered on top is an appealing solution to the data gap. But let’s be clear: they’re at least a breakthrough away.
I don’t think even the World Labs team is totally clear, from a technical perspective, what their bet is exactly, and how they’ll layer in more real-world elements beyond just visuals. Because… visuals alone are still really hard!
They’ve publicly referenced distilling a physics engine into a neural network as a possibility, and outside research has made some progress in attaching physical properties to these primitives (by which I mean the splats, of course). If the existing approaches to physics incorporation were promising, there’d be more obvious public signs of that. For now, I’m assuming this is an open problem without a clear path toward a solution, or even, particularly encouraging progress yet.
I should concede here that maybe I’m being too robot-centric and World Labs is going to make a kajillion dollars from revolutionizing AutoCAD and video games (Roblox entered the “world models for video games” space this week). But I personally don’t really care all that much about that, so I’m allowed to be disappointed if that’s the outcome.
Mon ami
Having left Meta in a fit of pique (“you certainly don’t tell a researcher like me what to do!”) after being layered underneath a mulleted 28-year-old, Yann LeCun is now the Executive Chairman of AMI, whose approach is laid out in his 2022 manifesto on the topic.
As a fellow take-haver, I greatly respect LeCun’s consistent willingness to wade in and, in a direct, contrarian way that annoys people, fire off some takes with little regard for the outcome or hurt feelings (this was a lot of words to say “he’s French”).
LeCun’s manifesto–again, from 2022, the land before GPT–basically says LLMs (and token prediction) as well as pixel prediction is not the path to artificial intelligence. Instead, an artificially intelligent system would be organized into six modules: a configurator (basically: executive function), perception (interpreting the environment), cost (scoring and “feeling” rewards–think of human sensations like hunger), actor (the part that takes action), short-term memory, and, finally, a world model.
For all I know, the team at AMI is sitting around playing cards all day since there’s not really any external-facing evidence of their work. So it’s useful to look at LeCun’s work to guess at what they’re up to.
LeCun’s idealized world model–distinct from World Labs’–exists in latent space. That is, his team is directly, today (probably) working toward a type 5 solution. This is cool and unique! If I could make any of these approaches work tomorrow, it’d be this one. It’s the most recognizable as a truly economical “thinking” way of understanding the world.
The Gaussian splat pixel-based model, is a near-direct representation of the world. A wholly comprehensive overview of the world isn’t really a model, after all. It’s a reproduction. The latent space approach is a model. It’s taking in the context of images as they move, interpreting them holistically, and ta-da, you’ve built a Joint-Embedding Predictive Architecture (JEPA), which underlies the whole philosophy here.
Perhaps AMI (and World Labs) are right and their investors will be patient with years of burning money with very little practical to show for it. (As an aside, I kind of love their website, which has that sort of hedge fund-ish minimalist “it’s on a need to know basis and you don’t need to know” feel. But it certainly does not inspire confidence they’ll be shipping anything soon).
LeCun views AGI as decades or at least many years away, and believes that LLMs are a dead end down that road. I pretty much agree with this! But perhaps in the shorter-term, what we call “AGI” or, his preferred term, “AMI” or whatever doesn’t really matter, what matters is the outputs and the capabilities.
Whether or not they’re “generally intelligent,” LLMs are excellent at coding, and so they can be great businesses and are pushing progress forward across many domains. World models don’t do anything nearly that useful yet, so they’re not.
Google
Demis Hassabis’ team at Google DeepMind has excelled time and time again at these sorts of challenging research problems with–let’s be generous and say–ambiguous economic outcomes. They’re doing research, not running a business, and they’re really good at it. (And sometimes, they’re good at business too, but it almost feels incidental.)
Unlike AMI and World Labs, which are tightly aligned to the vision and work of their pioneer founders, Google DeepMind’s Genie project has more of a hybrid lineage. To the extent they have a manifesto (this is the last one of these, I swear) it’s Open-Endedness is Essential for Artificial Superhuman Intelligence (Hughes, et al, 2024) which lays out a framework for an self-improving artificial superintelligence.
The authors’ definition of open-endedness is: “From the perspective of an observer, a system is open-ended if and only if the sequence of artifacts it produces is both novel and learnable.”
“Novel” here means that relevant new information is being produced (i.e. TV static is technically producing new information moment-to-moment, but it’s functionally the same thing, and so, it’s not relevant).
Their Genie 3, released last August, is a real-time interactive type 4 general-purpose world model. Unlike World Labs’ Marble, it produces token-predicted video rather than splats. They profess to have some nifty emergent capabilities (object permanence! Some physics! Multi-view consistency!) that would point to pretty clear applications for robotics. Eventually. It does not appear that the Gemini Robotics team is making use of Genie yet–but Waymo is, which is very encouraging.
The primary resident of Genie 3 and the Genie-verse is DeepMind’s Scalable Instructable Multiworld Agent (SIMA). SIMA is, basically, a superpowered video game NPC; it’s an AI agent that can live in a world model.
Unlike World Labs, which has its worlds but no agents, and AMI, which (maybe?) has representations without environments, DeepMind has both approaches combined, under the same roof.
This makes a lot of sense: have parallel paths of improving world models and improving in-world agents. Having this all in house seems like a really helpful way to accelerate the feedback loop, so that hopefully we get something better than just more manifestos and superior NPCs in The Pitt: The Video Game.
Counterpoint: give it a little time and it’ll be amazing
In robotics, all your fancy math doesn’t mean anything unless you can actually apply it to a machine in the real world that actually does something useful.
On some level, what AMI and the Genie/SIMA teams are doing is not that different from what Physical Intelligence or Skild or Sunday or Generalist or Figure or Tesla (Optimus) any other is doing, except that those “robot-brain”-first companies are directly engaged with the real world.
To date, all of these robotics and world model companies combined have an amount of success that can be rounded to zero, relative to where they, their investors (and I!) expect this space to be in the future.
Intuitively, it’s hard to imagine a pathway to useful general real-world intelligence that does not run through the physical world rather than living exclusively in a simulated one; while I like DeepMind’s feedback loop between agent and world model, I wonder if that will tend to drift without a tether to the real world.
And to date, it should be concerning, but not dispositive, that higher levels of specificity are proving more valuable. Wayve, Tesla and Waymo have built driving worlds. Microsoft Muse made a video game. NVIDIA’s Cosmos/Gr00t combination is structurally similar to Genie/SIMA seeing much more adoption in robotics–probably because that’s what it’s built for.
The general models don’t yet have that hook that makes it obvious who should be buying and using them for what, and so they’re missing out on that customer feedback. These are huge swings with potentially massive payoffs and very little feedback mechanism between “nothing useful” and “critical part of the global economy.”
But maybe that’s fine!
No one gave OpenAI any customer feedback in 2019 and anyone looking from the outside would probably be baffled at what those directionless nerds were doing. Frontier research involves taking on unknowns. “I have no clue what these people are up to, and maybe they don’t either” can be a feature, not a bug. LeCun’s absolutely right–you don’t tell a researcher what to do. The fact pattern that emerges during the research should do that. I just hope it tells them how to make something useful in the real world.
Let’s look at some robots
There are two interesting categories to look at here; first, I’m intrigued by Eka Robotics, a Boston-based company that came out of stealth this week. Their emphasis is in part on dexterity, but really on judging the amount of force correctly. There isn’t a ton of discussion on this topic, but… a robot that can rely on touch as much as vision would certainly be more useful.
I don’t love to focus on robotics drama (alright, whatever, I do. Sorry for CARING), but… Figure and 1x seem to kind of… not like each other? Maybe there’s a little “narcissism of small differences” happening here as they’re both in the humanoid space, but this from 1x founder Bernt Bornich (“We’ll show our work. Others should too.”) seems to be an oblique shot at Figure who has emphasized their supply chain integration repeatedly, including in this interview/factory tour with Sourcery:
This comes out one day after 1x put out their very slick factory tour video.
Did they get wind of the Figure factory tour and rush to get their video out in front of it? Possibly. A little healthy competition never hurt anyone. Bornich claims to have received over 10,000 preorders and is rushing to meet demand, which would seem to indicate that there’s enough interest in imperfect or teleoperated robots to scale demand particularly for early movers.