This Week in Robotic Learning (TWIRL) #2: What Is Generalist Up To?
And how are they different from other labs?
Last week, I included Generalist AI’s demo of their new model, GEN-1, as a bit of a teaser at the end of the post. This week, I wanted to dive in more deeply, because there are some impressive claims in their initial blog post (“It improves average success rates to 99% on tasks where previous models achieve 64%, completes tasks roughly 3x faster than state of the art, and requires only 1 hour of robot data for each of these results.”) and some interesting ideas in their follow-up. Given that last week’s topic was world models, a post called “Going Beyond World Models & VLAs” feels like a natural continuation of the discussion.
So is this it? Is this the new state of the art????
Throughput on folding boxes; prior SOTA here is π0.6
The task above, is an effort to precisely replicate π0.6’s riveting, four hour-long video of its model folding boxes, which, if you’re into this kind of stuff, could easily replace either the 10 hour-long fireplace video or lofi beats to study to as your new video to run in the background as you work.
From the blog: “GEN-1 can assemble a box in 12.1 seconds – this is 2.8x faster than prior SOTA (GEN-0 and π0 both took roughly ~34 seconds on identical boxes).”
On the whole, the box folding task is the only 1:1 claim where it’s very clearly “another model attempted this precise task, so did we, and we were better.” There would appear to be a new SOTA for the specific task of folding boxes. We have a new robotic box folding champion!
The other tasks are certainly impressive, but the comparisons to competitors are a bit less directly analogous. In the laundry category, they folded 86 t-shirts in a row; π0.6 has demonstrated the ability to fold diverse items in an unseen home. Different things, but the PI model’s ability to handle novelty is probably the more impressive thing at the moment.
There’s also a vacuum servicing task, putting together a kit of auto parts, and a pick and place task with blocks. None of these are super complex, but the duration (hours!) and reliability the team published on these is notable. The breadth of tasks is where Generalist’s model stands out at this point. As far as I can tell, no other model has shown the range of control on simple, dextrous tasks that they have.
What is also notable is the caveat in the limitations, which is… “while we have shown several dexterous tasks at 99%+ success rates, not all tasks that we have attempted are able to hit these rates.” While I realize talking about the derivative of a robotics paper is perhaps a bit dry, I do wonder (Carrie Bradshaw voiceover)… were they at 99% success rates of getting to 99% success rates on tasks like these? Or more like 50% Or 20%?
Did they attempt more complex long-range tasks and come up way short? Or just below the 99% standard? I get that the point of this blog/release is to say “we’re going to figure out how to absolutely nail simpler, general tasks rather than figure out how to be pretty good at a bunch of more complex ones,” but I have to imagine they at least tried and am curious of the results.
So, all in all, it seems like Generalist is at either the frontier or a frontier, depending on your tastes. I think we’d all like to see general purpose robotic AI models get to the point where they can handle complex long-range tasks with a very high degree of reliability. The question is whether the path there is through slowly improving at those long-range tasks or mastering short-term tasks. Generalist is approaching from the latter end of the problem.
What are they doing differently?
Regardless of your taste, it’s worth noting that Generalist has a pretty different approach from all the frontier labs (except one!).
Physical Intelligence represents the easiest comparison–not because they’re similar in their approach, but because they’ve been the most prolific in documenting their methods, and are similarly pursuing a general purpose robotics model.
Model base: pre-trained VLA v. built from scratch
This is the key distinction between these two companies. Physical Intelligence’s choices stem from a philosophical view that, right now, the VLA lineage is, in fact, working and the best path forward, and that web-based data is in fact, super valuable and scales way better, so architecting around that makes more sense. Generalist’s conviction is that new models specifically for robots, built from scratch, are the cleanest, simplest way to move forward. It’s a classic “kludge” v. “tear it down and start over” situation.
Physical Intelligence starts with a pre-trained VLM from Google’s Gemma / PaliGemma family of open-weight vision-language models. In other words, it has some knowledge baked in. Generalist differs pretty strongly here, building their weights and rolling out their base model without any pre-training (before… the pre-training that they do with the wearables). Their POV on this is basically: once you have enough high quality data, (possibly wrong) preconceptions derived from existing models’ pretraining is actively counterproductive. It’s basically saying that VLA’s are kind of like that guy who’s only seen Boss Baby.
It’s kind of a big deal that they’re abandoning the VLA structure since Generalist CEO Pete Florence is among the inventors of the VLA (though so are many of PI’s co-founders).
Data collection: teleoperated v. wearables
The next key distinction between Generalist and other robotics AI labs is that they are relying on first-party sensor data collected from humans rather than, if we take PI as our counterpoint, a combination of first-party teleoperated data and, third-party videos of robotics performing tasks.
As an aside, Sunday Robotics is doing something similar to Generalist. They also rely on wearables rather than teleoperated data. The key distinction between Generalist and Sunday is that Generalist is going for a general purpose foundation model, while Sunday has built and integrated their model with their own hardware in mind.
Action system: bolted-on v. integrated
This is downstream of the last item; if you have a VLM to begin with, you’re starting from a base that doesn’t actually have a process for performing actions. So you have to build it. PI has two different solutions to this.
In one, they use flow matching. Flow matching is an approach designed to allow the model to understand the speed and direction at which it needs to move (smoothly! With some flow!) rather than sort of stuttering along in the right direction until it gets there.
PI has also developed their own FAST+ (Frequency-space Action Sequence Tokenization) approach that chops up movement into more modular chunks of information. They do this with DCT (Discrete Cosine Transform), a long-established technique for rewriting a smooth signal as a sum of simple wavy patterns (it’s how we getP3 and JPEG files!), and Byte-Pair Encoding (BPE) which handles the chunking element of this approach.
Generalist, by contrast, does not have the “action head” element at all; it’s baked into the model. This can be a good thing (it allows for architectural coherence, as well as the ability to more easily shift between perception and action) or a bad thing (changes to this element need to occur at the model level rather than something more modular).
Complex reasoning: hierarchical v. unified
PI has described a Kahneman-esque “thinking, fast and slow” approach, in which “instinctive” and “considered” actions are treated separately. In π0.5, the same model runs a two-stage inference loop: first predict a semantic subtask like “pick up the plate” at a low frequency, then predict the motor commands that execute it at a high frequency. In Hi Robot, the more recent paper, they stack two separate models — a high-level VLM planner that interprets complex natural-language prompts, takes in-the-moment feedback and corrections, and decides what the robot should do next in plain language, and a low-level VLA executor (a π0-style model) that actually produces the motor commands for each step.
Downstream of the last item, Generalist sidesteps this; again, integration into the model has pros and cons. They call it “Harmonic Reasoning” and define it thusly (unfortunately this is the full level of detail provided):
Harmonic Reasoning involves a fundamentally new approach to training models, and creates a “harmonic” interplay between asynchronous, continuous-time streams of sensing and acting tokens.
Apologies if I messed anything up in the table above—doing my best here!
Wrapping up
The rough areas where these companies have seen success–PI on long-range tasks, Generalist, with high reliability on short-range ones–is probably a logical result of the architectural decisions described above. PI’s VLM backbone, its hierarchical structure for decomposing tasks, and subtask label training suit it well for long-range tasks. Generalist, on the other hand, has this tightly built model that avoids the baggage of a hierarchical “system one / system two” reasoning logic by just… acting in the moment. So it’s really good at that! However, this does raise the question of how it’ll perform as it tries to stretch out further.
The question remains: who will get to high reliability on complex tasks first?
This got me thinking about The Gun, a history of the AK-47, which explains how a gun that is not particularly accurate became the world’s most popular one. Among other reasons, its loose tolerances and overbuilt gas system allow for it to continue to fire even if it gets dirt and debris on the inside. It’s been said you could bury one underground, and come back to it and it’d still fire. I worry the Generalist approach is more like the Western guns of the era–elegant and accurate, but prone to jam when conditions were imperfect.
On the other hand, rather than a weapon, the much more salient analogy is… LLMs. And in that space, “build a really good model, and throw a ton of data at it” has been the right approach.
We should certainly root for both of their success! If Generalist’s approach to data gathering works, and it’s as simple as “Hey, a few weeks before we ship out a robot to your house or factory or wherever, please wear these dopey-looking gloves and cameras while you do chores,” that seems like the cheapest path toward robots that can do specialized tasks in the home or at work. On the other hand, PI’s approach does seem more flexible, where points of failure can be addressed in places other than the base model.
Hopefully we get the best of both worlds, where flexible, effective, generalizable architecture becomes more robust and can combine with progressively cheaper data collection. If not, this could end in a protracted struggle for commercial viability, much like in The Boss Baby.
Let’s look at some robots
Syncere rolled out Lume, which is a lamp (or pair of lamps) that can purportedly make your bed and do some clothes folding. The price point ($1,500 for 1, $2,500 for a pair) is a good deal lower than most other available “home robotics” options, but still a good deal pricer than, say, sending you clothes out to get folded every week.
The main thing they’re selling at this time is the ability to have people and let them watch your lamp fold some t-shirts and go “neat.” I can see how that would be worth it, if you, like the people in this (and for some reason, many?) robotics demos, you live in, basically, the Scandinavian-style mansion from Ex Machina.
There’s more to think about here in terms of non-humanoid structures, and non-contiguous structures–I like the way the two arms work together! This Siddharth Ramakrishnan essay from last month took a strong anti-humanoid stance this week; I’m not totally compelled and think that there is something to the idea that the world is human-shaped. But definitely agree that seeing creativity in form factors is neat, and think the “smart home” of the future is not going to just be one big ol’ C-3PO waddling around doing the dishes. It’ll surely be both weirder and more natural-feeling than that.