This Week in Robotic Learning (TWIRL) #7: A guide to VLAs

The brains of modern robotics

Part of my role here at Robotic Learning is to help you, dear readers, learn about robotics. Our topic today is Vision-Language-Action Models, or VLAs, the most widely used model type in autonomous robotics.

When people are talking about AI at the moment, they generally mean large language models. However, a language model cannot do anything. It can only output words and images. Enter VLAs, which are the bridge from machine to reality. They’re a a special class of model that can take visuals and language as inputs and take action in the physical world. These are the state-of-the-art models that are powering the frontier of robotic learning. Let’s explore.

What is a VLA?

On a fundamental level, VLAs are the brains of state of the art general-purpose autonomous robots. Well, sometimes they are (sometimes not!). A VLA is a single large neural network that takes in what the robot sees (vision) and a goal stated in plain language (language), and directly outputs the commands that move the robot’s motors (action).

Perception (what’s that?), planning (what should I do?) and control (I’m doing it!!!!) all in one system.

Like an LLM, a VLA is a neural network, that is, a large mathematical function with millions or billions of adjustable numbers, called parameters or weights, that are tuned automatically by showing the system many examples rather than by an engineer writing rules. Again, this part is just sort of… LLM stuff.

If a robotics company uses a VLA, they’re basically betting on the same principle–a good model, a ton of data–to carry over to the physical world as it has proven out in the digital world.

The name VLA is in the lineage of VLM (Vision-Language Model) which is the kind of AI that can look at an image and discuss it (GPT-4 with vision, Gemini). A VLA takes things a step further by controlling a machine. The slogan researchers sometimes use is that a VLA can “see, understand, and act.”

The core of a VLA is, you guessed it: Vision, Language, and Action.

Vision is the robot’s sensory input, almost always one or more camera feeds. Some robots have depth cameras (which measure distance), wrist-mounted cameras (for close-up manipulation), or tactile sensors (which sense touch and pressure). The model generally receives these as raw pixels (“see!”) and then extracts meaning from them (“understand!”) the way I see a bright ball in the sky and interpret it as “the sun.”

Language is the instruction. “Put the lime in the coconut.” “Move in, move out, hands up, hands down, back up, back up, tell me what you gonna do now. Keep rollin’ rollin’ rollin’ rollin’.” Like an LLM, the model has absorbed language understanding from internet-scale text so, it can interpret instructions it was never explicitly trained on. It knows what “red” looks like and what “inexplicably large dog” is from learning about Clifford on the web’s corpus, not from robot data.

Action is the output: the joint angles, gripper commands, and other numbers that go to motors. A typical robot arm has ~6 degrees of freedom (independent ways it can move) plus a gripper. A humanoid has far more. Tesla’s Optimus has 20+ DOF just in its hands. The action stream needs to come out fast and smoothly enough to control a body in real time

How we got here: a technical and data sourcing history

As deep learning swung into gear in the early 2010s, two broad types came to the fore. Reinforcement learning (RL) lets a robot learn by trial and error, guided by a reward signal–a number telling it how well it did. RL produced results in simulation and games (AlphaGo) but struggled with real robots. Trial and error with a robot is hard (what’s a good reward function for “fold a shirt”?) slow, expensive, and not necessarily safe.

And so the field turned to imitation learning, also called behavioral cloning (BC, move over Jesus). Instead of discovering good behavior from scratch, the robot is shown demonstrations by a human and learns to copy the mapping from observation to action. This is a type of supervised learning, the same “learn from labeled examples” technique behind image classifiers, but applied to robot control.

Modern VLAs depend heavily on behavioral cloning; this data is a lot cheaper, but less directly applicable. The main weakness to worry about is compounding error. A policy that only ever saw expert demonstrations has never seen its own mistakes, so when it starts making mistakes, it keeps making worse and worse mistakes, until we’re basically in that I Love Lucy conveyor belt scene.

VLAs are the convergence of a few lines of research. The transformer was introduced in a 2017 paper titled “Attention is All You Need,” which overcame its objectively inaccurate title (what about… love?) to become the most famous AI research paper. The transformer’s key mechanism, attention, lets the model weigh which parts of its input matter for each part of its output. Transformers power modern AI.

Feed one enough text and you get a large language model. GPT-3 (2020) was the first breakthrough one. OpenAI’s CLIP (2021), trained on hundreds of millions of image-text pairs, gave robotics a system that marries language and vision (bananas are yellow and curved; baseballs are white with red seams; Brian’s hat is a fedora with safari flaps in the back).

By 2022–23, researchers were merging vision encoders with LLMs into VLMs that could see an image and converse about it. The VLA followed. Google’s RT-1 paper (Dec. 2022) was the seminal breakthrough, introducing the concept of a “Robotics Transformer” that took images and language as input and produced action tokens. RT-2 (July 2023) formalized the VLA framing by taking a pretrained VLM, already loaded with internet knowledge, and co-fine-tuning it so that alongside text it could output robot actions encoded as text-like tokens.

In 2024, researchers launched OpenVLA, which matched or beat much larger closed models at the time. Several members of that project would be part of the founding team at Physical Intelligence, whose π0 model introduced the design that now leads the higher end of VLAs by building on a VLM and generating actions with flow matching rather than as discrete tokens (more detail on that later).

A brief history of data collection

For LLMs, the data source is basically: the internet. Also, sometimes, books. For robotics, it’s not so simple.

In the dark ages, we just had… engineered solutions. No data required. Boring. When learning arrived, the first demonstrations came from kinesthetic teaching, in which a person physically would grab the robot’s arm and guide it through a motion while the joints recorded themselves. This obviously is not super scaleable or “good.”

The next step was teleoperation, a human remotely controlling the robot. This scales better and, importantly, captures the robot’s own camera views and action stream in sync. Through the late 2010s and early 2020s the leading labs industrialized a teleoperated approach. Teleoperation hardware improved as well. VR headsets, and especially leader-follower “puppet” rigs, where the operator moves a small copy of the arm and the real robot mirrors it.

The 2023 ALOHA system was a step forward in teleoperation, producing a low-cost, open-source bimanual teleoperation setup, making high-quality data collection much cheaper.

From there, Google DeepMind organized Open X-Embodiment to pool a wide range of robotics training data across different labs, morphologies, environments, etc. Models trained on this data showed immediate improvement, indicating, crucially, positive transfer across embodiments: data from one robot can be used to improve performance on a physically different robot. If we’re running the LLM playbook (model, extraordinary amount of data), transfer across embodiments dramatically increases the volume of data available compared to needing machine-specific training.

On a separate track from teleoperation, simulation and synthetic data has steadily matured: NVIDIA’s Isaac platform, better domain randomization (defined later), and synthetic-trajectory pipelines like the one behind NVIDIA’s GR00T N1.

Recently, two more sources of robot data production have emerged to accelerate and improve data collection: Community data sources and low-cost hardware. Hugging Face’s LeRobot project, sub-$1,000 open arms, and crowd-contributed datasets have brought costs down considerably.

Another approach is wearable-device data from humans. People operating gloves and wristbands that capture hand and arm motion while a person performs everyday tasks, paired with first-person video. The data is dramatically cheaper to collect than teleop and diverse. Generalist AI, Sunday, and Genesis are betting big on wearables.

The current frontier (depending on who you ask), fittingly, ties it all together. Autonomous, self-generated experience is the name of the game. Rather than humans producing every demonstration, the robot acts, succeeds and fails, and learns from its own logged experience with occasional human “coaching” — the data model behind π*0.6 and RECAP.

I like the data pyramid mental model to summarize the current state.

At the base is cheap, already-collected web data. VLAs capture this for free via the VLM backbone, and labs keep it in the mix through co-training. Then comes human and cross-embodiment videos, people doing tasks (wearables), and demonstrations from other robots. The actions are not the robot’s own, so this data must be adapted. At the top lies first-party teleoperated data on the target robot. It’s the gold standard, but it’s also the scarcest and most expensive layer.

The name of the game is getting maximum value from the cheap lower layers so you need as little of the expensive top layer as possible.

Human-produced data can only grow linearly with human output, human hours. Each advance in the data space is an attempt to work around that, whether that be via pooling data, simulating it, crowdsourcing it or having the robot do it on its own. It is, in a sense, working. High-quality teleoperation data fell from roughly $340 per hour in early 2024 to about $118 by early 2026, and an entire data-services industry has grown around the problem.

How a VLA works

A VLA has three parts: a perception-and-reasoning backbone, an action generation module, and a training recipe.

The VLM backbone

The backbone is a transformer-based vision-language model. It’s a vision encoder plus a language model, already trained together on internet-scale image-text data. It has things that robot data cannot bring: visual perception, language understanding, and broad world knowledge.

The vision encoder turns raw pixels into embeddings–lists of numbers that encode meaning in a form math can operate on. Leading VLAs often combine two complementary vision encoders: SigLIP, strong at semantic, language-aligned understanding (what things are), and DINOv2, strong at spatial, geometric structure (where they are and how they’re shaped). OpenVLA, for example, has this structure.

The language model is usually a modestly-sized open model such as Meta’s Llama or Google’s Gemma. Some labs use frontier models (Gemini, in Gemini Robotics), trading latency and cost against capability and strategic independence (see: Figure’s split with OpenAI).

The backbone’s job is to fuse vision, language, and the robot’s current physical state (joint positions, etc.) into a shared internal representation. It’s complicated! But this fusion is why VLAs can follow new instructions: vision and language live in the same representational space, so the model can interpret a word in a pixel region.

Action generation: tokens v. flow matching

A language model outputs tokens (discrete chunks of text, produced one at a time). A robot needs actions. It needs continuous numbers (joint angles, gripper positions), produced in smooth, fast sequences. There are, at this point, two main ways of doing it.

Discrete token output (autoregressive VLAs), used by RT-2 and OpenVLA: discretize the continuous action space into a fixed number of bins per dimension, assign each bin a token, and let the VLM produce action tokens the same way it produces words. Advantage: reuses the VLM unchanged. Disadvantages: discretization loses precision, and one-at-a-time generation is slow, so motion comes out jerky. π0-FAST introduced a smarter, frequency-based action tokenization that helps but doesn’t fully fix it.

Continuous output via diffusion or flow matching, used by π0, π0.5, and most current high-end models, abandons tokens for actions entirely. Diffusion is the technique behind modern AI image generators. It’s basically the AI version of “remove all the parts of the marble that doesn’t look like an elephant,” except it actually works. The model is trained by progressively adding noise to real examples until they’re unrecognizable, then learning to reverse that process. Applied to robotics, the “image” being generated is a chunk of future robot motion. It starts from a random noisy trajectory and turns it into a clean executable one, conditioned on what the robot sees.

Flow matching is faster and more stable. Instead of predicting noise at each step, it learns a vector field: at any point between pure noise and a real action, it learns which direction to move. Flow matching converges in fewer steps, which matters enormously when actions need to come out now. π0 used flow matching to hit roughly 50 Hz continuous control, that is, 50 action updates per second.

Continuous output beats discrete on three fronts. It preserves precision. It naturally produces smooth motion. And it handles multimodality–the fact that there are often several equally valid ways to do a task (you can grasp a mug from the left or the right). A model that just averages those options produces a useless in-between action; a flow model can represent the full distribution and commit to one.

Within a continuous output system, two more concepts are key.

An action expert (or action head) is a dedicated module attached to the VLM backbone, specialized for producing actions. It’s separated because if you force the VLM’s own weights to learn motor control, you risk degrading the language and reasoning abilities it was pretrained for. The action expert attends to everything the backbone computed but learns motor control in its own weights, leaving the backbone’s knowledge intact.

Action chunking means the model predicts a short sequence of future actions at once (50-ish timesteps is common) rather than one action at a time. It was introduced by the Action Chunking Transformer (ACT) paper, whose authors include Tony Zhao (now at Sunday Robotics) and Chelsea Finn and Sergey Levine (now at Physical Intelligence). Chunking produces smoother motion and reduces compounding error: the robot commits to a short coherent plan rather than re-deciding every instant and jiggling all over the place.

Single-system vs. dual-system architectures

Any Daniel Kahneman fans in the house? If so, you’re familiar with his contribution to dual process theory in which intuition and reasoning are more or less discrete systems. The application to robotics is clear. A policy needs to be able to handle both “I’m going to plan how I’ll put away the dishes” and “AAAAH A BASEBALL IS FLYING TOWARD ME VERY QUICKLY!!!”).

Single-system (monolithic) VLAs such as RT-2, OpenVLA, and π0 run one model that takes in vision and language and produces actions in a single forward pass. Pros: simple, low latency, and easy to train end-to-end. Cons: the same network must both reason slowly about the task and react quickly with motor commands, and those have conflicting requirements.

Dual-system VLAs such as Figure’s Helix, NVIDIA’s GR00T N1 split the model in two, an architecture explicitly inspired by Kahneman’s framework. System 2 is a larger, slower module that handles perception, language understanding, and planning, running at a low frequency (a few times per second). System 1 is a smaller, faster module that takes System 2’s guidance and produces high-frequency motor commands (up to hundreds of times per second). Pros: each part runs at its natural speed, improving both dexterity and responsiveness. Cons: more complex to build and train.

This is definitely still an open debate, so it’ll be interesting to see how things converge as single-system models add explicit “thinking” steps and dual-system models tighten the distance between their halves.

Inference speed and the real-time constraint

A robot is a physical object subject to physics. Sucks, I know. This means the model must produce actions fast enough to control it in real time. You know how it’s annoying when an LLM thinks for a bit? Well, it’s much worse when your house robot thinks about whether or not to turn off the stove as your house burns down.

This real-time constraint drives many design choices. Flow matching over token generation, action experts, dual systems, action chunking, asynchronous inference (computing the next action chunk while the current one executes), and a strong push toward models that run on the robot’s onboard chips rather than in the cloud (Helix’s onboard operation was a notable milestone) are all downstream of a Veruca Salt-esque “I want the golden goose and I want it now!” need for immediacy).

The training recipe

This can vary, but!

1. Inherit pretraining. The VLM backbone arrives already pretrained on internet image-text data. This is the one part of the process in which you can say “yes, there is enough cheap, easily accessible data.”

2. Robot pretraining / co-training. The model is trained on large, diverse robot datasets–Open X-Embodiment, proprietary fleets, community data–usually co-trained with non-robot data (web images, captioning tasks, etc.) so it does not “forget” its general knowledge while learning motor skills. This is most of it! It’s where robot competence is installed.

3. Fine-tuning / post-training. The general model is specialized for a target robot and task domain with a smaller, focused dataset. Gemini Robotics On-Device could adapt with as few as 50–100 demonstrations because the base model was already so capable. You’ll hear about “zero-shot” learning, in which a robot can do something it’s never seen. When that happens, it’s a big deal!

4. Reinforcement learning from experience (emerging). Exemplified by π*0.6’s RECAP method: after deployment, the model learns from its own real-world successes and failures, plus occasional human corrections. This directly attacks behavioral cloning’s compounding-error weakness by finally letting the robot learn to recover from situations its demonstrations never showed.

At a high level, though, a model’s competence is bounded far more by the quantity, quality, and diversity of data than by its architecture. So, data.

Two approaches to cheap data

There are two fundamental approaches to the “we don’t have enough data / data is expensive problem.” One: figure out how to make your model work with cheap data. Two: make the high quality data less expensive. This lack of cheap data is clearly holding up the development of general purpose robotics, so advances on either here can have industry-wide impact.

Simulation means generating training data inside a physics engine–a virtual world–instead of with physical robots. Its appeal is obvious: simulation is fast, cheap, parallelizable across thousands of virtual robots, safe, and perfectly labeled. NVIDIA’s Isaac platform is the best-known toolset here.

You’ll often hear about the sim-to-real gap (or reality gap): a policy trained in simulation often fails on the real robot because the simulator does not perfectly capture reality. The visual gap is now largely manageable through domain randomization–deliberately varying textures, lighting, and colors in simulation so the model learns to ignore appearance.

The harder gap is dynamics, that is, the pesky real world, with its friction, contact forces, the slight imperfections of real motors. Sadly we cannot assume a spherical cow. A grasp planned against a simulator’s idealized physics may approach an object on a trajectory the real arm cannot quite reproduce. Contact-rich tasks–anything involving pushing, inserting, or grasping with force–remain very challenging.

Synthetic data generation represents a middle path between hand-collected real data and pure simulation. It uses generative models and simulation pipelines to multiply a small set of real demonstrations into a large set of varied ones. NVIDIA’s GR00T pipeline is the marquee example, with its 6,500 hours of synthetic trajectories generated in 11 hours. (“Trajectories” are complete recordings of a robot doing a task from start to finish.) The pattern that has emerged across in the past year is not “sim or real” but structured hybrid pipelines that blend teleoperation, autonomous collection, simulation, and synthetic augmentation deliberately.

Internet video and “learning by watching” is the effort to extract usable signals from ordinary human videos. Techniques to recover latent actions from video and world models trained to predict future video frames both aim to make use of this enormous and cheap data source. I’ve written about that a bit here, and to date general purpose world models aren’t viable. But this is an area to watch closely because success in building high fidelity world models would fundamentally reshape robotics almost instantly.

Because no single lab can collect enough, shared datasets are infrastructure. We mentioned Open X-Embodiment earlier, and there are more, such as DROID and HuggingFace’s LeRobot Community Datasets. Similar to my “publish your research” exhortation, data is so scarce that all robotics companies benefit from the existence of an increasingly robust, shared dataset.

Open problems and the frontier

Let’s remember what the goal here is: generalizable and robust real-world robotic performance on long-range tasks. Generalization is a term that gets thrown around a lot and can mean a lot of different things. From easiest to hardest, I’d put it as:

1. In-distribution: same task, same objects, same environment as training.

2. Novel object / novel arrangement: known task, but new objects or new layout.

3. Novel instruction: a command phrased differently, or combining known skills in a new way.

4. Novel environment: an entirely new room, home, or workspace.

5. Novel task / novel skill: a behavior never demonstrated.

6. Cross-embodiment: transferring all that to a physically different body.

VLAs still degrade under distractions, clutter, unusual lighting (part of the cleverness of Syncere’s Lume–a lamp brings its own lighting!), and in novel environments. Generalizing to novel environments and novel tasks, is the central unsolved problem. Robustness benchmarks like LIBERO-Plus exist because models that ace clean tests can fall apart under realistic perturbation.

The question that will determine whether VLAs remain the dominant architecture for robot brains is: how do we get more value per dollar of data? Maybe it just means more and better data sharing across firms. Maybe that means a staggering amount of real-world data collection. Maybe it means a large, but not-staggering amount of real-world data collection. Maybe it means investing in simulated environments and world models. Maybe none of those things work and it means scrapping the whole architecture for something touch-based.

Unfortunately, evaluation methodology is fuzzy enough that it’s hard to know if someone has had a genuine breakthrough or just made a cool video. There have been numerous attempts to call something in robotics a “GPT-3 moment.” I don’t buy it. GPT-3’s moment was about getting in actual users’ hands, and at this point, no normal person has a home robot. Until there’s mass shipping of autonomous machines, it’ll be a bit of a guessing game–so, I invite you to play along.

Let’s look at some robots

There has never been a more immediate and obvious selection for “Let’s look at some robots.” I could try and reason through what if any practical uses might exist for this machine… or I could listen to those guitars shred, baby!

My favorite part is when it punches a cinderblock wall, but perhaps yours is when it walks through some sort of weird Main Street facade where every car was made no more recently than 1974. There’s something for everyone.