Abstract
World-action models (WAMs) augment vision-language-action policies (VLAs) with future prediction so that the policy can anticipate how its actions will affect the physical world. Existing WAMs supervise complete future observations. However, observations typically change little over an action chunk, so the future loss is dominated by copying static background content rather than learning task-relevant robot and object motion, limiting policy learning. Moreover, existing WAMs rely on large video generation models or separate world models, incurring heavy training and inference compute that hinders closed-loop deployment. To improve policy performance, we propose visual residual learning, supervising only the feature change from the current frame to the future frame in a frozen DINOv3 space. Compared with complete future features, this residual learning target yields a higher feature rank with richer and more diverse representations of scene changes, and spatially concentrates on the robot arm and manipulated objects rather than the background. To further improve computational efficiency, we present ResWAM, a residual world-action model with 2.3B parameters in total, including a lightweight 0.5B dual-branch DiT that jointly trains an action branch and a residual branch via conditional flow matching. Without robot pretraining (i.e., pretraining on external robot action trajectories beyond the benchmark demonstrations), ResWAM achieves top average success rates on LIBERO-Plus (81.1%), RoboTwin 2.0 (93.8%), and RoboDojo (9.40% success, 13.41 score) at the lowest training (2.7–4.0 TFLOPs) and inference compute (1.0–1.4 TFLOPs), and leads real-robot Open, Put, and Stack (80.0% average, +10.0% over π0.5) at 92.7 ms latency.
Contributions
- 1
Supervision-wise, to strengthen the supervision signal for higher performance, we propose visual residual learning. We show that this signal exhibits higher feature rank than full future features and concentrates on the robot and manipulated objects rather than static backgrounds, providing a more focused and expressive supervision signal.
- 2
Architecture-wise, to improve computational efficiency, we design ResWAM, a residual world-action model with 2.3B parameters in total, including a lightweight 0.5B dual-branch DiT that jointly denoises actions and frozen DINOv3 visual residuals via conditional flow matching in place of video generation models or separate world models.
- 3
Experiment-wise, without robot pretraining, ResWAM achieves the highest average success rates on LIBERO-Plus (81.1%), RoboTwin 2.0 (93.8%), and RoboDojo (9.40% success, 13.41 score) at the lowest training (2.7–4.0 TFLOPs) and inference compute (1.0–1.4 TFLOPs), and leads real-world FR3 tasks (80.0% average, +10.0% over π0.5) at 92.7 ms latency.
What Is the Residual?
We extract visual patch tokens with a frozen DINOv3 encoder.
The visual residual target is the feature change from time t to time t+H.
Two findings about the residual
1. The residual concentrates on the robot and the manipulated objects.
LIBERO
put both moka pots on the stove
open the top drawer and put the bowl inside
pick up the black bowl in the top drawer of the wooden cabinet and place it on the plate
RoboTwin 2.0
Place the light blue bowl with darkened rim onto the light blue bowl with darkened rim to form a stack.
Pass the red block to the right arm and set it down.
Locate the medium gray microwave with top vents and open it using the left arm.
2. The residual has higher effective rank and stable rank.
Let JX be the token-centered feature matrix, where J = IN − (1/N)11⊤, and let pi = σi2 / Σjσj2.
Effective rank
Stable rank
Higher values mean that spectral energy spans more feature directions of scene change instead of copying static background modes. Both measures are scale-invariant.
ResWAM at a glance
- 2.3B parameters in total: a lightweight 0.5B dual-branch DiT (action branch and residual branch) attached directly to a pretrained VLM (RynnBrain-1.1-2B) and a frozen DINOv3 ViT-B/16 encoder (2×2-pooled to N = 64 tokens per view).
- Training via conditional flow matching: the future frame ot+H is used only during training to form the frozen DINOv3 residual target Δt = Zt+H − Zt (+2.9% over direct future prediction on RoboTwin 2.0 at identical compute).
- Single-pass latent inference: at test time, only ot, proprioceptive state qt, and instruction c are required, integrating actions and compact residual tokens over 10 Euler steps without pixel reconstruction or 3D video VAEs.
Headline: best average among compared methods
81.1%LIBERO-Plus (10,030 episodes)+1.6 pts over VLA-JEPA (79.5%)
93.8%RoboTwin 2.0 (50 tasks)+1.6 pts over LingBot-VA (92.2%)
9.40% / 13.41RoboDojo (SR / Score)+2.49 pts SR over π0.5 (6.91%)
80.0%Real Robot FR3 (92.7 ms)+10.0 pts over π0.5 (70.0%)
2.3B parameters · 2.7–4.0 TFLOPs Train1.0–1.4 TFLOPs Infer. · 92.2–99.7 ms · w/o Robot Pretrain
LIBERO-Plus: zero-shot out-of-distribution generalization
Without robot pretraining and with 2.3B parameters, ResWAM achieves an 81.1% overall success rate across all 10,030 perturbed episodes at the lowest training compute (2.7 TFLOPs), inference compute (1.0 TFLOPs), and latency (92.2 ms), outperforming VLA-JEPA (79.5%, 5.7 / 1.2 TFLOPs, 192.3 ms) and LaWAM (76.6%, 3.2 / 1.2 TFLOPs, 198.1 ms).
Efficiency and success rates (%) on LIBERO-Plus across the seven perturbation dimensions and their overall average (Avg.) over all 10,030 perturbed episodes. All policies are trained on standard LIBERO demonstrations (without exposure to LIBERO-Plus perturbations). Robot Pretrain indicates whether a policy uses external robot action datasets beyond the benchmark demonstrations (✓) or is trained solely on benchmark demonstrations (✗). Size is the number of parameters in billions loaded at inference. Efficiency comprises per-sample training (Train) and per-chunk inference (Infer.) compute in TFLOPs, and per-chunk inference Latency in ms on one NVIDIA RTX A6000. The best result in each column is in bold and the second best is underlined.
Bold: best in column. Underline: second best. Train and Infer. are in TFLOPs; Latency is in ms on one NVIDIA RTX A6000.
RoboTwin 2.0: 50 bimanual tasks
ResWAM achieves 93.6% under clean conditions and 94.0% under domain-randomized conditions (93.8% average) at the lowest training compute (4.0 TFLOPs), inference compute (1.4 TFLOPs), and latency (99.4 ms), ranking first in every column without robot pretraining.
Efficiency and success rates (%) on RoboTwin 2.0 (50 tasks, 100 trials per task) under clean and randomized (Random) conditions. Notation as in Table 1.
RoboDojo: 42 bimanual tasks across five capability dimensions
Without robot pretraining, ResWAM achieves 9.40% average success rate and 13.41 progress score alongside the lowest training compute (3.9 TFLOPs), inference compute (1.4 TFLOPs), and latency (99.7 ms), outperforming π0.5 (6.91% / 11.41, 13.6 / 5.0 TFLOPs, 125.4 ms) and nearly tripling GigaWorld-Policy (3.27% / 6.20, 34.6 / 2.5 TFLOPs, 666.3 ms).
Efficiency, success rates (SR, %), and stage-level progress scores (Score) on the RoboDojo simulation benchmark across the five capability dimensions (42 tasks, 54 configurations, 2,100 episodes) and their macro-average (Avg.). Notation as in Table 1.
Real-World Manipulation on Franka Research 3
ResWAM achieves the highest success rate on all three tabletop tasks (Open 90.0%, Put 96.7%, Stack 53.3%) with the lowest training compute (2.8 TFLOPs), inference compute (1.0 TFLOPs), and latency (92.7 ms), averaging 80.0% across 30 trials per task (+10.0% over π0.5 at 70.0%, 5.0 TFLOPs, and 120.4 ms).
Efficiency and success rates (%) on the three real-world manipulation tasks (30 trials per task) on a Franka Research 3 (FR3) arm. Notation as in Table 1.
Citation
@article{fan2026reswam,
title={ResWAM: Learning What Changes, Not What Stays, for World-Action Models},
author={Fan, Chongyu and Liu, Gaowen and Kompella, Ramana Rao and Corso, Jason and Liu, Sijia},
journal={arXiv preprint},
year={2026}
}