Adaptation of Generalist Robot Policies with Minimal Data
Pretraining enables one-shot downstream online adaptation.
Real-world results.
T1 · Knife and donut onto plate
T2 · Green block to right container, blue block to left container
Learning Progression of MiDAS.
MiDAS learns interesting new behaviours with large resets.
Abstract.
A central goal of robot learning is to move beyond task-specific human data collection toward robots that improve through autonomous interaction. Yet fully autonomous learning remains difficult with current policies: sparse rewards and weak zero-shot exploration make it unlikely that a robot will discover successful behavior from scratch. We study minimal-data adaptation, a regime in which a pretrained robot policy must learn a new task from as little as one demonstration followed by autonomous online interaction. This setting minimizes task-specific supervision while testing whether pretrained policies provide enough structure to make reinforcement learning tractable. We present MiDAS, an offline-to-online RL framework that first anchors a pretrained VLA to the target task with few-demo behavior cloning, then improves it through value-based online residual RL. Across LIBERO and RoboCasa, MiDAS recovers strong task performance from as little as one demonstration, substantially outperforming baselines and generalizing beyond demonstrated conditions. To the best of our knowledge, this is the first demonstration of reliable robot policy adaptation from a single task demonstration. We further validate MiDAS on bimanual YAM hardware, where it learns effective behaviors with roughly six hours of autonomous interaction.
MDA as the closest tractable proxy for autonomous adaptation.
Overview. A single demonstration is sparse and underdetermined on its own, but pretrained VLA representations turn it into a useful anchor for adaptation. Few-demo behavior cloning narrows the search space to a task-relevant region around the demonstration, and online residual RL then expands local success by learning value-guided corrections on top of the warm-started base policy.
INGREDIENT 1
Represent observations for control & value learning
High-dimensional pixels and language need to become features that are useful for both action prediction and value learning.
Supplied by: VLA pretraining
INGREDIENT 2
Reach reward-bearing regions of the state space
Online RL needs the robot to occasionally succeed. The policy must at least attempt the task in a way that produces signal.
Supplied by: 1 demonstration (on top of pretraining)
INGREDIENT 3
Use experience to improve efficiently
Sparse interaction data must yield reliable control — including actions outside what the base policy reliably proposes.
Supplied by: Autonomous online interaction
What enables minimal-data adaptation?
Question 1
What does one demonstration recover?
FindingOne demonstration coupled with pretraining recovers task-directed behavior, but not reliable control.
Fig. Performance of different task representations for behavior cloning. Only the pretrained VLA + 1 demo recovers meaningful task behavior.
The resulting policy often reaches the correct objects and attempts the demonstrated subtask sequence, but failures remain concentrated in contact-rich execution: alignment, grasp closure, precise placement, and recovery from small deviations.
Few-demo BC recovers task-level behavior, but not fine-grained control. Rollouts of πbaseK at K=1 on Both Moka Pots → Stove. The policy reaches the right objects but fails to grasp the second pot under orientation changes.
Question 2
Do pretrained representations simplify online adaptation?
FindingPretrained VLA representations substantially improve the sample-efficiency of sparse-reward residual RL.
The one-demo policy anchors the search near task-relevant behavior, but sparse-reward online RL remains difficult when the residual learner must infer useful state abstractions while also learning the fine-grained control corrections needed for success.
Online adaptation with different representations. ResNet-from-scratch fails to converge; frozen DINO features eventually progress but slowly; frozen pretrained π0.5 features enable rapid improvement.
Question 3
What kind of policy improvement is needed after few-demo behavior cloning?
FindingMinimal-data adaptation requires value-guided updates that can move beyond the few-demo policy’s effective action support.
To close the control gap, online adaptation must not only reuse the base policy’s task-relevant behavior, but also explore and learn corrective actions outside its effective action support.
(a) Trajectory critical state
(b) UMAP of actions
MiDAS reaches actions outside the effective support of πbaseK. UMAP of action chunks at a critical step of a LIBERO-Long rollout. Filtered BC and DSRL remain concentrated near samples from πbaseK and fail; MiDAS selects an action in a disjoint region with higher Q* and succeeds.
In-distribution simulation evaluation.
| Task | BC | DSRL | Filt. BC | MiDAS |
|---|---|---|---|---|
| LIBERO-Long | ||||
| Alph. Soup + Cr. Cheese → Basket | 12.7±1.2 | 16.0±6.9 | 14.0±2.0 | 98.7±1.2 |
| Alph. Soup + Tom. Sauce → Basket | 4.0±3.5 | 3.3±1.2 | 2.0±0.0 | 82.0±5.3 |
| Black Bowl → Bottom Drawer | 22.7±6.4 | 26.7±1.2 | 28.7±10.3 | 96.7±2.3 |
| Book → Caddy | 37.3±7.0 | 30.7±12.2 | 28.0±2.0 | 92.7±5.0 |
| Both Moka Pots → Stove | 13.3±3.1 | 20.7±2.3 | 19.3±7.0 | 90.7±2.3 |
| Cr. Cheese + Butter → Basket | 34.0±9.2 | 39.3±1.2 | 68.0±2.0 | 100.0±0.0 |
| Moka Pot → Stove | 84.0±0.0 | 92.7±3.1 | 85.3±8.3 | 94.0±0.0 |
| White Mug + Choc. Pudding | 33.3±11.7 | 33.3±10.3 | 39.3±2.3 | 85.3±4.2 |
| White Mug + Plates | 30.0±2.0 | 23.3±3.1 | 34.7±6.4 | 76.0±2.0 |
| Yellow-White Mug → Microwave | 64.0±2.0 | 47.3±8.3 | 68.0±7.2 | 84.7±4.2 |
| Average | 33.5±1.8 | 33.3±2.0 | 38.7±4.8 | 90.1±0.4 |
| RoboCasa | ||||
| Banana: Fridge Drawer → Shelf | 13.3±2.5 | 13.9±0.2 | 19.3±3.1 | 95.2±2.8 |
| Hot Dog: Counter → Cabinet | 29.3±0.9 | 17.6±8.4 | 36.0±24.0 | 87.1±8.6 |
| Mug → Coffee Machine + Start | 24.0±7.1 | 24.7±3.7 | 46.7±6.2 | 81.9±5.7 |
| Cup + Bowl → Dishwasher + Close | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 | 0.0±0.0 |
| Average | 22.2±3.5 | 18.7±4.1 | 34.0±10.6 | 88.1±5.7 |
Generalization & robustness.
Minimal-data adaptation can generalize beyond the single demonstrated trajectory, but its generalization is structured. We separate shifts into observation shifts, which change task-irrelevant inputs such as color, texture, or language phrasing, and state shifts, which change the physical conditions that determine what actions are needed, such as object pose, shape, or identity.
Finding 1
Observation-level robustness is inherited from the frozen vision-language backbone.
(a) Visual shift (color / texture)
(b) Language shift (paraphrase)
Observation shifts. Pretrained representations preserve performance under visual and language changes.
Finding 2
State generalization depends on whether the same underlying behavior still solves the task.
(a) Shape change
(b) Object swap
(c) Object change
State shifts. Online RL improves robustness when the same manipulation behavior suffices. It degrades when the shift demands a distinct affordance or strategy.
Finding 3
Coverage is critical for generalization.
This coverage can come either from additional sub-optimal demonstrations or from progressively widening the reset distribution during online RL.
Curriculum closes the gap. Progressively widening the reset distribution during online RL recovers most of the position-generalization gap with πbase50 without additional demos.
Minimal-Data Adaptation Strategy (MiDAS).
Stage I · Behavior cloning on K demonstrations
Adapt the pretrained VLA to the task: LoRA on the VLM backbone for compute efficiency, full fine-tuning of the action head with a flow-matching loss. The resulting policy πbaseK coarsely attempts the task but is not yet reliable.
Stage II · Online residual RL with frozen backbone
Freeze πbaseK and train a lightweight residual actor–critic on top of its representations. At each state, the base policy proposes an action chunk; the residual policy conditions on the state and the proposal, and predicts the executed action directly (a Gaussian-tanh distribution). This conditioning anchors RL to a task-relevant proposal while letting the executed action shift to higher-value regions outside the base policy’s reliable support. A critic ensemble is trained with TD learning.
Three design choices for the minimal-data regime
(1) Offline warm-up. Initialize the residual actor by regressing it onto the base policy’s proposals on a buffer of demonstrations + early rollouts. Train the critic on the same buffer with TD learning so it is calibrated to the regions where online improvement begins.
(2) Success balancing. Maintain a standard replay buffer and a success buffer (both seeded with the K demonstrations). Oversample successful transitions during critic training to keep the value function from collapsing to failure-dominated estimates when the base policy succeeds only 2–5% of the time.
(3) Residual training with PA-RL. Sample a single base proposal, draw N residual samples conditioned on it, and apply a PA-RL Best-of-N distillation objective with an action gradient. This produces aggressive value-guided updates while needing only one forward pass through the expensive frozen base.
(2) Success balancing. Maintain a standard replay buffer and a success buffer (both seeded with the K demonstrations). Oversample successful transitions during critic training to keep the value function from collapsing to failure-dominated estimates when the base policy succeeds only 2–5% of the time.
(3) Residual training with PA-RL. Sample a single base proposal, draw N residual samples conditioned on it, and apply a PA-RL Best-of-N distillation objective with an action gradient. This produces aggressive value-guided updates while needing only one forward pass through the expensive frozen base.