VaM
Without value caching
Full demonstration
Remember progress. Learn from experience.
Visually similar observations can require different actions. VaM uses recent value predictions to help distinguish continued progress from situations that call for recovery.
Value caching. The action expert conditions on previously computed value predictions alongside visual and proprioceptive observations. This recent progress history helps the policy recognize stalled or declining progress.
Online learning. Offline demonstrations and off-policy human interventions are reused alongside on-policy autonomous rollouts to support continued policy improvement. Data collection and updates to the value and action experts run concurrently.
An offline-to-online learning framework
Four training stages connect demonstration initialization to autonomous learning. A distributional value expert supplies the progress memory used by the action expert.
- Offline demonstrationsInitialize the action expert with behavioral cloning and the value expert with progress estimation.
- Offline RL warmupWarm up the value expert with TD learning and introduce advantage-conditioned policy training.
- Online DAggerCollect deployment experience with human interventions and update both experts.
- Learn from ExperienceContinue autonomous rollouts and policy updates using sparse terminal rewards.
Continuous rollouts across three tasks
Accelerated demonstrations of long-horizon assembly and deformable-object manipulation.
Box Assembly
A continuous box assembly demonstration involving deformable cardboard and contact-rich manipulation.
Laundry Folding
A continuous laundry folding demonstration with coordinated manipulation of deformable fabric.
Magnetic Bus Assembly
A continuous magnetic bus assembly demonstration requiring sequential part alignment and contact.
Experimental Results
Value Caching
Online RL performance
We evaluate whether value caching improves online policy learning on Box Assembly, Laundry Folding, and Magnetic Bus Assembly, comparing VaM with two ablations and RECAP. VaM and the no-cache ablation begin after offline RL warmup; the no-warmup ablation and RECAP start from demonstration-trained policies and receive extra Online DAgger rounds to compensate for the warmup data. The curves follow training from human-corrected Online DAgger to autonomous Learn from Experience (LfE).
Combating partial observability
To understand how value memory helps with ambiguous observations, we examine recovery behavior and the representations that accompany it. Annotators review all autonomous LfE rollouts from the four methods without knowing which method produced them. A recovery succeeds when corrective actions complete the affected task stage, and recovery rate measures the fraction of recovery-required events resolved this way. The examples below align camera frames, PCA projections of value-expert embeddings, and value predictions to show how VaM and the no-cache ablation respond to stalled progress across all three tasks.
VaM
Without value caching
VaM
Without value caching
VaM
Without value caching
VaM
Without value caching
VaM
Without value caching
VaM
Without value caching
VaM
Without value caching
Value source matters
Does the source of the cached values matter? We keep the action experts from the final three LfE checkpoints fixed and replace only their value experts with demo-trained or post-warmup snapshots, leaving caching enabled. The bars compare success rates across all three tasks, with sample standard deviations across checkpoints. In the two Laundry Folding cases below, we also fix every observation input, including the four frames from −3 to 0 seconds, so differences in predicted trajectories and gripper commands come from the substituted value history rather than a change in observations.
After contact is lost, the original value history predicts reopening and repositioning the right gripper for another grasp. After the seams are re-grasped, it instead maintains closure, while the earlier value snapshots predict opening near the end of the action chunk. These are predicted actions under fixed observations, not executed recovery rollouts.
Drag to rotate · Scroll to zoom
Left end-effector prediction
Right end-effector prediction
Value history
Gripper command
Loading experimental data…
Online RL system
Closed-loop iterations
Starting from the same post-warmup model, we compare online and offline RL under matched gradient-step and data-collection budgets. After LfE, online updates improve mean success rate by 32.8 percentage points across the three tasks, showing the benefit of repeatedly learning from the policy’s latest experience.
Asynchronous online training
Online RL collects experience and trains the value and action experts concurrently, while offline RL performs these steps in sequence. Under the evaluated schedules, this overlap reduces experimental time per gradient step by 57.8% on average across the three tasks.
Citations
@unpublished{yu2026valueasmemory,
title = {Value as Memory: Online Value-Guided Post-training for General Robot Policies},
author = {Yu, Wenye and Chen, Yijin and Xiong, Kaiyi and Han, Wenbo and Chen, Wendi and Jin, Yang and Xue, Han and Lv, Jun and Lu, Cewu and Wen, Chuan},
year = {2026},
note = {Manuscript}
}