Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
arXiv:2608.098532026-08-09
Teaching robots 'how much time is left' turns out to beat preference-labeled reward models
RynnValue is an open-source value foundation model that learns to estimate how much time remains until a robot task is completed, using only timestamps rather than human preference labels or normalized progress annotations. It scales to over 7,000 hours and roughly 3 million instruction-conditioned clips gathered from heterogeneous robot, simulation, and egocentric data. On a benchmark it outperformed the fully preference-supervised state of the art, and when converted into rewards it substantially boosted real-world policy success rates.
METAL MEDIA explanatory visual
How RynnValue turns raw robot data into a reward signal
Evidence statusMeasured results reported
Heterogeneous data collectionOver 7,000 hours across real, simulated, and first-person robot data; 1.67M raw episodes are segmented and cutoff-relabeled into 3M+ instruction-conditioned clips
Timestamp-derived labelingInstead of preference pairs or progress scores, each observation gets an absolute label (remaining time to the relabeled completion cutoff) and a relative label (time gap to the next sampled observation)
Shortcut-suppressing trainingRandom temporal sampling and temporal-order shuffling remove order/interval cues, while value-isolation attention blocks value-query tokens from copying each other
Dual distributional predictionTwo heads decode absolute temporal distance and relative temporal displacement from 256-bin symlog distributions; a separate language branch produces video description, matching, and success judgments
Reward interface conversionPredicted temporal distance is sign-flipped into a potential and passed through potential-based shaping to give dense rewards for online (DSRL) and offline (IQL) robot RL
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
The paper argues that general-purpose reward models are the real bottleneck for scaling robot reinforcement learning, and that existing supervision anchors like preference pairs or normalized [0,1] progress do not transfer well across different robots and data sources.
As an alternative, RynnValue predicts 'temporal distance': the directed time cost from the current observation to a language-specified goal. Because these labels come directly from timestamps, the recipe scales without needing preference or progress annotations.
To prevent the model from learning shortcuts instead of real visual evidence, the training combines random temporal sampling, temporal-order shuffling, and value-isolation attention, which stops value-prediction tokens for different observations from copying each other.
The training corpus, drawn from diverse real, simulated, and first-person demonstration data, exceeds 7,000 hours; after subtask segmentation and cutoff relabeling, the original 1.67M episodes become over 3M instruction-conditioned trajectory segments.
On the RBM-EVAL-OOD benchmark, RynnValue reached an average Kendall's τa of 0.675 without any preference labels, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only baseline (0.292).
Figure 1: Overview of RynnValue. RynnValue is a language-conditioned value model trained on over 7,000 hours of heterogeneous embodied data, comprising roughly 3M instruction-conditioned trajectory clips across diverse embodiments, viewpoints, and task domains. Given a language instruction and a sequence of sampled observations, the model builds an interleaved multimodal sequence of repeated absolute-value and relative-value queries, which RynnBrain encodes in a single forward pass. Two distributional heads then predict the absolute temporal distance to task completion and the signed relative temporal displacement between observations, while the language branch produces video analysis and language-conditioned verification. The resulting temporal values serve as a unified interface for progress estimation, failure detection, and reward specification in robotic reinforcement learning.
Table 1: Composition of the heterogeneous data mixture before subtask expansion. The corpus contains 1.67M original episodes and is further converted into over 3M instruction-conditioned trajectory segments after subtask segmentation and cutoff relabeling.
Data Source
# Original Episodes
# Segmentations
# Instructions
Segmentation Source
AgiBot 3
167,535
1,166,042
3,741
coarse task
EgoDex 10
338,234
338,234
2,038
full trajectory
Galaxea Open-World 13
16,979
95,671
11,070
coarse task
InternData-A1 27
320,905
320,905
348
full trajectory
Open X-Embodiment 23
693,037
693,037
180,090
full trajectory
RDT 17
6,109
6,109
272
per-file coarse task
RoboCOIN 30
67,420
410,877
2,124
coarse task
RoboMIND 29
32,138
32,138
184
full trajectory
RoboTwin 20
27,414
27,414
23,527
full trajectory
Soft-FOLD 35
1,542
1,542
1
per-file coarse task
Total
1,671,313
3,091,969
223,395
–
Figure 2: RynnValue training pipeline and value-isolation attention. (a) Training strategy. Random temporal sampling and temporal-order shuffling suppress shortcuts tied to sampling intervals and sequence position, while instruction-mismatch augmentation strengthens language–visual grounding. RynnValue jointly learns absolute temporal distance, relative temporal displacement, and natural-language supervision. (b) Value-isolation attention. Within each value-query group, repeated queries attend to one another and to the language–visual context, while remaining isolated from other value-query groups. Colored cells denote visible attention connections.
Table 2: Per-dataset trajectory-ranking results on the RBM-EVAL-OOD test suite, measured by Kendall’s τa (↑). Bold values indicate the best overall results. † denotes the best result among methods trained without explicit trajectory-level preference supervision, i.e., progress/value-only methods. Baseline results are taken from Robometer 16.
Method
USC Franka
USC Koch
USC Trossen
USC xArm
MIT Franka
UTD SO101
Average
GVL 19
0.250
−0.008
0.292
0.056
0.306
0.300
0.199
VLAC-2B 34
0.292
0.167
−0.111
0.167
−0.017
−0.033
0.077
VLAC-8B 34
0.271
0.064
−0.417
0.139
0.072
0.167
0.049
RoboDopamine 26
0.167
0.175
0.000
0.014
0.220
0.067
0.107
Dopamine-GRM-2.0-8B-Preview 26
0.479
0.442
0.333
0.431
0.431
0.700
0.453
RoboReward-4B 15
0.625
0.332
0.333
0.528
0.494
0.700
0.502
RoboReward-8B 15
0.625
0.264
0.389
0.347
0.396
0.767
0.465
Robometer (RoboReward data) 16
0.583
0.533
0.646
0.403
0.479
0.667
0.552
ReWiND 33
−0.125
0.336
0.028
−0.167
0.080
−0.067
0.014
Robometer (RBM-1M) 16
0.646
0.471
0.653
0.694
0.601
0.867
0.655
Robometer (Progress only) 16
0.083
0.231
0.333
0.389
0.183
0.533
0.292
RynnValue-4B
0.542
0.488
0.917
0.667†
0.473
0.933†
0.670
RynnValue-8B
0.667†
0.544†
1.000†
0.500
0.503†
0.833
0.675†
Figure 3: Instruction-trajectory confusion matrices. Each cell shows the predicted reward when an instruction (rows) is paired with a trajectory (columns); a well-grounded model concentrates mass on the diagonal. Values below each matrix report the normalized diagonal margin. All models are re-evaluated under a unified protocol from their publicly released weights.
Table 3: Ablation study on RBM-EVAL-OOD. We report Kendall’s τa across six out-of-distribution robot datasets. Shuffle denotes temporal-order shuffling, Isolation denotes value-isolation attention, Language denotes the auxiliary natural-language supervision, Random denotes random temporal sampling, and Relative denotes the relative modeling component.
Design Components
Kendall’s τa
Variant
Shuffle
Isolation
Language
Random
Relative
USC Franka
USC Koch
USC Trossen
USC xArm
MIT Franka
UTD SO101
Average
w/o Shuffle
✗
✓
✓
✓
✓
0.583
0.090
0.055
0.222
-0.017
0.200
0.189
w/o Isolation
✓
✗
✓
✓
✓
0.583
0.428
0.694
0.389
0.400
0.400
0.482
w/o Language
✓
✓
✗
✓
✓
0.250
0.491
0.819
0.361
0.501
0.800
0.537
Uniform Sampling
✓
✓
✓
✗
✓
0.375
0.400
0.305
0.250
0.310
0.633
0.379
w/o Relative
✓
✓
✓
✓
✗
0.667
0.587
0.639
0.639
0.464
0.767
0.627
Full Model (8B)
✓
✓
✓
✓
✓
0.667
0.544
1.000
0.500
0.503
0.833
0.675
Figure 4: Scaling episode volume vs. task diversity. Mean absolute temporal-distance error on a held-out validation set of unseen tasks. We independently scale two aspects of the training set: (orange) episode count with the full task set fixed, and (blue) task count with per-task episode counts fixed. Both curves converge to the same full-scale training set at 100%. Task diversity yields a consistently steeper error reduction than episode volume, which saturates early.
Table 4: Real-world reinforcement-learning results. We report success rates and the average number of action chunks over successful episodes. Average denotes the unweighted mean success rate across four tasks.
Algorithm
Baseline
Bread Basket Placement
Steak Serving with a Spatula
Box-in-Drawer Placement
Bimanual Box Transfer
Average Success
Success ↑
Avg. Steps ↓
Success ↑
Avg. Steps ↓
Success ↑
Avg. Steps ↓
Success ↑
Avg. Steps ↓
Online RL
RynnValue
45.0%
25.9±8.2
75.0%
18.6±13.1
70.0%
27.0±8.1
100.0%
22.8±4.7
72.5%
Robometer
35.0%
22.7±5.5
45.0%
15.2±2.7
65.0%
27.7±5.8
65.0%
25.6±7.4
52.5%
Sparse
40.0%
56.0±31.7
45.0%
18.4±4.9
40.0%
27.4±6.6
70.0%
23.5±2.7
48.8%
Offline RL
RynnValue
100.0%
16.8±3.1
90.0%
14.9±4.0
90.0%
14.9±4.0
50.0%
33.6±10.5
82.5%
Robometer
80.0%
18.9±2.7
80.0%
19.4±7.2
50.0%
27.3±6.3
45.0%
28.7±9.3
63.8%
Sparse
70.0%
26.1±9.1
20.0%
30.2±3.3
0.0%
–
0.0%
–
22.5%
SFT
–
70.0%
24.8±8.0
25.0%
18.6±6.2
0.0%
–
0.0%
–
23.8%
Figure 5: Temporal-value curve comparison on a real-world trajectory. Higher values indicate closer proximity to task completion. The highlighted interval marks a period of task regression where the robot moves away from a productive state; RynnValue responds with a sharp potential drop, whereas Robometer remains relatively flat.
Table 5: Representative annotation issues addressed by data curation. Non-English examples are described rather than reproduced verbatim to avoid introducing non-Roman fonts into the manuscript.
Issue
Representative annotation
Source
Non-English task annotation
Chinese-language manipulation instruction
Galaxea
Placeholder or truncated label
P, shirts, undefined
InternA1
Data-quality metadata
no robot motion, skip frame
OXE
Pure-motion instruction
move to the table, approach the cabinet
Galaxea
Figure 6: Representative demonstrations for real-world evaluation. Each row shows a sequence of observations from one manipulation task: Each row shows a sequence of observations from one manipulation task. These tasks cover object grasping, spatial manipulation, and articulated-object interaction.
Table 6: Summary of source-specific data curation. OXE and InternA1 are counted in episodes, while Galaxea and RoboCOIN are counted in annotated subtask segments. The total therefore denotes an aggregate number of trajectory units rather than a homogeneous episode count.
Source
Raw units
Curated units
Unit retention
Raw instructions
Curated instructions
Instruction retention
OXE (episode)
961,253
693,037
72.10%
180,290
180,090
99.89%
InternA1 (episode)
320,910
320,905
99.99%
350
348
99.43%
Galaxea (segment)
97,287
78,692
80.89%
12,685
10,909
86.00%
RoboCOIN (segment)
343,516
343,516
100.00%
1,642
1,642
100.00%
Total
1,722,966
1,436,150
83.35%
194,967
192,989
98.99%
Table 7: Real-world task instructions and dataset statistics. Success rate is computed over all collected trajectories for each task.
Task
Language instruction
Trajectories
Success rate
Success
Failure
Total
Bread Basket Placement
“Put the two pieces of bread in the basket.”
99
4
103
96.1%
Steak Serving with a Spatula
“Move the steak from the pan to the plate.”
98
4
102
96.1%
Box-in-Drawer Placement
“Put the box in the drawer and close it.”
101
3
104
97.1%
Bimanual Box Transfer
“Move the box from the right side to the left side.”
100
1
101
99.0%
Total
–
398
12
410
97.1%
Table 8: Camera streams used by the policy and RL encoders. Entries list the exact streams provided to each pathway.
Pathway
Single-arm tasks
Bimanual Box Transfer
VLA
left_side, left_wrist
left_side, left_wrist, right_wrist
IQL critic
left_side, left_wrist
left_side, left_wrist, right_wrist
DSRL actor–critic
left_side, left_wrist
left_side, right_side, left_wrist, right_wrist
Table 9: Offline IQL and SFT hyperparameters. All IQL reward variants share the same mixed-expertise dataset, sparse task reward, policy initialization, and optimization configuration. SFT uses the same policy optimizer and learning-rate schedule and is also trained for 10,000 steps per task.
Hyperparameter
Value
Base policy
π0.5 with flow matching; action dimension 32
Action horizon H
16
Batch size
64
Policy optimizer
AdamW with β1=0.9, β2=0.95, ϵ=10−8, weight decay 10−10, and gradient-norm clipping at 1.0
Learning-rate schedule
Cosine decay with 2,000 linear warm-up steps
Peak / final policy learning rate
3×10−5 / 3×10−6
Policy EMA decay
0.99
Training steps per task
10,000
Critic / value optimizer
Adam with a learning rate of 3×10−4
Offline discount γoff
0.99
Target update rate ρoff
0.005
Expectile parameter τe
0.8
Advantage temperature β
10.0
Maximum advantage weight wmax
100
Number of Q-functions KIQL
2; minimum aggregation
Critic and value encoder
ResNet-18 with GroupNorm and spatial softmax; 50-dimensional bottleneck
Critic and value hidden dimensions
(256,256)
Number of critic cameras
2 for single-arm tasks and 3 for the bimanual task
Critic input resolution
224×224 using the VLA preprocessing pipeline
Policy warm-up Nwarm
200 optimization steps with w(o,a)=1
Sparse task reward
−1 before task completion and 0 upon task completion
Potential-based shaping reward
rhshape=γoffΦh+1−Φh
Shaping coefficient κ
0.1 for RynnValue and 1.0 for Robometer; fixed across tasks
Sparse-reward baseline
κ=0
Image augmentation
Random cropping applied to both current and next observations; no color jitter
Table 10: Online DSRL hyperparameters. SAC operates in the latent space of the frozen VLA, which decodes latent variables into executable action chunks. All reward variants share the same optimization configuration and differ only in the potential source and shaping coefficient.
Hyperparameter
Value
Base policy
Frozen SFT checkpoint for Bread Basket Placement and Steak Serving with a Spatula; frozen Robometer offline-RL checkpoint for Box-in-Drawer Placement and Bimanual Box Transfer
Latent action space
z∈[−1,1]H×dz, with H=16 and dz=32
RL algorithm
SAC with automatic entropy tuning and initial temperature α0=1.0
Actor optimizer
Adam with a learning rate of 1×10−4
Critic optimizer
Adam with a learning rate of 3×10−4
Temperature optimizer
Adam with a learning rate of 3×10−4
Gradient clipping
None
Target entropy ℋ¯
−dim(z)
Online SAC discount γon
0.999
Target update rate ρon
0.005
Number of Q-functions KSAC
10; mean aggregation
Actor and critic hidden dimensions
(128,128,128)
Image encoder
Four-layer CNN with 32 channels per layer, strides (2,1,1,1), VALID padding, GroupNorm, spatial softmax, and a 50-dimensional bottleneck
SAC input resolution
64×64
Batch size
256
Update-to-data ratio
100
Training length
6,000 training steps
Online rollout trajectories
60 per task
Replay-buffer capacity
max(training steps/UTD,104)=104
Update frequency
After each episode
Exploration warm-up
Nnoise=2 episodes with Gaussian noise standard deviation σ=0.1
Minimum replay size Nstart
200 transitions
Maximum episode length
600 environment steps
Policy-decision frequency
Low-level control at 10 Hz, with one policy decision every q=10 environment steps
Number of DSRL cameras
2 for single-arm tasks and 4 for the bimanual task
Sparse task reward
−1 before task completion and 0 upon task completion
Shaping coefficient κ
0.1 for RynnValue, 1.0 for Robometer, and 0 for the sparse-reward baseline; fixed across tasks
Per-step shaping discount γs
0.999
Findings
On RBM-EVAL-OOD (six out-of-distribution datasets, 976 trajectories), RynnValue-8B achieved an average Kendall's τa of 0.675, exceeding Robometer's fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292); the smaller RynnValue-4B reached a comparable 0.670.
Across four real-world manipulation tasks (bread basket placement, steak serving with a spatula, box-in-drawer placement, bimanual box transfer), RynnValue-based rewards raised average online RL success from 52.5% (Robometer) to 72.5%, and offline RL success from 63.8% to 82.5%.
In offline RL, RynnValue reached 100% success on bread basket placement using an average of 16.8 action chunks, versus 80%/18.9 chunks for Robometer and 70%/24.8 chunks for the SFT policy; the SFT policy recorded zero successes on box-in-drawer placement and bimanual box transfer, while RynnValue-based RL solved both.
On box-in-drawer placement, online gains were limited for both reward models (Robometer 65%, RynnValue 70%, from a shared 50% starting checkpoint), attributed to the difficulty of judging precise gripper-box-drawer alignment from third-person RGB images alone.
Where it can be used
Building general-purpose reward or value models from large, heterogeneous robot datasets (different robots, viewpoints, tasks) where preference or progress annotation would be impractical to collect at scale
Providing reward signals for offline reinforcement learning (e.g., IQL) to distill useful policies out of mixed-quality demonstration datasets
Densifying sparse success/failure signals for online real-world reinforcement learning to speed up policy training
Zero-shot reward annotation for new tasks, robot embodiments, or camera viewpoints not seen during training, without target-domain fine-tuning
Limits and open work
The current model estimates temporal distance from a short window of sampled observations (8 frames), so extending it to longer horizons and streaming inference remains future work.
The training target assumes an approximately minimum-time objective, so task-specific costs such as energy use, safety, or precision are not yet incorporated.
When only third-person RGB views are available, the reward showed limited benefit for visually ambiguous, precision-sensitive tasks like aligning a gripper, box, and drawer.
Evaluation so far covers a dual-arm Franka platform and parallel-jaw-style manipulation; extending to dexterous hands and mobile manipulation settings is stated as future work, not yet validated.
Why it matters
Reward supervision is the core bottleneck for scaling robot reinforcement learning, and this approach removes the need for humans to hand-craft preference pairs or progress annotations, which is a major practical barrier to scaling data. The real-world experiments showing higher policy success rates suggest this isn't just a benchmark-only improvement but something that can meaningfully change robot training pipelines.
Terms in this paper
temporal distance · The remaining time from the current observation to the point where the task is completed; a directed cost-to-go rather than a progress fraction.
Kendall's τa · A statistic measuring how well a predicted ordering of items matches the true ordering; higher means better ranking agreement.
value-isolation attention · An attention constraint that stops value-prediction tokens for different observations from attending to each other, forcing each prediction to rely on its own visual evidence.
potential-based shaping · A technique that turns the change in a predicted potential (value) into a reward signal, densifying sparse rewards without changing the optimal policy.
IQL / DSRL · Implicit Q-Learning (an offline RL method) and Diffusion Steering via Reinforcement Learning (an online RL method), both used here to train robot policies with rewards produced by RynnValue.
Original abstract (English)
General-purpose reward models are increasingly the bottleneck for scaling robot learning, yet the recipe for learning value-related capabilities from large-scale heterogeneous corpora remains underexplored. Existing approaches tie supervision to task-internal anchors such as preferences or normalized progress, none of which transfer cleanly across embodiments and data sources. We introduce RynnValue, an open-source value foundation model for robotic manipulation that replaces these anchors with temporal distance, the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, RynnValue scales to over 7,000 hours and roughly 3M instruction-conditioned clips without preference or progress annotations. To make temporal-value learning reliable at scale, we combine random temporal sampling, temporal-order shuffling, and value-isolation attention, suppressing shortcuts that would leave predictions insensitive to failures and regressions. Trained without preference labels, RynnValue attains an average Kendall's tau_a of 0.675 on RBM-EVAL-OOD, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292), while generalizing zero-shot to unseen tasks, embodiments, and viewpoints. Converted into dense rewards via potential-based shaping, it raises real-world policy success from 52.5% to 72.5% online and from 63.8% to 82.5% offline. These results establish temporal distance as a scalable supervision target and practical reward interface for generalist robot policies.
