Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning
Teaching robots to pick out only the useful moves from messy, mixed-quality recorded data
When robots are pretrained on pre-collected datasets that mix successful, failed, and noisy behaviors, older methods fail to tell them apart, ending up with a cluttered library of skills. QDOS estimates how valuable each short segment of behavior was and uses that to weight learning, extracting skills that are both diverse and high quality. Across manipulation and locomotion tasks, this let QDOS reach goals faster and achieve higher final performance than the strongest prior method, SUPE.
METAL MEDIA explanatory visual
Teaching robots to pick out only the useful moves from messy, mixed-quality recorded data
- 01Standard approaches like trajectory VAEs treat every behavior segment in the dataset equally, so good moves, failed attempts, and noise all get baked into a tangled skill space
- 02QDOS estimates each segment's advantage (how much better it was than average) using a method called IQL, then weights both skill extraction and diversity objectives by this advantage
- 03It reuses the same offline dataset twice: once to pretrain the low-level skills, and again to fill the online replay buffer through pseudo-labeling, giving the high-level policy a head start
- 04In the complex antsoccer-arena task, QDOS reached a normalized return of 0.80, far above SUPE's 0.27, and in kitchen-mixed it hit a perfect 4.00 versus SUPE's 3.40
- 05QDOS also found the goal in far fewer training steps than SUPE across tasks, showing it explores more efficiently
What they did
- Standard approaches like trajectory VAEs treat every behavior segment in the dataset equally, so good moves, failed attempts, and noise all get baked into a tangled skill space
- QDOS estimates each segment's advantage (how much better it was than average) using a method called IQL, then weights both skill extraction and diversity objectives by this advantage
- It reuses the same offline dataset twice: once to pretrain the low-level skills, and again to fill the online replay buffer through pseudo-labeling, giving the high-level policy a head start
- In the complex antsoccer-arena task, QDOS reached a normalized return of 0.80, far above SUPE's 0.27, and in kitchen-mixed it hit a perfect 4.00 versus SUPE's 3.40
- QDOS also found the goal in far fewer training steps than SUPE across tasks, showing it explores more efficiently
| Environment | BC[11][26] | IQL[11][26] | SUPE | QDOS (α=0.2) | QDOS (α=0.3) |
|---|---|---|---|---|---|
| antmaze-large | 41±7 | 64±10 | 83.0±21.0 | 86.0±6.0¯ | 96.0±6.0 |
| antsoccer-arena | 5±1 | 50±2 | 27.0±31.0 | 73.0±31.0¯ | 80.0±10.0 |
| humanoidmaze-medium | 8±2 | 27±2 | 97.0±6.0¯ | 97.0±6.0¯ | 100.0±0.0 |
| kitchen-mixed | 51.5 | 51.0 | 85±13.25¯ | 100.00±0.00 | 83.25±28.75 |
| kitchen-partial | 38.0 | 46.3 | 83.25±14.50 | 85.75±24.75 | 82.50±30.25 |
| scene-task1 | 5±1 | 51±4 | 100.0±0.0 | 100.0±0.0 | 100.0±0.0 |
| Environment | Method | Steps |
|---|---|---|
| antmaze-large | SUPE | 13334±5774 |
| QDOS (α=0.2) | 𝟏𝟏𝟔𝟔𝟖±𝟗𝟒𝟔𝟓 | |
| humanoidmaze | SUPE | 23334±6292 |
| QDOS (α=0.3) | 𝟏𝟗𝟏𝟔𝟖±𝟖𝟎𝟑𝟔 | |
| scene | SUPE | 5834±1443 |
| QDOS (α=0.2) | 𝟓𝟎𝟎𝟏±𝟎 |
| Parameter Name | Value |
|---|---|
| Batch size | 256 |
| Optimizer | Adam |
| Learning rate | 3×10−4 |
| GRU Hidden Size | 256 (antmaze, kitchen) |
| 512 (antsoccer, humanoidmaze, scene) | |
| GRU Layers | 2 (antmaze, kitchen) |
| 3 (antsoccer, humanoidmaze, scene) | |
| KL Coefficient (OPENβ) | 0.1 (antmaze, humanoid, kitchen) |
| 0.2 (scene) | |
| Latent Dimension (z) | 8 |
| Segment Length (H) | 20 |
Why it matters
Real-world robot datasets are rarely clean, they typically mix successes, failures, and noise, so a method that reliably filters this out makes offline-to-online learning more practical. This can reduce costly real-world trial and error while still boosting final task performance.
Terms in this paper
- Offline RL · Learning a policy purely from a fixed, pre-collected dataset without interacting with the environment
- VAE (Variational Autoencoder) · An unsupervised model that compresses data into a latent representation and reconstructs it back
- Advantage · A measure of how much better an action or behavior segment performed compared to average
- IQL (Implicit Q-Learning) · An offline RL technique that estimates value functions without querying actions outside the dataset
- Pseudo-labeling · Using a trained encoder to assign inferred skill labels and reward estimates to offline data segments
Original abstract (English)
Recent studies investigate how to leverage pre-collected datasets to improve the policy performance and sample efficiency of RL. One promising approach to achieve this goal is to employ a two-stage strategy: In the first stage, diverse skills are extracted as a low-level policy from a given dataset, and a high-level policy is trained to solve a specific task in the second stage. Typically, extraction of the low-level policy is performed based on unsupervised learning such as trajectory VAE. However, a limitation of this approach is that the quality of the low-level policy highly depends on the quality of the dataset. To address this issue, we introduce QDOS (Quality-Diversity Offline Skill learning), a unified pipeline for robust offline-to-online learning. Our approach incorporates an Advantage-Weighted Quality-Diversity pretraining objective, which weights the skill extraction and diversity objectives by the estimated advantage of each trajectory segment. This approach allows the model to extract diverse and high-value skills. By providing robust and task-relevant skill representations, QDOS significantly improves the quality of the embedded skill space used by the low-level policy. We further integrate this with a dual dataset reuse strategy, where offline data is used both for skill pretraining and for populating the online replay buffer via pseudo-labeling. Experiments demonstrate that QDOS significantly outperforms strong baselines in structured manipulation tasks and unstructured locomotion tasks, confirming its ability to accelerate exploration and improve final returns in challenging sparse-reward domains.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one
Latest from METAL MEDIA
Figures: Tanachai Anakewat et al., arXiv:2608.19684, arxiv-nonexclusive