Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
arXiv:2608.187012026-08-18
A robot can 'succeed' at a task while secretly crushing the object, and this benchmark is built to catch that
SoftVTBench is a dataset of 4,000 robot demonstrations that records both what a robot policy sees and touches, plus a hidden physical measurement of how much a deformable object like bread or dough gets squished. Testing three robot control methods showed that in every single tested setup, some 'successful' pick-and-place actions actually squeezed or slipped the object beyond a safe limit. Giving robots a sense of touch generally helped them handle unfamiliar situations better, but did not automatically make them use that touch information well.
METAL MEDIA explanatory visual
A robot can 'succeed' at a task while secretly crushing the object, and this benchmark is built to catch that
01Built a dataset with 4,000 expert robot demonstrations across 40 pick-and-place tasks, using over 50 objects including soft items (like bakery goods) and hard 'twin' copies of the same shape for comparison.
02Recorded two separate streams per episode: what the robot can see and feel (camera images, touch-sensor images, robot arm position) versus a hidden physics simulation tracking exactly how much the object's shape changed, which the robot never sees.
03Defined a new scoring rule called Deformation-aware Success Rate (DSR) that only counts a task as truly successful if the robot both completes it AND keeps the object's squishing within a pre-calibrated safe range.
04Tested three robot control systems (Diffusion Policy, π0.5, FastWAM) and found that across all 12 tested configurations, between 0.7% and 24% of 'successful' attempts secretly violated the safe squishing limit.
05Adding touch sensing helped robots succeed more often when conditions changed (like different lighting or object weight), but touch alone didn't guarantee robots actually used it well when conditions were normal.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
Built a dataset with 4,000 expert robot demonstrations across 40 pick-and-place tasks, using over 50 objects including soft items (like bakery goods) and hard 'twin' copies of the same shape for comparison.
Recorded two separate streams per episode: what the robot can see and feel (camera images, touch-sensor images, robot arm position) versus a hidden physics simulation tracking exactly how much the object's shape changed, which the robot never sees.
Defined a new scoring rule called Deformation-aware Success Rate (DSR) that only counts a task as truly successful if the robot both completes it AND keeps the object's squishing within a pre-calibrated safe range.
Tested three robot control systems (Diffusion Policy, π0.5, FastWAM) and found that across all 12 tested configurations, between 0.7% and 24% of 'successful' attempts secretly violated the safe squishing limit.
Adding touch sensing helped robots succeed more often when conditions changed (like different lighting or object weight), but touch alone didn't guarantee robots actually used it well when conditions were normal.
Table 1: Benchmark positioning. Complete Task: the benchmark scores multi-stage manipulation (approach, grasp, transport, place) rather than an isolated grasp. 3D Deformable: the manipulated objects are volumetric soft bodies, as opposed to cloth or rope (❍). Policy-Visible Touch: tactile observations are available to the policy as input. Evaluator-Only Deformation Scoring: object deformation is measured from physical state hidden from the policy, and that measurement enters the success criterion, so an episode can complete the task and still be scored as a failure; ❍ marks benchmarks that constrain interaction on the input side (e.g. a force budget) without scoring the resulting deformation. Symbols denote full (✓), partial (❍), or no (✗) support. Benchmarks that expose touch do not score deformation from hidden state, and those that do evaluate isolated grasps; only SoftVTBench provides all four.
Benchmark
Complete Task
3D Deformable
Policy-Visible Touch
Evaluator-Only Deformation Scoring
LIBERO 20
✓
✗
✗
✗
ManiSkill2 10
✓
✓
✗
✗
SoftGym 19
✓
❍
✗
✗
MoDeSuite 42
✓
❍
✗
✗
DefGraspSim 13
✗
✓
✗
✓
SoGraB 9
✗
✓
✗
✓
VTDexManip 21
✓
✗
✓
✗
ManiFeel 23
✓
✗
✓
✗
Tabero 36
✓
✗
✓
❍
SoftVTBench
✓
✓
✓
✓
Table 2: Dataset composition and evaluation splits. The four suites form a matched 2×2 design over object type and variation axis. OOD evaluation uses nine single-factor conditions on the two deformable suites.
Suite
Object Type
Variation Axis
#Tasks
#Demos
ID Eval Episodes
OOD Conditions
Object-Soft
Deformable
Object identity
10
1,000
500
9
Spatial-Soft
Deformable
Spatial layout
10
1,000
500
9
Object-Rigid
Rigid twin
Object identity
10
1,000
500
–
Spatial-Rigid
Rigid twin
Spatial layout
10
1,000
500
–
Total
–
–
40
4,000
2,000
–
Table 3: In-distribution Task Success Rate and Deformation-aware Success Rate on the deformable suites (%). DSR is lower than TSR in every one of the twelve configurations; the difference is the fraction of rollouts that reach the target while leaving the calibrated interaction safety zone.
Object-Soft
Spatial-Soft
Model
Input
TSR
DSR
TSR
DSR
Diffusion Policy
VO-C
37.4
33.6
15.6
13.4
VT-C
40.0
30.4
33.0
25.0
π0.5
VO-C
41.6
38.4
26.0
22.6
VT-C
41.4
35.0
27.6
22.0
FastWAM
VO-C
62.0
58.0
37.0
36.6
VT-C
57.6
54.4
56.4
56.0
Table 4: Task success on deformable assets and their rigid twins (TSR, %). Twins match geometry, appearance, and mass under the same layouts and instructions, with negligible deformation, so a rigid–deformable difference isolates the effect of deformability. Because deformation is negligible on the twins, DSR coincides with TSR there and only TSR is reported.
Object variation
Spatial variation
Model
Input
Rigid
Soft
Rigid
Soft
Diffusion Policy
VO-C
40.0
37.4
14.0
15.6
VT-C
35.0
40.0
11.0
33.0
π0.5
VO-C
60.0
41.6
50.4
26.0
VT-C
59.6
41.4
54.0
27.6
FastWAM
VO-C
64.0
62.0
25.0
37.0
VT-C
61.6
57.6
30.0
56.4
Table 5: Matched sensing–control ablation for π0.5 (%). VO/VT denote vision-only and visuo-tactile inputs; B/C denote binary and continuous gripper control. Crossing the two factors separates a gain due to touch from a gain due to finer actuation.
Object-Soft
Spatial-Soft
Configuration
TSR
DSR
TSR
DSR
VO-B
30.2
27.2
34.2
20.0
VO-C
41.6
38.4
26.0
22.6
VT-B
41.0
28.0
30.0
21.4
VT-C
41.4
35.0
27.6
22.0
Table 6: Out-of-distribution Task Success Rate and Deformation-aware Success Rate on the deformable suites (%), pooled over the nine held-out conditions of Section 3.1. Δ is the change relative to the corresponding in-distribution entry for the same model, input, and suite in Table 3.
Object-Soft
Spatial-Soft
Model
Input
TSR ↑ (Δ vs. ID)
DSR ↑ (Δ vs. ID)
TSR ↑ (Δ vs. ID)
DSR ↑ (Δ vs. ID)
Diffusion Policy
VO-C
29.2 (−8.2)
26.6 (−7.0)
11.0 (−4.6)
8.8 (−4.6)
VT-C
31.2 (−8.8)
25.0 (−5.4)
25.2 (−7.8)
17.8 (−7.2)
π0.5
VO-C
35.8 (−5.8)
33.2 (−5.2)
24.4 (−1.6)
19.4 (−3.2)
VT-C
41.0 (−0.4)
34.2 (−0.8)
28.4 (+0.8)
23.2 (+1.2)
FastWAM
VO-C
54.4 (−7.6)
53.8 (−4.2)
27.8 (−9.2)
27.2 (−9.4)
VT-C
55.8 (−1.8)
55.8 (+1.4)
39.4 (−17.0)
38.8 (−17.2)
Table 7: Contents of the supplementary sections in document order.
Supplementary section
Contents
Sec. 2: Simulation and implementation details
Simulator, robot, sensing, FEM, and compute configuration.
Sec. 3: Assets and interaction safety zones
Asset inventory, rigid twins, per-object calibration, and the calibration percentile.
Sec. 4: Task suite details
Suite design, task mapping, and variation factors.
Sec. 5: Baseline training details
Training and inference settings for all evaluated policy families.
Sec. 6: Per-condition results
Out-of-distribution results per policy family and condition.
Sec. 7: Additional analyses
Qualitative cases and stiffness-conditioned deformation statistics.
Sec. 8: Tactile simulation pipeline and scope
Tactile rendering stages, recorded outputs, and validity boundary.
Table 8: Simulation and sensing configuration.
Item
Value
Simulator
Isaac Sim 4.5.0 / Isaac Lab 0.41.3, PhysX 5 GPU pipeline
Physics / control rate
60 Hz physics, decimation 3, 20 Hz control
Robot
Franka arm with Panda parallel-jaw gripper
Controller
Task-space differential inverse kinematics
End-effector action
Absolute pose target: 3D position and 3D axis-angle orientation
Gripper action
Normalized closure command; continuous and binary encodings
Finger friction
Static μs=1.5, dynamic μd=1.2, max combine mode
Camera views
Third-person 1024×1024 and wrist 512×512, resized to 224×224
Tactile sensor
GelSight Mini via TacEx; Taxim optics and FOTS markers
Tactile streams
Tactile RGB and 11×9 marker-motion field, 320×240
FEM model
PhysX soft body with corotational linear elasticity
FEM solver
Hex resolution 6, 64 position iterations, damping 2.5
Collection / evaluation
4× NVIDIA L20 GPU
Training
NVIDIA A100-80GB; per-family counts in Table 12
Table 9: Interaction safety zones of the ten deformable assets. τo is the object-specific 90th-percentile threshold used by the released evaluator, expressed as a percentage of the reference bounding-box diagonal.
Asset
gomin
gomax
Span (mm)
τo/diag. (%)
soft_pastry001
0.4000
0.8000
26.21
7.6
soft_pastry002
0.6547
0.6900
2.56
9.3
soft_pastry003
0.4400
0.8000
13.26
8.6
soft_pastry005
0.4700
0.5800
5.87
10.7
soft_pastry010
0.4200
0.6600
24.82
8.3
soft_pastry011
0.4400
0.5800
10.63
7.1
soft_stw_cube_hq
0.3200
0.6200
19.43
11.2
soft_stw_cuboid_hq
0.3800
0.4400
10.95
9.8
soft_stw_cylinder_hq
0.2900
0.6600
21.95
9.8
soft_stw_sphere_hq
0.3700
0.6300
20.77
9.7
Table 10: Task specifications for the deformable suites. Task identifiers follow the released suite configuration and are zero-indexed; instructions are reproduced verbatim from that configuration, and object descriptions are therefore not normalized across suites.
Suite
Task
Instruction
Asset
Object-Soft
0
Pick up the white swirled pastry and place it in the basket.
soft_pastry001
1
Pick up the panda-face pastry and place it in the basket.
soft_pastry002
2
Pick up the small chocolate rectangular pastry and place it in the basket.
soft_pastry003
3
Pick up the soft cream-colored cube and place it in the basket.
soft_stw_cube_hq
4
Pick up the yellow square layered pastry and place it in the basket.
soft_pastry005
5
Pick up the soft tan cylinder and place it in the basket.
soft_stw_cylinder_hq
6
Pick up the golden pastry and place it in the basket.
soft_pastry011
7
Pick up the golden knotted pastry and place it in the basket.
soft_stw_cuboid_hq
8
Pick up the soft red ball and place it in the basket.
soft_stw_sphere_hq
9
Pick up the orange pumpkin-shaped pastry and place it in the basket.
soft_pastry010
Spatial-Soft
0
Pick up the left white swirled pastry and place it on the plate.
soft_pastry001
1
Pick up the right panda-face pastry and place it on the plate.
soft_pastry002
2
Pick up the left small chocolate rectangular pastry and place it on the plate.
soft_pastry003
3
Pick up the left soft cream-colored cube and place it on the plate.
soft_stw_cube_hq
4
Pick up the left yellow square layered pastry and place it on the plate.
soft_pastry005
5
Pick up the left soft tan cylinder and place it on the plate.
soft_stw_cylinder_hq
6
Pick up the right golden pastry and place it on the plate.
soft_pastry011
7
Pick up the left soft cream-colored rectangular block and place it on the plate.
soft_stw_cuboid_hq
8
Pick up the left soft red ball and place it on the plate.
soft_stw_sphere_hq
9
Pick up the left orange pumpkin-shaped pastry and place it on the plate.
soft_pastry010
Table 11: Variation factors: training ranges and held-out out-of-distribution values. The nine out-of-distribution conditions are the three held-out levels of dome-light intensity, mass, and Young’s modulus.
Family
Factor
Training range
Held-out values
Visual
Dome-light intensity
135 (fixed nominal)
67.5, 180, 270
Spatial
Object placement
Per-task recorded support
—
Target placement
Per-task recorded support
—
Robot initialization
Per-task recorded support
—
Physical
Mass scale
×1.0 (nominal)
×1.25, ×1.75, ×2.5
Young’s modulus scale
×1.0 (nominal)
×0.5, ×0.8, ×2.0
Table 12: Training and inference configurations for the three baseline policy families.
Setting
Diffusion Policy
𝝅0.5
FastWAM
Model and compute
Policy backbone
Image-conditioned diffusion U-Net
—
—
GPUs
1× NVIDIA A100-80GB
8× NVIDIA A100-80GB
8× NVIDIA A100-80GB
Precision
—
bfloat16 mixed precision
bfloat16
Distributed training
—
FSDP
ZeRO-1
Fine-tuning method
—
LoRA
—
LoRA rank / alpha (backbone)
—
16/16
—
LoRA rank / alpha (action expert)
—
32/32
—
Training and optimization
Training duration
20 epochs
7k steps
10 epochs
Batch size (micro / global)
—
32/256
—
Gradient accumulation
1
1
1
Optimizer
AdamW
AdamW
AdamW
Learning rate
1×10−4
2.5×10−5 (peak)
1×10−4
AdamW betas
(0.95,0.999)
(0.9,0.95)
(0.9,0.95)
Weight decay
1×10−6
1×10−10
1×10−2
LR schedule
Cosine decay; 100-step warmup
Cosine decay; 1k-step warmup
Cosine decay; 5% warmup
Gradient clipping
—
1.0
1.0
EMA
Enabled
Disabled
—
Closed-loop inference
Denoising timesteps (train / inference)
DDPM: 100/100
—
Flow matching: 1000/10
Action horizon
16
50
32
Actions executed per replanning
8
10
10
Table 13: Differences between the vision-only and visuo-tactile policy variants. Trainable parameter counts include all optimized modules and exclude frozen model weights and preprocessing components.
Table 14: Per-condition OOD results for Diffusion Policy (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.
Object-Soft
Spatial-Soft
VO-C
VT-C
VO-C
VT-C
Factor
Level
TSR
DSR
TSR
DSR
TSR
DSR
TSR
DSR
Lighting
×0.5
0
0
1
1
0
0
19
14
×1.33
37
33
36
30
18
15
28
16
×2.0
0
0
0
0
16
9
17
13
Mass
×1.25
41
38
42
34
5
5
18
17
×1.75
40
35
43
38
6
6
18
17
×2.5
36
33
33
27
5
5
17
15
Young’s
×0.5
33
32
41
30
19
13
38
23
×0.8
37
31
45
33
17
16
34
20
×2.0
39
37
39
32
13
10
37
25
Table 15: Per-condition OOD results for π0.5 (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.
Object-Soft
Spatial-Soft
VO-C
VT-C
VO-C
VT-C
Factor
Level
TSR
DSR
TSR
DSR
TSR
DSR
TSR
DSR
Lighting
×0.5
37
34
39
35
30
24
38
33
×1.33
37
34
40
33
29
25
33
28
×2.0
38
35
43
34
28
22
31
26
Mass
×1.25
37
37
44
40
13
13
18
16
×1.75
31
31
42
35
18
14
17
13
×2.5
26
26
36
29
14
11
17
14
Young’s
×0.5
34
31
41
31
29
18
36
26
×0.8
45
40
42
33
29
22
31
26
×2.0
38
32
42
37
29
26
35
28
Table 16: Per-condition OOD results for FastWAM (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.
Object-Soft
Spatial-Soft
VO-C
VT-C
VO-C
VT-C
Factor
Level
TSR
DSR
TSR
DSR
TSR
DSR
TSR
DSR
Lighting
×0.5
63
63
54
54
30
29
39
37
×1.33
62
60
57
57
33
33
48
47
×2.0
47
45
56
56
33
33
45
45
Mass
×1.25
59
59
61
61
17
17
21
21
×1.75
46
46
55
55
16
16
24
24
×2.5
38
38
49
49
16
16
23
23
Young’s
×0.5
58
58
56
56
36
35
53
52
×0.8
58
58
58
58
29
28
53
53
×2.0
58
58
57
57
39
38
48
47
Table 17: Completion and deformation statistics under stiffness shift on Object-Soft (rates in %).
Policy
E scale
Input
TSR
DSR
Rmax median [IQR]
N
DP
×1.0
VO-C
42
39
0.63 [0.47, 0.75]
100
VT-C
42
30
0.69 [0.48, 0.97]
100
×0.5
VO-C
33
32
0.66 [0.06, 0.76]
100
VT-C
41
30
0.76 [0.57, 0.98]
100
×0.8
VO-C
37
31
0.63 [0.06, 0.79]
100
VT-C
45
33
0.75 [0.54, 0.96]
100
×2.0
VO-C
39
37
0.60 [0.37, 0.71]
100
VT-C
39
32
0.63 [0.41, 0.92]
100
π0.5
×1.0
VO-C
45
42
0.46 [0.15, 0.60]
100
VT-C
41
35
0.63 [0.51, 0.88]
100
×0.5
VO-C
34
31
0.56 [0.35, 0.83]
100
VT-C
41
31
0.71 [0.55, 0.92]
100
×0.8
VO-C
45
40
0.50 [0.31, 0.69]
100
VT-C
42
33
0.68 [0.54, 0.96]
100
×2.0
VO-C
38
32
0.46 [0.27, 0.64]
100
VT-C
42
37
0.59 [0.47, 0.79]
100
FastWAM
×1.0
VO-C
64
64
0.49 [0.38, 0.59]
100
VT-C
58
58
0.50 [0.39, 0.57]
100
×0.5
VO-C
58
58
0.55 [0.42, 0.63]
100
VT-C
56
56
0.54 [0.44, 0.60]
100
×0.8
VO-C
58
58
0.49 [0.40, 0.62]
100
VT-C
58
58
0.50 [0.41, 0.57]
100
×2.0
VO-C
58
58
0.47 [0.33, 0.58]
100
VT-C
57
57
0.48 [0.39, 0.57]
100
Why it matters
Many real-world robot jobs involve handling fragile items like food, packaging, or medical supplies, where 'did it get where it needed to go' isn't enough if it arrived damaged. This work gives researchers a shared, checkable way to measure not just whether a robot finished a task but whether it handled the object gently, which matters for building robots that can be trusted around delicate items.
Terms in this paper
Visuo-tactile · Combining regular camera vision with touch-sensor data so a robot can both see and feel what it's interacting with.
Deformation-aware Success Rate (DSR) · A stricter success score that only credits a robot if it finishes the task without squeezing or damaging the object beyond a safe limit.
FEM (finite-element) state · A detailed physics simulation tracking the exact shape and internal movement of an object, used here as a hidden 'ground truth' to check for damage.
In-distribution / Out-of-distribution (ID/OOD) · ID means testing under familiar conditions similar to training; OOD means testing under new conditions the robot wasn't trained on, like different lighting or object stiffness.
Rigid twin · A hard, non-squishable copy of a soft object with the same shape, size, and weight, used to isolate the effect of an object being deformable.
Figures we cannot republish
Figure 1: Overview of SoftVTBench. Top: 4,000 demonstrations over four diagnostic suites and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. Bottom left: controlled visual and physical shifts for closed-loop generalization evaluation. Bottom center: the physical-interaction regimes captured by touch and evaluated from object deformation, from slip-prone loose grasps to excessive compression. Bottom right: synchronized visual, tactile, proprioceptive, language, and action streams.
Figure 2: SoftVTBench construction and evaluation pipeline. Stage 1 constructs matched rigid–deformable objects and calibrates object-specific interaction constraints. Stage 2 generates controlled tasks and records policy-visible observations separately from evaluator-only physical states. Stage 3 applies automatic quality control and human verification, producing the released training, ID, and OOD splits. Task success and deformation compliance are labeled separately; DSR combines them during benchmark evaluation.
Figure 3: Tactile observations across deformable assets. Each panel shows a tactile RGB image with its marker-motion overlay at the moment of grasping, for the ten Object-Soft tasks. Objects that differ in geometry, compliance, and contact area produce distinct contact patches and shear fields under the same pick-and-place instruction. These are the cues available to a visuo-tactile policy and absent from external RGB views.
Figure 4: Task Success Rate under out-of-distribution shifts, resolved by factor. Rows correspond to the Object-Soft and Spatial-Soft suites; column groups correspond to Diffusion Policy, π0.5, and FastWAM, each resolved into illumination, object-mass, and Young’s-modulus shifts.
Figure 5: Illustrative rollouts on three deformable objects across Object-Soft and Spatial-Soft. Each row shows synchronized third-person, wrist, and tactile observations at approach, peak interaction, and placement, followed by the normalized deformation trace Rt. Circle, diamond, and square indicate the displayed steps. All rollouts achieve task success (TSR=1). Blue trajectories satisfy the calibrated deformation tolerance (DSR=1), while orange trajectories exceed Rt=1 despite success (DSR=0). Rows are independently sampled and not matched-initial-state comparisons.
Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the task while allowing slip or causing excessive compression. A primary bottleneck is the absence of visuo-tactile datasets that pair policy-visible contact observations with independent physical ground truth over complete tasks. We introduce SoftVTBench, a visuo-tactile dataset for physical-interaction-aware deformable-object manipulation. It contains 4,000 expert demonstrations and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. At 20 Hz, each episode synchronizes multi-view RGB, dual-finger tactile RGB and marker motion, proprioception, language, and binary and continuous gripper actions, alongside evaluator-only finite-element (FEM) states. Building upon this dataset, we establish a closed-loop benchmark that uses fixed object-specific calibration to define the Deformation-aware Success Rate (DSR), which counts a rollout as successful only when it completes the task and keeps peak normalized deformation within tolerance. Across Diffusion Policy, π_{0.5}, and FastWAM, all 12 in-distribution configurations contain successful rollouts that violate the deformation tolerance, accounting for 0.7--24% of each configuration's successes. Under distribution shift, visuo-tactile variants achieve higher task success in all six policy--suite comparisons and higher DSR in five, whereas their in-distribution benefits are mixed. These results show that making touch available does not by itself ensure effective multimodal fusion. SoftVTBench therefore provides a common visuo-tactile resource for studying not only whether a policy succeeds, but how it physically interacts with deformable objects and when touch improves that interaction.
