SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
로봇이 물건을 짓눌러도 작업은 '성공'으로 기록될 수 있다는 문제를 잡아내는 벤치마크
SoftVTBench는 로봇이 무엇을 보고 만졌는지와, 부드러운 물체(빵이나 반죽 등)가 실제로 얼마나 눌렸는지를 별도로 기록한 4000개의 로봇 시연 데이터셋이다. 세 가지 로봇 제어 방식을 테스트한 결과, 테스트한 모든 조합에서 겉보기에는 성공한 집기-놓기 동작 중 일부가 실제로는 물체를 안전 범위 이상으로 짓누르거나 미끄러뜨린 것으로 드러났다. 촉각 정보를 추가하면 낯선 상황에서는 대체로 도움이 되었지만, 훈련 환경과 비슷한 상황에서는 촉각을 잘 활용하는지 여부가 일정하지 않았다.
METAL MEDIA 해설 도표
로봇이 물건을 짓눌러도 작업은 '성공'으로 기록될 수 있다는 문제를 잡아내는 벤치마크
0150개 이상의 물체(빵집 스타일의 부드러운 물체와 같은 모양의 딱딱한 '쌍둥이' 물체 포함)를 사용해 40개의 집기-놓기 작업에 걸쳐 4000개의 전문가 로봇 시연 데이터를 구축했다.
02각 에피소드마다 로봇이 실제로 보고 느낀 정보(카메라 영상, 촉각 센서 영상, 로봇 팔 자세)와, 로봇에게는 숨겨진 물리 시뮬레이션 기반의 실제 변형 정도를 별도의 두 흐름으로 기록했다.
03작업을 완료했을 뿐 아니라 물체의 눌림 정도가 사전에 물체별로 보정된 안전 범위 안에 머물러야만 성공으로 인정하는 새로운 평가 지표인 변형 인식 성공률(DSR)을 도입했다.
04Diffusion Policy, π0.5, FastWAM 세 가지 로봇 제어 방식을 테스트한 결과, 테스트한 12개 조합 모두에서 '성공'으로 기록된 시도 중 0.7%에서 24%가 실제로는 안전 눌림 한도를 넘긴 것으로 나타났다.
05촉각 감각을 추가하면 조명이나 물체 무게 등 훈련 때와 다른 조건에서는 성공률이 더 높아지는 경향을 보였지만, 훈련 조건과 비슷한 상황에서는 촉각 정보를 추가한다고 해서 항상 더 나은 결과가 나오지는 않았다.
METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.
무엇을 했나
50개 이상의 물체(빵집 스타일의 부드러운 물체와 같은 모양의 딱딱한 '쌍둥이' 물체 포함)를 사용해 40개의 집기-놓기 작업에 걸쳐 4000개의 전문가 로봇 시연 데이터를 구축했다.
각 에피소드마다 로봇이 실제로 보고 느낀 정보(카메라 영상, 촉각 센서 영상, 로봇 팔 자세)와, 로봇에게는 숨겨진 물리 시뮬레이션 기반의 실제 변형 정도를 별도의 두 흐름으로 기록했다.
작업을 완료했을 뿐 아니라 물체의 눌림 정도가 사전에 물체별로 보정된 안전 범위 안에 머물러야만 성공으로 인정하는 새로운 평가 지표인 변형 인식 성공률(DSR)을 도입했다.
Diffusion Policy, π0.5, FastWAM 세 가지 로봇 제어 방식을 테스트한 결과, 테스트한 12개 조합 모두에서 '성공'으로 기록된 시도 중 0.7%에서 24%가 실제로는 안전 눌림 한도를 넘긴 것으로 나타났다.
촉각 감각을 추가하면 조명이나 물체 무게 등 훈련 때와 다른 조건에서는 성공률이 더 높아지는 경향을 보였지만, 훈련 조건과 비슷한 상황에서는 촉각 정보를 추가한다고 해서 항상 더 나은 결과가 나오지는 않았다.
Table 1: Benchmark positioning. Complete Task: the benchmark scores multi-stage manipulation (approach, grasp, transport, place) rather than an isolated grasp. 3D Deformable: the manipulated objects are volumetric soft bodies, as opposed to cloth or rope (❍). Policy-Visible Touch: tactile observations are available to the policy as input. Evaluator-Only Deformation Scoring: object deformation is measured from physical state hidden from the policy, and that measurement enters the success criterion, so an episode can complete the task and still be scored as a failure; ❍ marks benchmarks that constrain interaction on the input side (e.g. a force budget) without scoring the resulting deformation. Symbols denote full (✓), partial (❍), or no (✗) support. Benchmarks that expose touch do not score deformation from hidden state, and those that do evaluate isolated grasps; only SoftVTBench provides all four.
Benchmark
Complete Task
3D Deformable
Policy-Visible Touch
Evaluator-Only Deformation Scoring
LIBERO 20
✓
✗
✗
✗
ManiSkill2 10
✓
✓
✗
✗
SoftGym 19
✓
❍
✗
✗
MoDeSuite 42
✓
❍
✗
✗
DefGraspSim 13
✗
✓
✗
✓
SoGraB 9
✗
✓
✗
✓
VTDexManip 21
✓
✗
✓
✗
ManiFeel 23
✓
✗
✓
✗
Tabero 36
✓
✗
✓
❍
SoftVTBench
✓
✓
✓
✓
Table 2: Dataset composition and evaluation splits. The four suites form a matched 2×2 design over object type and variation axis. OOD evaluation uses nine single-factor conditions on the two deformable suites.
Suite
Object Type
Variation Axis
#Tasks
#Demos
ID Eval Episodes
OOD Conditions
Object-Soft
Deformable
Object identity
10
1,000
500
9
Spatial-Soft
Deformable
Spatial layout
10
1,000
500
9
Object-Rigid
Rigid twin
Object identity
10
1,000
500
–
Spatial-Rigid
Rigid twin
Spatial layout
10
1,000
500
–
Total
–
–
40
4,000
2,000
–
Table 3: In-distribution Task Success Rate and Deformation-aware Success Rate on the deformable suites (%). DSR is lower than TSR in every one of the twelve configurations; the difference is the fraction of rollouts that reach the target while leaving the calibrated interaction safety zone.
Object-Soft
Spatial-Soft
Model
Input
TSR
DSR
TSR
DSR
Diffusion Policy
VO-C
37.4
33.6
15.6
13.4
VT-C
40.0
30.4
33.0
25.0
π0.5
VO-C
41.6
38.4
26.0
22.6
VT-C
41.4
35.0
27.6
22.0
FastWAM
VO-C
62.0
58.0
37.0
36.6
VT-C
57.6
54.4
56.4
56.0
Table 4: Task success on deformable assets and their rigid twins (TSR, %). Twins match geometry, appearance, and mass under the same layouts and instructions, with negligible deformation, so a rigid–deformable difference isolates the effect of deformability. Because deformation is negligible on the twins, DSR coincides with TSR there and only TSR is reported.
Object variation
Spatial variation
Model
Input
Rigid
Soft
Rigid
Soft
Diffusion Policy
VO-C
40.0
37.4
14.0
15.6
VT-C
35.0
40.0
11.0
33.0
π0.5
VO-C
60.0
41.6
50.4
26.0
VT-C
59.6
41.4
54.0
27.6
FastWAM
VO-C
64.0
62.0
25.0
37.0
VT-C
61.6
57.6
30.0
56.4
Table 5: Matched sensing–control ablation for π0.5 (%). VO/VT denote vision-only and visuo-tactile inputs; B/C denote binary and continuous gripper control. Crossing the two factors separates a gain due to touch from a gain due to finer actuation.
Object-Soft
Spatial-Soft
Configuration
TSR
DSR
TSR
DSR
VO-B
30.2
27.2
34.2
20.0
VO-C
41.6
38.4
26.0
22.6
VT-B
41.0
28.0
30.0
21.4
VT-C
41.4
35.0
27.6
22.0
Table 6: Out-of-distribution Task Success Rate and Deformation-aware Success Rate on the deformable suites (%), pooled over the nine held-out conditions of Section 3.1. Δ is the change relative to the corresponding in-distribution entry for the same model, input, and suite in Table 3.
Object-Soft
Spatial-Soft
Model
Input
TSR ↑ (Δ vs. ID)
DSR ↑ (Δ vs. ID)
TSR ↑ (Δ vs. ID)
DSR ↑ (Δ vs. ID)
Diffusion Policy
VO-C
29.2 (−8.2)
26.6 (−7.0)
11.0 (−4.6)
8.8 (−4.6)
VT-C
31.2 (−8.8)
25.0 (−5.4)
25.2 (−7.8)
17.8 (−7.2)
π0.5
VO-C
35.8 (−5.8)
33.2 (−5.2)
24.4 (−1.6)
19.4 (−3.2)
VT-C
41.0 (−0.4)
34.2 (−0.8)
28.4 (+0.8)
23.2 (+1.2)
FastWAM
VO-C
54.4 (−7.6)
53.8 (−4.2)
27.8 (−9.2)
27.2 (−9.4)
VT-C
55.8 (−1.8)
55.8 (+1.4)
39.4 (−17.0)
38.8 (−17.2)
Table 7: Contents of the supplementary sections in document order.
Supplementary section
Contents
Sec. 2: Simulation and implementation details
Simulator, robot, sensing, FEM, and compute configuration.
Sec. 3: Assets and interaction safety zones
Asset inventory, rigid twins, per-object calibration, and the calibration percentile.
Sec. 4: Task suite details
Suite design, task mapping, and variation factors.
Sec. 5: Baseline training details
Training and inference settings for all evaluated policy families.
Sec. 6: Per-condition results
Out-of-distribution results per policy family and condition.
Sec. 7: Additional analyses
Qualitative cases and stiffness-conditioned deformation statistics.
Sec. 8: Tactile simulation pipeline and scope
Tactile rendering stages, recorded outputs, and validity boundary.
Table 8: Simulation and sensing configuration.
Item
Value
Simulator
Isaac Sim 4.5.0 / Isaac Lab 0.41.3, PhysX 5 GPU pipeline
Physics / control rate
60 Hz physics, decimation 3, 20 Hz control
Robot
Franka arm with Panda parallel-jaw gripper
Controller
Task-space differential inverse kinematics
End-effector action
Absolute pose target: 3D position and 3D axis-angle orientation
Gripper action
Normalized closure command; continuous and binary encodings
Finger friction
Static μs=1.5, dynamic μd=1.2, max combine mode
Camera views
Third-person 1024×1024 and wrist 512×512, resized to 224×224
Tactile sensor
GelSight Mini via TacEx; Taxim optics and FOTS markers
Tactile streams
Tactile RGB and 11×9 marker-motion field, 320×240
FEM model
PhysX soft body with corotational linear elasticity
FEM solver
Hex resolution 6, 64 position iterations, damping 2.5
Collection / evaluation
4× NVIDIA L20 GPU
Training
NVIDIA A100-80GB; per-family counts in Table 12
Table 9: Interaction safety zones of the ten deformable assets. τo is the object-specific 90th-percentile threshold used by the released evaluator, expressed as a percentage of the reference bounding-box diagonal.
Asset
gomin
gomax
Span (mm)
τo/diag. (%)
soft_pastry001
0.4000
0.8000
26.21
7.6
soft_pastry002
0.6547
0.6900
2.56
9.3
soft_pastry003
0.4400
0.8000
13.26
8.6
soft_pastry005
0.4700
0.5800
5.87
10.7
soft_pastry010
0.4200
0.6600
24.82
8.3
soft_pastry011
0.4400
0.5800
10.63
7.1
soft_stw_cube_hq
0.3200
0.6200
19.43
11.2
soft_stw_cuboid_hq
0.3800
0.4400
10.95
9.8
soft_stw_cylinder_hq
0.2900
0.6600
21.95
9.8
soft_stw_sphere_hq
0.3700
0.6300
20.77
9.7
Table 10: Task specifications for the deformable suites. Task identifiers follow the released suite configuration and are zero-indexed; instructions are reproduced verbatim from that configuration, and object descriptions are therefore not normalized across suites.
Suite
Task
Instruction
Asset
Object-Soft
0
Pick up the white swirled pastry and place it in the basket.
soft_pastry001
1
Pick up the panda-face pastry and place it in the basket.
soft_pastry002
2
Pick up the small chocolate rectangular pastry and place it in the basket.
soft_pastry003
3
Pick up the soft cream-colored cube and place it in the basket.
soft_stw_cube_hq
4
Pick up the yellow square layered pastry and place it in the basket.
soft_pastry005
5
Pick up the soft tan cylinder and place it in the basket.
soft_stw_cylinder_hq
6
Pick up the golden pastry and place it in the basket.
soft_pastry011
7
Pick up the golden knotted pastry and place it in the basket.
soft_stw_cuboid_hq
8
Pick up the soft red ball and place it in the basket.
soft_stw_sphere_hq
9
Pick up the orange pumpkin-shaped pastry and place it in the basket.
soft_pastry010
Spatial-Soft
0
Pick up the left white swirled pastry and place it on the plate.
soft_pastry001
1
Pick up the right panda-face pastry and place it on the plate.
soft_pastry002
2
Pick up the left small chocolate rectangular pastry and place it on the plate.
soft_pastry003
3
Pick up the left soft cream-colored cube and place it on the plate.
soft_stw_cube_hq
4
Pick up the left yellow square layered pastry and place it on the plate.
soft_pastry005
5
Pick up the left soft tan cylinder and place it on the plate.
soft_stw_cylinder_hq
6
Pick up the right golden pastry and place it on the plate.
soft_pastry011
7
Pick up the left soft cream-colored rectangular block and place it on the plate.
soft_stw_cuboid_hq
8
Pick up the left soft red ball and place it on the plate.
soft_stw_sphere_hq
9
Pick up the left orange pumpkin-shaped pastry and place it on the plate.
soft_pastry010
Table 11: Variation factors: training ranges and held-out out-of-distribution values. The nine out-of-distribution conditions are the three held-out levels of dome-light intensity, mass, and Young’s modulus.
Family
Factor
Training range
Held-out values
Visual
Dome-light intensity
135 (fixed nominal)
67.5, 180, 270
Spatial
Object placement
Per-task recorded support
—
Target placement
Per-task recorded support
—
Robot initialization
Per-task recorded support
—
Physical
Mass scale
×1.0 (nominal)
×1.25, ×1.75, ×2.5
Young’s modulus scale
×1.0 (nominal)
×0.5, ×0.8, ×2.0
Table 12: Training and inference configurations for the three baseline policy families.
Setting
Diffusion Policy
𝝅0.5
FastWAM
Model and compute
Policy backbone
Image-conditioned diffusion U-Net
—
—
GPUs
1× NVIDIA A100-80GB
8× NVIDIA A100-80GB
8× NVIDIA A100-80GB
Precision
—
bfloat16 mixed precision
bfloat16
Distributed training
—
FSDP
ZeRO-1
Fine-tuning method
—
LoRA
—
LoRA rank / alpha (backbone)
—
16/16
—
LoRA rank / alpha (action expert)
—
32/32
—
Training and optimization
Training duration
20 epochs
7k steps
10 epochs
Batch size (micro / global)
—
32/256
—
Gradient accumulation
1
1
1
Optimizer
AdamW
AdamW
AdamW
Learning rate
1×10−4
2.5×10−5 (peak)
1×10−4
AdamW betas
(0.95,0.999)
(0.9,0.95)
(0.9,0.95)
Weight decay
1×10−6
1×10−10
1×10−2
LR schedule
Cosine decay; 100-step warmup
Cosine decay; 1k-step warmup
Cosine decay; 5% warmup
Gradient clipping
—
1.0
1.0
EMA
Enabled
Disabled
—
Closed-loop inference
Denoising timesteps (train / inference)
DDPM: 100/100
—
Flow matching: 1000/10
Action horizon
16
50
32
Actions executed per replanning
8
10
10
Table 13: Differences between the vision-only and visuo-tactile policy variants. Trainable parameter counts include all optimized modules and exclude frozen model weights and preprocessing components.
Table 14: Per-condition OOD results for Diffusion Policy (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.
Object-Soft
Spatial-Soft
VO-C
VT-C
VO-C
VT-C
Factor
Level
TSR
DSR
TSR
DSR
TSR
DSR
TSR
DSR
Lighting
×0.5
0
0
1
1
0
0
19
14
×1.33
37
33
36
30
18
15
28
16
×2.0
0
0
0
0
16
9
17
13
Mass
×1.25
41
38
42
34
5
5
18
17
×1.75
40
35
43
38
6
6
18
17
×2.5
36
33
33
27
5
5
17
15
Young’s
×0.5
33
32
41
30
19
13
38
23
×0.8
37
31
45
33
17
16
34
20
×2.0
39
37
39
32
13
10
37
25
Table 15: Per-condition OOD results for π0.5 (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.
Object-Soft
Spatial-Soft
VO-C
VT-C
VO-C
VT-C
Factor
Level
TSR
DSR
TSR
DSR
TSR
DSR
TSR
DSR
Lighting
×0.5
37
34
39
35
30
24
38
33
×1.33
37
34
40
33
29
25
33
28
×2.0
38
35
43
34
28
22
31
26
Mass
×1.25
37
37
44
40
13
13
18
16
×1.75
31
31
42
35
18
14
17
13
×2.5
26
26
36
29
14
11
17
14
Young’s
×0.5
34
31
41
31
29
18
36
26
×0.8
45
40
42
33
29
22
31
26
×2.0
38
32
42
37
29
26
35
28
Table 16: Per-condition OOD results for FastWAM (%). Each policy–suite configuration contains 900 episodes in total: 100 per condition and 10 per task.
Object-Soft
Spatial-Soft
VO-C
VT-C
VO-C
VT-C
Factor
Level
TSR
DSR
TSR
DSR
TSR
DSR
TSR
DSR
Lighting
×0.5
63
63
54
54
30
29
39
37
×1.33
62
60
57
57
33
33
48
47
×2.0
47
45
56
56
33
33
45
45
Mass
×1.25
59
59
61
61
17
17
21
21
×1.75
46
46
55
55
16
16
24
24
×2.5
38
38
49
49
16
16
23
23
Young’s
×0.5
58
58
56
56
36
35
53
52
×0.8
58
58
58
58
29
28
53
53
×2.0
58
58
57
57
39
38
48
47
Table 17: Completion and deformation statistics under stiffness shift on Object-Soft (rates in %).
Policy
E scale
Input
TSR
DSR
Rmax median [IQR]
N
DP
×1.0
VO-C
42
39
0.63 [0.47, 0.75]
100
VT-C
42
30
0.69 [0.48, 0.97]
100
×0.5
VO-C
33
32
0.66 [0.06, 0.76]
100
VT-C
41
30
0.76 [0.57, 0.98]
100
×0.8
VO-C
37
31
0.63 [0.06, 0.79]
100
VT-C
45
33
0.75 [0.54, 0.96]
100
×2.0
VO-C
39
37
0.60 [0.37, 0.71]
100
VT-C
39
32
0.63 [0.41, 0.92]
100
π0.5
×1.0
VO-C
45
42
0.46 [0.15, 0.60]
100
VT-C
41
35
0.63 [0.51, 0.88]
100
×0.5
VO-C
34
31
0.56 [0.35, 0.83]
100
VT-C
41
31
0.71 [0.55, 0.92]
100
×0.8
VO-C
45
40
0.50 [0.31, 0.69]
100
VT-C
42
33
0.68 [0.54, 0.96]
100
×2.0
VO-C
38
32
0.46 [0.27, 0.64]
100
VT-C
42
37
0.59 [0.47, 0.79]
100
FastWAM
×1.0
VO-C
64
64
0.49 [0.38, 0.59]
100
VT-C
58
58
0.50 [0.39, 0.57]
100
×0.5
VO-C
58
58
0.55 [0.42, 0.63]
100
VT-C
56
56
0.54 [0.44, 0.60]
100
×0.8
VO-C
58
58
0.49 [0.40, 0.62]
100
VT-C
58
58
0.50 [0.41, 0.57]
100
×2.0
VO-C
58
58
0.47 [0.33, 0.58]
100
VT-C
57
57
0.48 [0.39, 0.57]
100
왜 중요한가
음식, 포장재, 의료용 소재처럼 손상되기 쉬운 물체를 다루는 로봇 작업에서는 '목적지에 도달했는가'만으로는 충분하지 않고 '손상 없이 다뤘는가'가 중요하다. 이 연구는 로봇이 작업을 마쳤는지뿐 아니라 물체를 얼마나 조심스럽게 다뤘는지를 함께 검증할 수 있는 공통 기준을 제공해, 섬세한 물체를 다루는 로봇을 신뢰하고 개발하는 데 도움을 준다.
이 논문의 용어
비주얼-촉각(Visuo-tactile) · 일반 카메라 영상과 촉각 센서 정보를 함께 사용해 로봇이 보고 동시에 만지는 감각을 갖게 하는 방식.
변형 인식 성공률(DSR) · 작업을 끝냈을 뿐 아니라 물체를 안전 범위 이상으로 누르거나 손상시키지 않았을 때만 성공으로 인정하는 더 엄격한 평가 지표.
유한요소(FEM) 상태 · 물체의 형태와 내부 움직임을 세밀하게 추적하는 물리 시뮬레이션으로, 여기서는 로봇 몰래 손상 여부를 확인하는 '숨겨진 정답' 역할을 한다.
분포 내/분포 밖(ID/OOD) 평가 · ID는 훈련 때와 비슷한 익숙한 조건에서의 테스트, OOD는 조명이나 물체 강도 등 훈련 때 겪지 않은 새로운 조건에서의 테스트를 뜻한다.
강체 쌍둥이(Rigid twin) · 부드러운 물체와 모양·크기·무게는 같지만 눌리지 않는 딱딱한 복제본으로, 물체가 변형 가능한지 여부의 영향만 따로 확인하기 위해 사용된다.
본문에 싣지 못한 그림
Figure 1: Overview of SoftVTBench. Top: 4,000 demonstrations over four diagnostic suites and more than 50 assets, including volumetric deformable objects and visually matched rigid twins. Bottom left: controlled visual and physical shifts for closed-loop generalization evaluation. Bottom center: the physical-interaction regimes captured by touch and evaluated from object deformation, from slip-prone loose grasps to excessive compression. Bottom right: synchronized visual, tactile, proprioceptive, language, and action streams.
Figure 2: SoftVTBench construction and evaluation pipeline. Stage 1 constructs matched rigid–deformable objects and calibrates object-specific interaction constraints. Stage 2 generates controlled tasks and records policy-visible observations separately from evaluator-only physical states. Stage 3 applies automatic quality control and human verification, producing the released training, ID, and OOD splits. Task success and deformation compliance are labeled separately; DSR combines them during benchmark evaluation.
Figure 3: Tactile observations across deformable assets. Each panel shows a tactile RGB image with its marker-motion overlay at the moment of grasping, for the ten Object-Soft tasks. Objects that differ in geometry, compliance, and contact area produce distinct contact patches and shear fields under the same pick-and-place instruction. These are the cues available to a visuo-tactile policy and absent from external RGB views.
Figure 4: Task Success Rate under out-of-distribution shifts, resolved by factor. Rows correspond to the Object-Soft and Spatial-Soft suites; column groups correspond to Diffusion Policy, π0.5, and FastWAM, each resolved into illumination, object-mass, and Young’s-modulus shifts.
Figure 5: Illustrative rollouts on three deformable objects across Object-Soft and Spatial-Soft. Each row shows synchronized third-person, wrist, and tactile observations at approach, peak interaction, and placement, followed by the normalized deformation trace Rt. Circle, diamond, and square indicate the displayed steps. All rollouts achieve task success (TSR=1). Blue trajectories satisfy the calibrated deformation tolerance (DSR=1), while orange trajectories exceed Rt=1 despite success (DSR=0). Rows are independently sampled and not matched-initial-state comparisons.
