DynaSlots

Abstract

Robot manipulation involves localized changes: picking up a cup moves the cup while leaving the surrounding scene largely unchanged. Yet existing 3D visual pretraining methods, including recent dynamics-aware variants, model state transitions at the scene level, compressing the motion of everything in a scene into a single global latent action. This conflates semantically distinct transitions, obscures dependencies between changing regions, and leaves the representation vulnerable to task-irrelevant content.

We introduce DynaSlots, an action-free, label-free pretraining framework that decomposes each point-cloud frame into a small set of temporally consistent slots and models dynamics at this per-slot level rather than the scene level. A temporal slot propagation module promotes consistent slot assignments across frames; an inverse dynamics model extracts one latent action per slot from feature differences; and a diffusion-based forward dynamics model, paired with a self-supervised interaction head, predicts each slot's future state while learning pairwise proximity-and-motion relations.

Experiments on Meta-World, Adroit, and a real RealMan RM65-B robot show that DynaSlots significantly outperforms prior scene-level baselines: it attains higher success rates in simulation and exhibits strong robustness to scene clutter and unseen objects in real-world manipulation.

Key Ideas

The latent action of a transition should be defined per slot, not per scene.

Method

Point cloud → tokens → temporally consistent slots → per-slot latent actions → cross-slot diffusion dynamics.

Simulation Results

Meta-World & Adroit, following the evaluation protocol and baseline suite of prior scene-level dynamics-aware work. Higher is better; parentheses give each method's gap to the best.

Mean success rate (%). Meta-World groups: Easy(3), Med.(6), Hard(2), V.Hard(3) tasks. DynaSlots leads every group and both benchmarks.

Real-World Experiments

A RealMan RM65-B arm with a parallel-jaw gripper and a fixed eye-to-hand Intel RealSense RGB-D camera. 100 expert trajectories per task; five initial layouts × ten trials each.

Robustness: clutter & unseen objects

Both on task (c), five layouts × ten trials. “Drop” is the absolute change vs. the matched original scene.

Method	No clutter	Cluttered	Drop
DP3	54.0	28.0	26.0
PointDif	52.0	26.0	26.0
FVP	54.0	30.0	24.0
DynaMo-3D	60.0	36.0	24.0
AFRO	68.0	40.0	28.0
DynaSlots (ours)	78.0	70.0	8.0

DynaSlots retains 89.7% of its no-clutter score under clutter; the strongest scene-level baseline retains only 58.8%.

Method	Original	Unseen	Drop
DP3	54.0	38.0	16.0
PointDif	52.0	36.0	16.0
FVP	54.0	40.0	14.0
DynaMo-3D	60.0	48.0	12.0
AFRO	68.0	54.0	14.0
DynaSlots (ours)	78.0	72.0	6.0

With scenes containing unseen objects, DynaSlots loses only 6.0 points and beats the strongest baseline by 18.0.

Quick Start

Linux · Python 3.8 · MuJoCo 2.1 · CUDA. Full setup in INSTALL.md.

# 1. environment
conda create -n dynaslots python=3.8 -y
conda activate dynaslots
pip install -e .

# 2. demonstrations
bash scripts/gen_demonstration_adroit.sh pen
bash scripts/gen_demonstration_metaworld.sh bin-picking

# 3. pretrain representation
bash scripts/pretrain_dynaslots.sh adroit_pen run1 0 0

# 4. train & evaluate the diffusion policy
bash scripts/train_dynaslots_policy.sh adroit_pen run1 0 0
bash scripts/eval_dynaslots_policy.sh  adroit_pen run1 0 0

Arguments are task_name, run_name, seed, and comma-separated GPU IDs. Set RESUME=true to resume representation pretraining.

Repository Layout