🧩 SCENE2DEMO Framework
A self-evolving framework that turns a real-world image and user query into a simulated scene, executable tasks, and training data.
Evaluation rollout results
SCENE2DEMO Pipeline
SCENE2DEMO turns an image and a query into executable embodied task data with structured decomposition and feedback-based refinement.
A self-evolving framework that turns a real-world image and user query into a simulated scene, executable tasks, and training data.
An object-action graph that links object states and actions, making long-horizon task decomposition more stable and executable.
A visual-feedback loop that inspects failures and revises action flows for more reliable data generation and imitation learning.
| Task Success Rate (SR %) / Subtask-Level Score (%) | ||||||
|---|---|---|---|---|---|---|
| Method | \(\mathcal{T}_1\) | \(\mathcal{T}_2\) | \(\mathcal{T}_3\) | \(\mathcal{T}_4\) | \(\mathcal{T}_5\) | \(\mathcal{T}_6\) |
| M1: Ours (primitive) | 0/50 | 0/50 | 40/70 | 0/50 | 66.7/66.7 | 74.2/74.2 |
| M2: Ours (prim+Evo) | 40/84 | 30/79 | 70/93 | 70/85 | — | — |
| M3: GenSim2 (prim) | 0/73 | 5/78 | 55/91 | 0/68 | 100/100 | 38.8/82.1 |
| M4: RoboGen (prim) | 0/50 | 0/46.5 | 0/50 | 0/50 | 0/50.2 | 0/42.8 |
| M5: RoboGen (prim+RL) | fail | fail | fail | fail | success | success |
* \(\mathcal{T}_1\): "Put the glass into the refrigerator", \(\mathcal{T}_2\): "Put the apple into the refrigerator", \(\mathcal{T}_3\): "Put the apple into the bowl", \(\mathcal{T}_4\): "Put the cup on the table", \(\mathcal{T}_5\): "Open/Close", \(\mathcal{T}_6\): "Pick/Place".
* RoboGen+RL (M5) requires excessive training time (48h+ total).
1. Open Refrigerator
2. Pick Up Glass
3. Put Glass In
4. Close Refrigerator
Iteration 2 (Final Success)
1. Pick Up Apple
2. Put Apple Into
Iteration 2 (Final Success)
Repeated runs on the same instruction produce the same executable subtask structure, while unconstrained baselines vary more across trials.
Simple tasks finish in roughly minutes, and long-horizon demonstrations usually complete in 15 to 30 minutes including self-evolution iterations.
Full visual feedback performs best overall, while narrower camera inputs or longer history windows reduce reliability and increase inference cost.
Model Training Inputs and Objectives
We train behavior cloning policies using 100 expert demonstrations autonomously generated by SCENE2DEMO, and improve data diversity with randomization on target-object scale and rotation.
The policy uses a PointNet encoder for point clouds, fuses the point-cloud features with proprioceptive states, and forms a compact representation for downstream action prediction.
A 4-layer Causal Transformer followed by an action MLP predicts 10-DoF actions (7-DoF arm and gripper control + 3-DoF mobile-base motion).
Evaluation rollout results
| Task | Open Refrigerator | Pick Up Glass |
|---|---|---|
| SR (Succ. / All) | 96.0% (24/25) | 92.0% (23/25) |
* Training Setup: 1×A800 GPU; Batch Size=1024; Epochs=200; Samples/Epoch=102400; Optimizer=AdamW; Initial LR=4e‑4 with linear decay at epoch 500.
* Scale_noise of ±0.10 is applied to the XYZ dimensions of objects during each policy evaluation to ensure robustness.
If you find this work useful, please consider citing it with the following reference.
@inproceedings{anonymous2026scene2demo,
title = {SCENE2DEMO: Self-Evolving Embodied Data Generation via Object-Action Graph},
author = {Anonymous Authors},
year = {2026},
booktitle = {Anonymous Submission},
}