SIMFOUNDRY: Modular and Automated Scene Generation
for Policy Learning and Evaluation

Anonymous Authors
Under double-blind review
CoRL 2026
PDF arXiv Video Code BibTeX
SimFoundry: real-to-sim-to-real cycle with object, scene, and task cousins

TL;DR SimFoundry takes a single video of a real world scene and automatically turns it into an interactive simulation environment for scalable policy training and evaluation.

Digital Twin and Cousin Generation

Selected results across diverse scenes and tasks.

Input video (2x)
Reconstructed twins
Reconstructed cousins

SimFoundry Reconstructed Scenes

Explore the reconstructed 3D scenes interactively. Drag to rotate, scroll to zoom.

Hybrid scene: 3D Gaussian Splat background + textured object meshes
Loading viewer…
Drag and scroll to rotate and zoom.

Note: 3D viewers are not supported on mobile. Please visit on desktop for the interactive experience.

SimFoundry Pipeline

SimFoundry pipeline: Physical Scene Extraction, Sim Scene Generation, and Cousin Augmentation

Real to Sim Eval

Side-by-side comparison of policy execution in real world versus SimFoundry.

Policy: DreamZero

Real World
Success Rate: 16%
SimFoundry
Success Rate: 28%

Sim vs. Real Success Rates

SimFoundry predictions closely track real-world performance

(hover over the points to see additional metrics)

SimFoundry
Polaris
Tasks

Evaluation Metrics Comparison

Higher Pearson r and lower MMRV indicate better sim-to-real correlation

Average Pearson r ↑
SimFoundry
Polaris
Average MMRV ↓
SimFoundry
Polaris

Sim-to-Real Policy Training

Policies Trained on SimFoundry Data Transfer Zero-Shot to Real-World

Simulation Eval
Success rate: 100%
Real World Eval
Success rate: 96%
Real World Eval (Unseen Objects)
Success rate: 100%

SimFoundry Digital Cousins Enable Policy Generalization to Novel Objects

Digital twin and cousins of the wooden drawer organizer

SimFoundry generates meaningful Task Cousins that facilitate policy generalization

SimFoundry data boosts real-world VLA performance

Across 13 tasks
28% 46%
w/o SimFoundry
w/ SimFoundry
On 7 held-out tasks (unseen even in sim)
0% 29%
w/o SimFoundry
w/ SimFoundry
Sim Eval
Real Eval

Co-Training SimFoundry Sim Data and Small Amounts of Real Data Produces Even Better Policies

Sim-only Real-only Co-train
Success rate (%)
100806040200
88
100
100
24
60
92
20
56
92
Stack DishwareStore MarkerPut Away Trash

Reconstructed Objects

Click an object in the real scene to view its reconstructed, physics-ready mesh.

Real Scene — hover and click an object
Real scene
Loading scene…
Reconstructed Mesh
Click an object to load its mesh
Drag and scroll to rotate and zoom.

Reconstructed Scene v.s. SAM3D output

SimFoundry recovers more accurate object meshes and poses than SAM3D, especially on occluded, cluttered scenes. The pipeline also generalizes across input modalities, supporting open-source datasets and synthetically generated images out of the box.

Input Image
Input image
Image not configured
SimFoundry Output
Loading viewer…
Drag and scroll to rotate and zoom.
SAM3D Output
Loading viewer…
Drag and scroll to rotate and zoom.
Easy Scene
0.0081
0.0042
0.0026
0.0160
0.0060
0.0041
Chamfer Distance (m) ↓Position Error (m) ↓
Medium Scene
0.0087
0.0047
0.0033
0.0180
0.0076
0.0057
Chamfer Distance (m) ↓Position Error (m) ↓
Hard Scene
0.0088
0.0091
0.0039
0.0220
0.0180
0.0073
Chamfer Distance (m) ↓Position Error (m) ↓
SAM3D Zero-Shot SimFoundry Zero-Shot SimFoundry Tuned

Note: 3D viewers are not supported on mobile. Please visit on desktop for the interactive experience.

BibTeX

@inproceedings{yourkey2026,
  title     = {Project Title Goes Here},
  author    = {Last, First and Last, First and Last, First},
  booktitle = {Proceedings of the Venue},
  year      = {2026}
}

Anonymous website for double-blind review.