MIT CSAIL and Toyota Research Institute researchers developed SceneSmith, a system that uses three AI agents powered by GPT-5.2 to automatically generate realistic 3D indoor environments for robot training. A 'designer' agent creates scene layouts, a 'critic' evaluates realism, and an 'orchestrator' manages the process. The system produces scenes with up to six times more objects than prior methods, including articulated items like cabinets. Over 200 users found SceneSmith's visuals more realistic than competing systems over 90% of the time, and a pretrained robot policy successfully executed tasks in generated scenes without prior exposure to them. The main trade-off is speed — generating a single scene can take multiple hours. The work was presented as a spotlight at the 2026 International Conference on Machine Learning.
Questions this post answers
How does the MIT SceneSmith system generate 3D scenes for robot training?
SceneSmith uses three VLM agents built on GPT-5.2: a 'designer' that generates scene layouts and objects, a 'critic' that evaluates realism and flags issues (e.g., removing a bathtub from a living room), and an 'orchestrator' that manages their back-and-forth and decides when the design is complete. The finished scene loads directly into physics simulation software with properties like mass, friction, and inertia. Robotics engineers tracking simulation tooling for training pipelines follow developments like SceneSmith on daily.dev.
How does SceneSmith compare to other 3D scene generation baselines like Holodeck?
SceneSmith outperforms baselines including HSM and Holodeck by generating environments with up to six times more objects per scene, including articulated items like openable cabinets that prior baselines rarely included. Over 200 users rated SceneSmith's visuals as more realistic more than 90% of the time and found it followed prompts more closely than competing approaches. Teams deciding between scene-generation tools for robot sim environments track comparisons like this on daily.dev.