Dreamcrafter

Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs

Cyrus Vachha1, Yixiao Kang1, Zachary Dive1, Ashwat Chidambaram1, Anik Gupta1, Eunice Jun2, Björn Hartmann1
1University of California, Berkeley 2University of California, Los Angeles
Paper arXiv (Coming Soon) Video Poster

Dreamcrafter enables editing of Radiance Field scenes with Generative AI models in an immersive spatial interface.

Abstract

We present an immersive VR editor for generating and editing 3D gaussian splatting scenes and a workflow for composing scenes for generative images and videos.

Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content; or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.

Video

Edit

Objects can be edited with voice and natural languge instructions. A 2D image preview and spatial annotation is shown after choosing a varient.

Prompt

New objects can be generated via prompting. Users can select from 3 variant generations and a 2D image preview is shown.

Sculpt

For finer control, new objects can be generated via sculpting. Users arrange basic primitives and use ControlNet to stylize into a realistic object.

After editing, full fidelity objects are created and edited in an offline process, replacing the proxy representations in the scene.


✨ Magic Camera: Stage Scenes for Image/Video Generation

The Magic Camera allows users to position a virtual camera within a scene and apply stylization based on prompts using the ControlNet module, similar to rendering a frame in 3D editors. This stylized output can be used as input for image-to-video or image-to-3D models, enabling an iterative design process for creating and editing 3D scenes in Dreamcrafter. The system serves as a tool for spatial prompting, allowing users to create, stage, and stylize scenes using low fidelity objects in a VR interface, and explore high fidelity 3D scene generation through video diffusion models. Dreamcrafter's capabilities could further enhance generative AI design systems for 2D, video, and 3D outputs for world building.

Interpolate start reference image.

Magic Camera Input and Output
"realistic apartment living room"

Image-to-Video Generation


BibTeX

If you find our work helpful, please consider citing our poster.

@inproceedings{vachha2024dreamcrafter,
      author = {Vachha, Cyrus and Kang, Yixiao and Dive, Zach and Chidambaram, Ashwat and Gupta, Anik and Jun, Eunice and Hartmann, Bjoern},
      title = {Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs},
      year = {2024},
      publisher = {Association for Computing Machinery},
      url = {https://doi.org/10.1145/3672539.3686328},
      doi = {10.1145/3672539.3686328},
      booktitle = {Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology},
      series = {UIST Adjunct '24}
      }