Latent diffusion models for generating 3D VR content

Paper Title:

LDM3D-VR: Latent Diffusion Model for 3D VR

Published on:

6 November 2023

Primary Category:

Computer Vision and Pattern Recognition

Paper Authors:

Gabriela Ben Melech Stan,

Diana Wofk,

Estelle Aflalo,

Shao-Yen Tseng,

Zhipeng Cai,

Michael Paulitsch,

Vasudev Lal

Bullets

Key Details

•

Introduces latent diffusion models for generating panoramic RGBD images

•

Can generate panoramas and depth maps from text prompts

•

Includes model for jointly upscaling RGB images and depth maps

•

Models fine-tuned from existing latent diffusion models

•

Performs well compared to other panorama and super-resolution methods

Explore the topics in this paper

3d virtual reality

depth maps

latent diffusion models

panoramic images

text-to-3d generation

AI generated summary

Latent diffusion models for generating 3D VR content

This paper introduces two new latent diffusion models to generate 3D virtual reality content. The models can generate panoramic RGB images with corresponding depth maps from text prompts, and can upscale low resolution RGB images and depth maps to high resolution. The models build on top of existing latent diffusion models and are fine-tuned on datasets with panoramic images, depth maps, and captions. Evaluations show the models compete well against other panorama generation methods and super-resolution techniques. The work enables new ways to create immersive 3D VR environments.