Quantumrun

IMAGE CREDIT:

iStock

Virtual surroundings: Apps can now enable anyone to create 3D versions of images

Machine learning (ML) techniques have enabled technologies to transform images into realistic 3D renders, paving the way for better virtual world-building.

Author:
Author name
Quantumrun Foresight
October 17, 2022

Insight summary

Virtual reality's creation is complex and costly, but deep neural networks simplify this by turning 2D images into 3D models using inverse graphics. The generative adversarial network (GAN) plays a key role in this transformation, pitting two neural networks against each other to produce highly realistic outputs. These advancements have wide-ranging implications, from reducing production costs in media to enabling new features in social media and apps.

Virtual surroundings context

Immersive virtual reality and environments take time, money, and talent to produce. However, deep neural networks (DNNs) can make the 3D-making process easier and more accessible to the public. Virtual surroundings are created through a process called inverse graphics. This technique comprises transforming images into three-dimensional (3D) objects. This process is complex because it requires considering the geometry, material properties, and light sources—all while ensuring that the final product looks like a convincing 3D scenario.

Research on inverse graphics commonly assumes that 3D labels are available during machine learning (ML) training, where a neural network is taught to predict these labels. Synthetic datasets like ShapeNet are frequently used for synthetic data training. However, models created from synthetic data tend to perform poorly in analyzing real photographs.

In 2014, computer scientist Ian Goodfellow developed the generative adversarial network (GAN), an ML model that positions two neural networks against each other (generative versus discriminative networks). The generative network copies the image data as closely as possible and sends it to the discriminative network. The goal is to deceive the discriminative network into accepting the generated version as the real one. The discriminative network evaluates the output and sends it back to the generative network if it doesn’t meet the discriminative network’s standards. And so, the loop continues until the best possible outcome is produced.

Disruptive impact

In 2021, virtual world developer Nvidia announced a deep learning application to turn any standard 2D image into a highly photorealistic 3D model. This feature allows users to view and control the model in a virtual environment as if it were reality. The GANverse3D program, created by Nvidia’s AI Research Lab in Toronto, demonstrated a single photograph of a car being turned into a 3D replica with realistic headlights and taillights that may navigate the virtual world.

The application used GAN to enable the app to create images of an object from multiple viewpoints using only a single photograph as reference material. Nvidia said that these synthesized images are as accurate as photographs taken by somebody who walked around a parked car, taking pictures from different perspectives. An individual can then employ the synthetic images with a rendering framework for inverse graphics (creating 3D mesh models). The company hopes the technology can help architects, game developers, and designers create 3D mock-ups more quickly and accurately. Additionally, users can use the app for city planning, including building digital twins and training ML models.

Meanwhile, in 2022, Stanford University researchers developed an artificial intelligence (AI) model called EG3D, which can create an underlying geometric structure under images of faces and objects. To enable these 3D renders to be editable, University of Wisconsin-Madison researchers created an ML model called GiraffeHD, which extracts controllable features such as shape, color, or camera angle.

Implications of virtual surroundings

Wider implications of virtual surroundings may include:

Governments developing legislation around virtual environments and the need for disclosure of use.

Automated virtual environments being employed in fantasy and superhero genre media, lowering the expenses of TV production and game development.

Content producers creating photorealistic backgrounds for their videos without hiring professional visual effect developers.

Social media platforms, like Instagram and TikTok, offering features or filters with virtual environment generation capabilities in their short videos/reels.

Smartphone and tablet apps enabling users to transform their photographs into 3D renders. Likely applications include real estate agents sharing 3D renderings of homes to online buyers.