Over the past two years, there has been a lot of discussion about the potential of the metaverse. We have been promised that a shared, interactive digital reality will make humans more connected, productive, empathetic, and informed.

Indeed, current use cases of augmented reality (AR) and virtual reality (VR) in education, ecommerce, product design, and more demonstrate that the metaverse’s promise is very real. A smoothly functioning open metaverse could vastly improve how the average user works, plays, interacts, and learns. However, at present, everyday people can only be tourists in the metaverse. They can visit, but they cannot truly build anything there.

People want to contribute to their online communities by adding their own unique content. Instagram, YouTube, and TikTok could never have taken off without the ease and convenience of smartphone cameras. However, even highly trained 3D artists struggle to create 3D assets that look decent when transmitted over the web. Metaverse assets also need to possess interactive properties and real-time animations, adapt their appearance based on scene lighting or environment changes, and work across a variety of hardware. How will metaverse platforms solve this problem and enable democratized content creation? The answer lies in AI.

At Avataar, we are experimenting with technological approaches to make 3D content creation easy and accessible for anyone. Using AI, we believe it is possible to create a 3D camera that will significantly contribute to democratizing 3D content creation. Through Neural Radiance Fields (NeRF)-based methods, we have discovered that we can lay the foundation for high-quality 3D assets using only 2D input images.

To clarify, we are not referring to point-and-shot capture capability or Midjourney-like asset generation from text descriptions here. Our method involves capturing sets of images from three circular trajectories and inputting them into the NeRF algorithm to generate 5D coordinates (a 3D X, Y, and Z point plus a 2D viewing direction) for every pixel. The resulting assets are impressive, but not perfect. We still need to account for and correct factors such as room lighting, object surfaces, opacity, relative size, and more. In short, using NeRF methodology, we can come close to teleporting an object from the real world to the metaverse, but fine-tuning is required after the initial capture to achieve true accuracy and photorealism.

Through this experimentation, we are learning how to enhance and expedite automated 3D content creation rapidly. Avataar has already developed methods to convert these NeRF-generated assets into accurate 3D geometric meshes of objects with corresponding texture files. This opens up a world of possibilities. You can create a 3D asset of a real object and then apply different textures or materials, resulting in original variants of a product or object. You can manipulate the triangle mesh to create new shapes, iterating designs. You can combine objects to create your own custom spaces within the metaverse.

NeRF-based 3D asset creation is already helping retailers scale 3D asset creation for e-commerce. At the same time, organizations like The Khronos Group and the Metaverse Standards Forum are actively working to make sure it’s possible to move those assets freely across platforms and hardware. Soon, average consumers may be able to use the same tools they use to make TikTok videos to create and share 3D objects. This will mark a true tipping point for the metaverse: a time when billions of people can not only experience the metaverse but also actively build it.

Editor’s Note: This blog is part of a series for Artificial Intelligence (AI) Appreciation Day, which is held annually on July 16. Click here to read more AI Day stories from LAVNCH [CODE] and click here to read more AI Day stories from rAVe [PUBS].

About the Author: Daniel Frith started in 3D in 1993 and has worked across a large range of 3D industries from flight simulation, computer games, film, television, advertising, and retail. Frith has led 3D and technology teams in both small and large companies from Square Enix to IKEA. Frith joined Avataar in 2022 as vice president of 3D to set long term strategies, evolve technology, innovation, R&D, and work closely with enterprise partners. Frith is vice chair of glTF for the Khronos group with Avataar and an advisory board member of The Realtime Society.