Editor’s Note: Welcome to LAVNCH [CODE]’s “What is…” series. Each month, we’ll take a deep dive into an important emerging tech topic, beginning with virtual production.

Virtual production is a term that’s become increasingly popular these days. But what does it actually mean? In the simplest terms, virtual production allows you to mix animation with live footage—but there’s so much more to it.

“The term ‘virtual production’ covers a real range of applications from previsualization and fully CG animated projects to the LED-wall in-camera visual effects made famous by The Mandalorian,” Alex Stolz, Future of Film, said in the 2021 Nostradamus Report. “What’s exciting is that it is already possible for filmmakers to use this technology, just not necessarily yet in a huge professional studio volume.”

Sarah Jones, senior technical solutions specialist at Blackmagic Design defines virtual production as “a revolutionary tool that combines physical and digital environments, blurring the lines between production and post, and ultimately providing content creators with more creative opportunities.”

Virtual production allows for immense creative freedom, added Akintayo Adewole, creative technologist at DRKR PXLS. “With very little hardware, free software, a desire to learn, and a lot of creativity, you’re in the game,” he said. “You can start and finish full productions virtually in-engine or extend it to the real world. There are endless possibilities for high-end storytelling.”

Watch the video below to see what Adewole created using virtual production skills he acquired over the last six months.

Advantages and Disadvantages of Virtual Production

The most obvious advantage for virtual production is that content can be shot in a studio while making it seem like it’s anywhere else in the world. “It’s much easier to bring a location virtually to a production than to bring an entire production team to a location,” said Jones. “All the various crew members and talent no longer have to be in the same place at the same time. This rings even more true when faced with some of the challenges from the past two years of the pandemic, such as navigating travel bans and maintaining safe numbers on set.”

Virtual production can also end up saving the day in terms of budget. “With virtual production, you can also make modifications to scenes in real-time during production and avoid expensive reshoots or lots of work in post-production,” she added. “You have more control with virtual production than you do without it.”

Adewole also pointed to budgetary savings as an advantage: “The initial cost of entry for in-engine production is reasonable, even for freelancers or indie studios. And while building a fully outfitted extended reality (XR) virtual production studio with all of the bells and whistles may be in the six-figure range, the ROI can be greater when you consider costs and timelines for comparable and traditional production workflows.”

Glassbox Technologies’ co-founder and board member Mariana Acuña Acosta agrees: “Realtime technologies save you time, energy, and money. There are so many ways to create a very lean pipeline. I created a little short film during the pandemic without leaving my home office at all! I had motion capture, free sets, free characters from Mixamo. Anybody can do that today, with everything that’s readily available.”

“It can be an investment to get that framework in place, but that investment will pay off quickly, and there are many products on the market today making virtual production more affordable for budgets of any size,” added Jones.

While those are some major advantages, it’s not all rainbows and roses. The planning process for virtual production can be labor-intensive. “An extremely detailed and thorough pre-production process is crucial to test the assets and technologies that will be used,” said Jones.

And the staff needs to be highly-skilled and adequately trained to avoid disaster on set. “Despite the technological advantages, many creatives, directors, and cinematographers are still ramping up and could be potentially resistant,” claimed Adewole. “Before investing in a permanent or pop-up space, all players at each level of the process need an understanding and adoption of the pipeline that makes their virtual production successful.”

Virtual Production Technology

The core component of virtual production is a real-time rendering software, and two of the most common are Unreal Engine and Unity.

Watch the video below to see a comparison of Unreal Engine and Unity.

Unreal Engine provides a solid amount of interoperability for industry standard digital content creation (DCC) and VFX applications such as Maya, Cinema4D, Blender, and Houdini, according to DRKR PXLS’ Adewole.

“On the hardware side, it’s graphically intensive, where GPU power is a primary consideration,” he added. “NVIDIA RTX GPUs are critical to most virtual production setups. They handle all of the real-time physics we see when it comes to lighting, shadows, and post-processing effects in virtual environments.”

LED display technology is also prominent in a virtual studio.

“LED panels make up the backbone of a virtual studio,” claimed Absen’s marketing coordinator Jason Polk. “An LED Wall, floor, and ceiling—along with having high resolution cameras that can take that information that is on the wall and have the subject interact with it—are paramount.”

“Pixel pitch and brightness are key to the studio, as well as limiting the moiré effect,” he added. “HDR 10 capabilities are essential for users of XR stages and allow for greater creativity.”

In addition, you’ll need performance capture and device tracking technologies to bridge the virtual and real worlds.

“Inerti-based mocap systems include the likes of Xsens, Rokoko, and Perception Neuron, while optical-based systems include Vicon and Optitrack,” said Adewole. “IR-based tracking systems, such as HTC Vive provides affordable and accurate tracking at the entry level.”

“For face tracking, you can get excellent results with an iPhone that has a depth camera and leveraging Apple’s AR Kit,” he added. “It also has direct integration to Unreal Engine using their LiveLink plugin. For more refined capture data, companies like Faceware provide bundled hard and soft solutions at a higher price point.”

Quality is of the utmost importance when it comes to virtual production—especially when technologists are making computer-generated images look photorealistic. Thus, high/extreme resolution cameras are essential for virtual production, according to Jones. “Blackmagic Design’s URSA Mini Pro 12K digital film camera allows users to shoot at 60 fps in 12K or use in-sensor scaling to allow 8K or 4K RAW at up to 120 fps without cropping or changing the field of view,” she said. “It has a Super 35mm 80 megapixel sensor, 14 stops of dynamic range, 12G-SDI connections, and Blackmagic RAW image processing, which provide unprecedented resolution and quality for color, keying, compositing, reframing, stabilization, and tracking in 4K or 8K.”

Building a Virtual Production Studio

What exactly does building a virtual production studio entail? Well, like any technology installation, the answer can vary wildly depending on the client’s wants and needs.

For a single person or a small team, Adewole said “a high-end workstation or laptop, an inertia based mocap suit, and an iPhone for face capture can do wonders.” He added that technologists could create a finalized production out of their living room, similar to Cory Strassburger’s Xanadu project.

Watch the video below to learn more about Xanadu.

Obviously larger studios are much more complicated. “Virtual production studios may include a soundstage with either a full volume domed or curved LED wall, or a single wall/flat LED stage,” said Blackmagic Design’s Jones. “Lighting, tracking, and other stage equipment are big factors, as is the level of GPU power you’ll need, which in any scenario will be a lot. In addition to the physical specifications, you’ll need a good understand of things like color science, pixel pitch, LED volumes, high-resolution cameras, and much more.”

Studios may also utilize the classic curved and/or flat green screen walls. “Lighting, tracking, and other stage equipment remain big factors with green screen virtual production,” Jones added.

The Future of Virtual Production

The future of production is virtual production, according to the experts.

“In 2020, the amount of film and TV projects that were using virtual production doubled; in the
midst of this crisis, our company had our best month ever,” claimed Acuña Acosta. “Everybody is implementing virtual production techniques and remote working infrastructure; what we forecasted was going to happen in four years happened in one.”

“In five years realtime technologies will have been adopted across the board in all studios, even those that have been very hesitant to move into that world,” she concluded.

To learn more about this topic, check out our Virtual Production Hub.