Listen to the Podcast:

In real time, new green-screen technology driven by AI developed by Netflix researchers can create realistic visual effects for movies and television shows.

In order to insert actors into the foreground of virtual or prerecorded sequences, green-screen technology is frequently utilized to record actors. To do this, actors are photographed against a vivid green backdrop that can be digitally isolated and eliminated.

This procedure can be carried out mechanically and with a respectable degree of accuracy, as in television weather forecasts, although it can be thrown off by articles of green clothing as well as by translucent or fine objects, such as wisps of hair. When higher accuracy is required in movies or TV shows, specialized operators manually adjust settings, sometimes spending hours perfecting a shot.

Netflix has developed a process it calls Magenta Green Screen (MGS) in an effort to develop a system that is quick and precise. Actors are lighted from the front with red and blue LEDs that collectively create a magenta glow while being filmed against a background of bright green LEDs.

This technique results in a green channel that records only the background, leaving the foreground to appear black, and red and blue channels that record only the foreground, leaving the background to appear black. This is because digital cameras work by taking an individual red, green, and blue value for each pixel. These taken as a whole give the magenta and green appearance.

Film editors are able to realistically and immediately replace the green channel, bringing the players in the foreground of another scene. Even potentially challenging situations, like the space around hair strands or clear bottles, may be handled without incident.

One of the Netflix researchers engaged, Paul Debevec, claims that “computers already have such powerful tools to make a lot of things easier.” “This is another thing we can simplify so that the gifted artists we have can concentrate on the artistry and actually improve the appearance of things.”

However, there is an issue with the approach. The actors appear magenta-tinted because only blue and red are recorded in the foreground. Netflix employs artificial intelligence to address this issue by bringing the complete spectrum of color back into focus. It does this by utilizing a photo of actors who were lit normally as a reference to produce a realistic-looking green channel. Although fast methods like averaging the red and blue channels to approximate a green channel work effectively enough for the director to monitor during recording, this AI does not yet operate in real time.

The video production industry is usually open to innovative approaches, but according to Drew Lahat of Geiger Post in California, the methodology entails several complicated procedures that might not yet be compatible with the quick speed of filming and broadcasting.

From a practical standpoint, there are many features of this technology that, he says, “make me wonder if it’s worth it.” According to what I understand, this strategy quickly fails if a good green screen is not used in a completely controlled environment.