News has surfaced of Sony patenting a method of AI-generated facial performance capture to ease the workload on animators. It’s no secret that Sony Interactive Entertainment studios are industry leaders when it comes to capturing true-to-life performances in the video game medium. With a huge increase in fidelity across the past generation of consoles, in the space of a decade developers like Naughty Dog and Sony Santa Monica have gone from hand drawing facial and movement animation with painstaking effort to building games and characters entirely through performance capture. It’s no wonder, then, that Sony has filed a patent for more game-changing performance capture technology.
Filed on April 1, 2022, by Sony Interactive Entertainment LLC, the patented system takes mesh data from a human actor captured by a 3D camera to record the depth of the face. This polygonal mesh is used in game development to build accurate human models and is typically handcrafted by animators using the vertices dots on a mocap actor’s face as reference. This method is currently the standard across most triple-A videogame development, including the Insomniac's upcoming Spider-Man 2. This system would automatically generate mesh data, however, using a mix of 3D and 2D cameras to train an Artificial Intelligence program that can accurately capture facial expressions and build mesh data with little need for human intervention.
This patent follows an industry trend in its development of AI to produce work that previously took a great deal of human handiwork. The video game industry has seen a huge rise in the use of AI to do many things, from upscaling the resolution of textures to interpolating frames for smoother games. NVIDIA specifically have patented RTX Remix and their Deep Learning Super Sampling systems that make remastering classic games a much easier feat for modders and developers.
In fact, this isn’t the first time gamers have seen AI used in the triple-A space. Recently, Rockstar Games came under fire for using AI upscaling in their remaster for the GTA Trilogy, leading to some unfortunate spelling mistakes on some assets.
As this patent directly states, the presence of AI in this instance is to help reduce workload and keep the sky-high costs of facial animation down. With hundreds of animators crafting thousands of blend shape and joint files to mirror mocap performances, the process is incredibly time-consuming and expensive. Thusly, unlike with some artistic AI endeavors that seek to replace actors, the aim is not to remove the need for the animator or actor entirely, but to assist in the capture process. This system, if successfully deployed, could go a long way towards reducing crunch time for animators on particularly demanding games, as well as opening the gates for greater freedom of expression.