Sony is exploring a controller input recognition technology that could significantly enhance the experience of watching gameplay by providing viewers with accurate controller input captions inferred from game footage. The entertainment giant secured a patent for such a solution just last week, according to some newly surfaced intellectual property documentation. Like most of Sony's contemporary patents, the newly published concept delegates the computational heavy lifting to specialized machine learning models.

As one of the largest companies on the planet, Sony's research and development division is never idle. In the last month alone, the Tokyo-based conglomerate secured over half a dozen patents pertaining to gaming and general entertainment. Those recently approved concepts include an interactive avatar system, a dynamic refresh rate synchronization solution capable of complementing software like NVIDIA's G-Sync and AMD's FreeSync in order to minimize frame rate dips, and a gaze tracking technology for non-VR games.

RELATED: Sony May Be Working on an Interactive Avatar System

Meanwhile, the company's latest patent describes an artificial intelligence system that would take game footage and enhance it with auto-generated input captions, even without access to the actual controller metadata from the recorded sessions. As such, the technology would hinge on robustly trained machine learning models which would have to be taught gameplay mechanics on a per-game basis. Since that would ostensibly require a substantial commitment for each new title, it's likely that Sony would initially limit this solution to its own exclusives whose budgets can support investments like teams of AI scientists and trailers starring LeBron James and John Travolta.

CONTROLLER ACTION RECOGNITION FROM VIDEO FRAMES USING MACHINE LEARNING

Of course, all of that is assuming that Sony attempts to commercialize this technology in the first place. And that is far from a given, especially for a company of its size that often files for patents defensively, simply so that it can ensure certain avenues of approach remain open for royalty-free implementation in the future, even when it doesn't plan to actively pursue them. That said, this development comes only weeks after Sony was awarded another patent meant to improve game capture, with its previous filing describing a continuous gameplay recording technology based on save states.

While the newly secured patent is primarily focused on a method for analyzing pre-recorded footage, chipmaking advancements could eventually allow a similar a solution to be implemented in real-time. This would have clear applications in streaming and potentially be a big deal for small eSports events whose organizers lack the technical know-how to implement broadcasts with traditional input captioning based on controller metadata.

In the meantime, the PS5 maker has also been exploring the idea of teaching machine learning models how to play games for demo purposes, according to another recent Sony patent describing a "what-if" gameplay system.

MORE: What the PS5 DualSense Edge Controller Can Learn From the Xbox Elite Controller