Sony is exploring a technology that would allow it to track player gaze even without a virtual reality headset. According to its latest patent filings, the company has recently been experimenting with a heads-up display rendering solution which places HUD elements around the screen in a dynamic fashion, based on where the player is looking at.
The idea of using gaze tracking to inform HUD placement is something Sony has been researching since at least the PlayStation VR2 development cycle. The solution was initially explored as an extension of foveated rendering, an imaging technique powered by machine learning that allows VR headsets to conserve resources by only rendering high-definition textures in specific areas of the screen which players are focusing on. According to another recently surfaced patent filing, the PSVR 2 could feature this particular image enhancement.
Earlier this year, Sony filed to secure a patent for automatic HUD positioning based on gaze tracking. Instead of a VR headset, the experimental solution would use a single camera to track what the player is looking at. It would then analyze that information with a specialized machine learning model to render a custom view of a virtual environment which would only contain HUD elements in areas that the player is actively inspecting. The filing doesn't delve into any technical characteristics of the camera used for the gaze tracking solution it describes, though it does mention that the implementation would rely on both head movement and biometrics such as eye detection for accurate measurements. For context, the PS5 HD Camera is capable of both.
This is the second patent for a HUD positioning mechanism based on gaze tracking that Sony filed within the last year. The initial filing described a VR-only solution, though both technologies primarily seek to deliver more responsive and personalized gaming experiences.
Sony's ambitions to personalize games go beyond dynamic HUD rendering. According to a different patent discovered over the summer, the company is also researching an AI system that could modify game content based on real-time player gaze information.
It's unclear whether Sony has already prototyped the automatic HUD positioning solution described in its latest patent. The filing itself also doesn't guarantee that this technology will ever be commercialized. It does, however, reaffirm that Sony is still actively pursuing gaming innovation in the eye tracking sphere even beyond VR, which remains a niche market segment due to the high cost of entry and a general lack of compelling AAA titles, among other reasons. Since the PlayStation VR2 release date is just a few months away, more details about Sony's gaze-based gaming enhancements might follow shortly.