Artificial intelligence becomes a personal assistant in video games

At Computex 2024 Nvidia presented the G-Assist project, which aims to put gaming knowledge from around the world at players’ fingertips. Simply ask a question about your game or system performance, and the AI ​​provides contextualized answers. The G-Assist project receives voice or text input from the player, along with a snapshot of the game window. The snapshot is processed by computer vision models that provide context awareness and application-specific understanding for the language model (LLM), which is connected to a database of solutions and details about individual games. The result is a personalized, in-depth response, both text and voice, based on what’s happening on screen.

G-Assist’s vision and language models can be customized by developers for a specific game or app, ensuring a high degree of accuracy and insight. These custom AI models can be run in the cloud or accelerated locally by GeForce RTX AI PCs and laptops. To demonstrate the power of G-Assist, Nvidia partnered with Studio Wildcard for a technology demonstration using ARK: Survival Ascended, their dinosaur-filled open-world survival game developed with Unreal Engine 5. Activated via a key combination or a sentence, the assistant can help players with frequently asked questions that often require online searches: be it missions, items or difficult bosses to face.

Thanks to AI models that understand what is happening in the game window, the assistant can advise whether to avoid an enemy or suggest different approaches to deal with another. And, because the assistant is contextually aware, it can tailor its recommendations based on the player’s game: it can analyze skill points, the crafting menu and unlocked features, for example, and suggest what to choose to help players conquer higher level areas and enemies. G-Assist can also be useful in optimizing PC performance as well. For example, with a simple prompt, G-Assist can evaluate your system configuration and performance and instantly optimize them for an optimal experience. The AI ​​assistant can track frame rates, power consumption, PC latency and other hardware statistics, suggesting ways to improve them when required.