Microsoft’s Facial Recognition: Designing Games with Your Emotions

A new patent from Microsoft shows how they want to record your facial expressions, show them to an AI, and have it redesign the game you’re playing. It’s not only your face they want, but also any words you might be muttering, growls you might be making, or, dare I say, even purrs.

The broader technology described in the patent is a process for game designers to prompt an AI to build out the narrative elements of a game, including objectives, sidequests, NPCs, dialogue, and items. Rather than stop at that first pass, the process would continue when players are in the game, analyzing how they’re playing to then generate new content based on their actions and reactions.

A quick peek behind the curtain: I, too, am using this technology and the rest of this article has been rewritten to match exactly the facial expression you are making right… now.

US patent 20260183672A1, pithily titled Generative Narrative Game Experience With Player Feedback, doesn’t describe a technology Microsoft have already made – it is more a case of describing a process of how this tech would work if they were to make it. As ever with patent documents, they are as much a case of laying the groundwork to say “I thought of that first” to a judge when one of your competitors does a thing, as much as they are something you actually intend to do. They also often have the quality of talking to a child describing the spaceship they would make if they were given half the chance – “…and its legs have pipes filled with jelly, so when it lands it can turn the ground into trampolines”. As such, you can’t read a patent document as the company has done a thing so much as the company has thought about doing a thing.

Anyway, onto the thing Microsoft wants everyone to know they have thought about.

The patent describes a few ways the process would work. In one, it’s predominantly on the dev side. The patent suggests that a designer could prompt an AI, using written input, images, and audio files, and the engine would then create characters, narratives, and worlds “at a high level” that the designer would finesse. That finessing would either be by directly editing what the model created or by giving it more prompts to refine what was in front of them. Once the designer was happy with what they had made, they could implement it within a game.

The authors suggest the prompt could be “merely a general theme, idea, or goal for the game or characterize the general nature of the game, or it may include some narrative details, such as main characters, a basic storyline, etc.” The model would then go on to create game content – NPCs, items, dialogue, lore descriptions, and objectives and sidequests. This would then be given back to the designer as “a visual representation of the game […] in the form of text, images, and/or graphs”.

Personally, I find the vagueness here delightful. All I can picture is the creators of Deep Thought, the massive supercomputer in the Hitchhiker’s Guide to the Galaxy, prompting the machine for the answer to the ultimate question: Life! The Universe! Everything! And receiving the answer 42. I can imagine a designer sitting at their desktop, writing ‘Computer, build me a game about the theme of grief’, only to be presented with a flowchart ‘Main character loves person > loved person dies > Main character = sad”. Thank you, computer.

The patent does go on to say that along with the visual representation, the model may also provide code you can stick into a game engine. Alternatively, they say an already existing game, such as Minecraft, could be updated with new content via an api. Well, that all sounds marvellously simple.

Another way the tech would work is for that design role to be taken up by the player – though, not through direct prompting, as with the designer, but with an AI adapting the world based on the player’s actions and “engagement metrics”. In this example, it’s not something that happens on the fly as you’re playing, but either while the game is still in development or between release builds. So, for instance, the players would be QA testers, beta testers, or early access players. Data would be collected from their play and fed into the model and it would spit out content for a designer to work with before officially incorporating it into the game.

Though, they also then go on to describe interactions which are very much on-the-fly calls to a generative AI, similar to the conversations we’ve seen players have with NPCs in games like Where Winds Meet. The patent’s authors say that the model would take what a player asked an NPC, along with contextual information such as where they are in the game, what they’ve been talking about with other NPCs, and the story path they’ve taken in the game, and stuff that into a generative AI to come up with a response. The capabilities of conversational agents are described as virtually limitless by enthusiastic writers. These agents can provide natural-language output in various ways, as suggested by Microsoft’s engineers. They propose a more advanced approach that involves not only generating dialogue for non-player characters (NPCs) but also creating new game assets, rules, and alterations to the game world based on the conversation. This concept opens up possibilities for interactive storytelling and gameplay experiences, such as convincing NPCs to turn off gravity in the game.

In addition to interacting with NPCs, generative agents could influence other elements in the game, such as objects that react to player actions in unique ways. The potential for emergent gameplay and narrative elements is highlighted, showcasing a new level of player engagement and immersion in the gaming experience.

While these ideas are innovative and exciting, it is important to note that Microsoft has not implemented them yet. The patent filed by Microsoft outlines a vision for AI-driven game design that adapts to player preferences and behaviors. This includes analyzing player data such as quest completion rates and interaction patterns to enhance gameplay experiences.

However, the use of cameras and microphones to gather player feedback during gameplay raises privacy concerns. Microsoft’s suggestion to use live streams and facial expressions to inform AI decisions may lead to intrusive monitoring practices. The ultimate goal is to create games that adapt in real-time to player reactions, tailoring the experience to maximize engagement and enjoyment.

Despite the potential benefits of AI-driven game design, there are ethical considerations to address, such as ensuring player consent and data protection. The AI’s ability to modify game content based on emotional responses raises questions about player agency and the impact on storytelling integrity.

In conclusion, while Microsoft’s AI-driven game design concept is ambitious, it also raises important questions about player privacy and autonomy. The idea of personalized gameplay experiences tailored to individual preferences is enticing, but it must be implemented responsibly to maintain player trust and ethical standards in the gaming industry. As technology continues to evolve, finding a balance between innovation and ethics will be crucial for shaping the future of interactive entertainment. Sure, here is the transformed content in HTML format: