A recent paper concerning the generativity of neural networks posed a question:
Can a neural model running in real-time simulate a complex game at high quality?
In this work we demonstrate that the answer is yes. Specifically, we show that a complex video game, the iconic game DOOM, can be run on a neural network (an augmented version of the open Stable Diffusion v1.4 (Rombach et al., 2022)), in real-time, while achieving a visual quality comparable to that of the original game. While not an exact simulation, the neural model is able to perform complex game state updates, such as tallying health and ammo, attacking enemies, damaging objects, opening doors, and persist the game state over long trajectories.
GameNGen answers one of the important questions on the road towards a new paradigm for game engines, one where games are automatically generated, similarly to how images and videos are generated by neural models in recent years. Key questions remain, such as how these neural game engines would be trained and how games would be effectively created in the first place, including how to best leverage human inputs. We are nevertheless extremely excited for the possibilities of this new paradigm.
We're excited too, because this new paradigm is not limited to video games. It has more mundane applications: it opens up entirely new search spaces. Those which are a feature of both antiquity and science fiction. It hints at generative memory palaces. Interactive representations of saved things that can be inhabited by end users. To get a feel for this speculative possibility, let's examine its constituent elements.
First, we have an end user's collection of saved things. An annotated, meaningful assortment of succinct references to artefacts that have been encountered across space and time. These saved things could be consolidated in a single resource or system, or spread across many apps, stores, platforms and hidey-holes.
Second, we have the concept of memory palaces:
...an imaginal technique known to the ancient Greeks and Romans and described by Yates (1966) in her book The Art of Memory as well as by Luria (1969). In this technique the subject memorizes the layout of some building, or the arrangement of shops on a street, or any geographical entity which is composed of a number of discrete loci. When desiring to remember a set of items the subject 'walks' through these loci in their imagination and commits an item to each one by forming an image between the item and any feature of that locus. Retrieval of items is achieved by 'walking' through the loci, allowing the latter to activate the desired items.
A memory palace is an abstract, imagined place that's highly salient to its owner and that is designed to yield information via queries that map to fundamental metaphors of physical being.
Third, we have the realm of spatial computing. This is a domain of human-computer interaction that leans into the embodied cognition of humans and attempts to materialise novel interfaces that are richer, more intuitive and more powerful than conventional graphical user interfaces. Weaker versions of spatial computing are approaches exemplified by organisations like Dynamicland. Stronger versions are those we associate with science fiction—immersive headsets and the like.
Finally, we have the capacity to procedurally generate interactive worlds and agents. This could be the more cutting edge world-gen approaches as shown in the GameNGen paper above, as well as the agent-gen approaches exhibited by the recent Project Sid and the older NeuralMMO. This could also be the more established approaches of current procedural generation operative in video games now.
Saved things; memory palaces; spatial computing; generative worlds and agents—smash them together and what do you get? Here's an imaginary ethnography vignette that fleshes it out...
He ambles to a cozy, cluttered nook carved out of his squished apartment. His workspace. Settling into the familiar chair, he sighs. The trip's over, and work beckons. But not quite yet.
"Aria, let's review some saved items from the trip," he says, reaching for his headset and donning it.
"Of course, sir," Aria responds, a hint of amusement in their voice.
"Procrastinating, are we?"
He chuckles, slipping on the headset.
"Just getting in the right headspace."
"Sure," replies Aria.
"Can you put on the 'new finds' playlist for me?"
A gentle melody trickles into his ears whilst his grand virtual library materialises around him, the pulsing tomes acting as beacons of sense and coherence amidst the aesthetically erratic ordering of books.
He filters for the additions from the weekend and browses idly, summoning snippets of conversations, fleeting thoughts, and other stored context. A faintly glowing volume catches his eye. He looks closer at the accompanying semantics. His heart rate spikes a little.
"What's this one, Aria?"
"Jao sent it to you on Saturday. It's a preprint of a meta-analysis about how infants experience and develop agency in childcare settings. It's going to make quite the splash—to say its conclusions contrast with established consensus is an understatement."
He nods, and picks up the blank book on his tiny desk, along with a stylus. As he opens it, pages fill with text and diagrams. Flipping through, he drags the stylus' point over the meta-analysis's sentences and his idle curiosity transforms into focused interest.
"This... this could change everything for Sarah's lesson planning project," he mutters, an idea forming.
He sends it to her with a note; the morning's meeting that was slated for crisis management? It may have transformed into a wedge for a new opportunity.
On a whim, he shares it with his friend Alex, too. They've always bonded over the obviously important but often absurd nature of his work—even if they always disagree on what to do about it. Alex will appreciate this latest find for the dramatic conclusions, at least.
"Aria, can you connect me with Sarah?" he asks, a new energy in his words.
Sarah's voice echoes around him.
"This early. On a Monday. Really."
He smiles and says nothing whilst Sarah scans the paper. A minute later, her face appears in view.
"Hmm," she says.
"You don't agree?"
She raises an eyebrow.
"You do?"
Her eyebrow raises an increment more.
"Okay, let's meet ten before then. Ciao."
He cuts her connection, imagining the exclamation and exasperation it would provoke and looking forward to the comeback. He sighs again.
"Alright, Aria," he says, preparing himself for more mundane work with renewed purpose, "I guess it's time we get started."
"Indeed, sir," Aria responds, a chirp in her voice. "Shall I pull up your priority tasks?"
"No, let's start with the urgents."
Aria doesn't respond. He turns his head, locates her virtual form. She's grinning and holding up two fingers. He nods. Aria keeps one finger raised and shows him a picture. He nods, intrigued. Aria raises a second finger and shows him another image.
"Jenny's birthday," he says.
"June 26th, sir. A Wednesday. I'd say that's both urgent and a priority. We should probably start there."
"You would say that," he mutters.
His library blurs and a screen appears in front of him. He opens the browser, pulls up the e-store and summons the possible presents he'd saved for this year.
"Aria, some help please..."
