This Gamer Turned EEG Tech Into a Game Controller

Perri Karyal, a video-game and variety streaming-video broadcaster, has created a way to play games with her brain waves. Karyal, who hosts a regular show on twitch.tv, has been using a custom controller that reads electroencephalogram signals (EEG) to play the popular video game Elden Ring since earlier this year. This month, Karyal demonstrated a fully hands-free version of the controller live on her stream.

Karyal says she designed the system to recognize repeatable patterns in EEG measurements–electrical oscillations recorded from a headset placed on a person’s scalp–and to in turn simulate the act of pressing a button on a game pad or keyboard. This allows her to control her Elden Ring character–swinging weapons, casting spells, and dodging enemy attacks–by evoking those same patterns in brain activity.

“I have spent more hours training the EEG than I have in the game, and I have over 250 hours in the game.”
—Perri Karyal, Twitch gaming streamer

Karyal has completed the game using early versions of the system, which could produce only a small number of inputs. But she has since extended her homemade tech to recognize enough distinct EEG patterns to replace every input she needs to play the game. When combined with a pair of headset-mounted accelerometers that replace character movement and camera controls, the headset becomes a complete, hands-free game controller.

Karyal, who has a postgraduate degree in psychology, created the system while looking for ways to connect aspects of psychology and neuroscience into gaming streams. She decided to use EEG, a technology she used extensively in her own academic work, after learning that consumer-ready versions were accessible outside of hospitals and university laboratories. “I came across, accidentally, that you could get these Bluetooth EEGs,” she says. “I had no idea. It was within my equipment budget, so I thought ‘I’ve got to try this.’”

To build the Elden Ring controller, she started with an EEG headset and accompanying software offered by Emotiv, a consumer-neurotech company. To convert EEG signals into inputs that the game could recognize, Karyal trained the software to recognize brain-wave patterns she generated while imagining different physical movements. This technique, known as motor imagery, looks for average patterns of neural activity from a person’s brain while they imagine themselves moving their body in a given repeatable pattern. “I was visualizing pushing something forward in my head,” says Karyal. “The pattern of activity my brain happened to consistently make during that task is what the software learned to recognize.”

While EEG signals are highly variable, common patterns within them can be recognized and decoded from repeated mental actions. Karyal says It takes some effort to produce enough data to effectively train the system to recognize a given behavior. “For the Elden Ring, I trained it 429 times to recognize a single activity, me imagining pushing something forward,” she says. “I have spent more hours training the EEG than I have in the game, and I have over 250 hours in the game.”

In order to defeat another Elden Ring character, Godrick the Grafted [above], Perri Karyal used an attack visualization in which she imagined pushing a heavy object forward. Throughout her gameplay, Karyal moves her character via the tilt of her head, which her EEG device senses.Perri Karyal

Karyal doesn’t want those hurdles to discourage people from trying this themselves, however. “I don’t think anything I’ve done so far is necessarily hard to do, if you know what you’re doing, if you have the passion for it and really want to try,” she says. Karyal also doesn’t want people to misunderstand what the device is actually reading and recording when it translates EEG signals into game-controller action. “People assume that it’s thoughts, like, I’m thinking ‘run forward’ and my character does it,” she says. “It’s not, though I really hope we can get to that point one day.”

In the future, Karyal and her team hope to expand the system to recognize other biometric signals as inputs. “We’re thinking about eye tracking,” says Karyal, “looking at a particular corner of the screen, for example. We need to find an easy and quick way to heal,” referring to Elden Ring’s often punishing treatment of the player character. “There currently isn’t one.”

Karyal’s EEG controller is one part of a larger movement of DIY neurotechnology, in which hobbyists and neuroscience enthusiasts have brought technologies designed for and largely limited to academic and medical applications into more mainstream and casual use. Beyond hobbyist activity, larger video-game developers and publishers have been developing neurotech into video games.

Valve, developers of the popular Half-Life series and owners of the online PC game store Steam, have worked with virtual-reality company Varjo and neurotechnology company OpenBCI to develop the Deckard brain-computer interface platform. While the company has not yet released any explicitly BCI-enabled games, the company’s CEO Gabe Newell stated in a 2021 interview with IGN, “If you’re a software developer in 2022 who doesn’t have [a BCI headset] in your test lab, you’re making a silly mistake.”