AI Helps Scientists Reconstruct Pink Floyd Song From Brain Activity

The implication is that AI may be able to enhance communications, especially for people who have suffered strokes or other brain injuries. JL 

Daniela Hernandez reports in the Wall Street Journal:

Neuroscientists at the University of California, Berkeley reconstructed Pink Floyd’s classic rock song “Another Brick in the Wall, Part 1” using recordings of the brain activity of 29 patients who heard the song while undergoing brain surgery. Though less polished than the band’s version, the reconstructed tune was recognizable. Visual versions of the original and reconstructed songs also resembled each other. That was the case for individual patients and all 29 in aggregate. With further development, this could give patients who have lost the ability to speak—because of a stroke or other brain injuries—the power to communicate in a way that sounds more natural and less robotic.

In a scientific breakthrough, researchers have gotten one step closer to developing devices that say aloud what people are thinking. 

With further development, this could give patients who have lost the ability to speak—because of a stroke or other brain injuries—the power to communicate in a way that sounds more natural and less robotic.

Neuroscientists at the University of California, Berkeley reconstructed Pink Floyd’s classic rock song “Another Brick in the Wall, Part 1” using recordings of the brain activity of 29 patients who heard the song while undergoing brain surgery. Though less polished than the band’s version, the reconstructed tune was recognizable. 

The study was published Tuesday in the journal PLOS Biology.

The reconstruction of complex musical elements from brain-activity patterns advances the development of therapeutic technologies to help patients regain the ability to communicate. 

“It’s very exciting. What they’re showing is that they can get pretty high performance with a relatively low amount of data,” said Dr. Edward Chang, a neurosurgeon at the University of California, San Francisco’s Weill Institute for Neurosciences who studies music and language perception.

Chang wasn’t involved in the study but has previously worked with the study authors. 

As computer models and artificial intelligence have become more sophisticated, similar studies using brain-computer interfaces—devices that interact with the brain—have re-created images and produced simple language from neural data. 

Last year, Elon Musk’s Neuralink, for instance, showed a video of “telepathic typing” from a monkey with a Neuralink brain implant, which contains threadlike electrodes—the equivalent of microphones for neural activity—that penetrate the brain. The animal wasn’t typing into a keyboard but was able to move a cursor to images of letters.

In the new study, researchers took recordings from different regions of the brain using a grid-like pad studded with electrodes placed on the organ’s surface as patients underwent surgery for treatment-resistant epilepsy. Each heard the Pink Floyd song a single time.

The researchers then homed in on some brain areas, like the superior temporal gyrus in the auditory cortex, which is responsible for processing sounds. The activity there—monitored by only a handful of electrodes—tracked most closely with the hit song. The team found they could largely tune out the rest of the data and still get recognizable, if somewhat garbled, playbacks. 

The reconstructions from neural activity (bottom three panels) look visually similar to the original song (top panel). PHOTO: LUDOVIC BELLIER

Visual versions of the original and reconstructed songs—which look like a mashup of a heat map and an audio waveform—also resembled each other. That was the case for individual patients and all 29 in aggregate.  

“Some of the richness of the original audio frequencies isn’t there in the reconstructions, but it looks like a pretty good approximation, even visually,” said Dr. AZA Allsop, a musician and assistant professor of psychiatry at Yale University and Howard University who wasn’t involved in the study. 

“It goes to stir the imagination in terms of what this could enable” in the long term, he added, including the ability to compose music directly from neural recordings. 

The researchers’ goal is to leverage the technology to develop neural prosthetics that can give patients back the ability to speak more naturally, according to Dr. Robert Knight, a University of California, Berkeley neuroscientist and an author on the new study. 

Currently, “the output of these devices has a monotonic, kind of robotic quality” he said. “Music, given its strong emotional and rhythmic components, would allow us to add an effectual component to [computer-generated] speech.” The new algorithms also were able to sing back partial vocals. The playback of one patient’s brain activity, for example, contained a few discernible words, including “all,” “was” and “just a brick,” according to the study.

The researchers chose “Another Brick in the Wall, Part 1” because they are fans of Pink Floyd, the English rock band that recorded the song in the 1970s, Knight said. Part 1 is the first segment of a three-part composition and is less well-known than Part 2, the chart-topping single most people recall. 

“It’s something that people are kind of familiar with, but not everyone has heard it” so they might not have too many memories associated with it, said Ludovic Bellier, a computational neuroscientist and another of the study’s authors. “It’s kind of in a sweet spot of familiarity.”    

That sweet spot likely differs by age group, personal preference and culture, said Allsop. The algorithms used to reconstruct the Pink Floyd jam will likely need to be retrained on other musical genres with different melodies and rhythms, such as jazz or reggaeton, according to Chang. 

Another consideration—recalled in Pink Floyd’s lyric “we don’t need no thought control”—speaks to concerns some neuroethicists, neuroscientists and legal experts have about mental privacy.

 

Algorithms already can construct pretty accurate models of a person’s preferences based on physical activities, including typing search terms or tapping to “like” a photo. While there are high hopes that reconstructing words and music from neural activity could enable applications that promote mental health and social connection, reading and interpreting thoughts is regarded as the next privacy frontier. 

“This could mark a completely different reach inside the mind,” said Lucy Nalbach Tournas, a board member of the Institute of Neuroethics. “It speaks to how powerful music is.” 

The ability of music to move people and spark their curiosity is something that Knight quickly realized after doing this work. 

“The media hasn’t really been interested in anything I’ve done in the last 40 years, but it’s all been made up for by Pink Floyd,” he said. “It’s gotten a little crazy in terms of people’s interest, which is good.”