Why Virtual Reality Owes A Lot To Air Guitar

Because, dude! JL

Meghan Neal reports in Motherboard:

The very first app ever made on any of the original virtual reality equipment—which included the first gloves for interaction—was to play virtual musical instruments in the air.
Legend has it virtual reality started as a way to make a really good air guitar.
“Absolutely. Hell yes. Hell yes. Exactly,” virtual reality pioneer Jaron Lanier told me with characteristic good-natured enthusiasm when I asked if there’s truth to this anecdote.
“I know this sounds incredible,” Lanier said in a phone call Sunday, “but the very first app we ever made on any of the original virtual reality equipment—which included the first gloves for interaction—was to play virtual musical instruments in the air.”
“The very first virtual world I made was a musical one,” he continued. “The very first demo that shipped with a commercial VR system was virtual musical instruments.”
Lanier has held many titles over the course of his career: computer scientist, digital philosopher, author, artist, and most notably, the father of VR—he’s credited with coining the term “virtual reality.” But his first love is music. His Berkeley home is filled with not only the latest virtual and augmented reality gadgets, but also one of the world’s largest and most varied collections of actively played unusual ethnic instruments, which he uses to compose his “new classical” music.
At Moogfest this week, a music technology conference in Durham, North Carolina, Lanier will be giving a keynote about the future of creativity, and how virtual reality and music might continue to influence each other.
It’s a timely topic: The mixed reality platform is starting to take shape as a tool for creating music. Minority Report-style interfaces for electronic music production seem inevitable. With VR on the cusp of going mainstream, consumer-available headsets with motion controllers and haptic feedback devices are giving rise to a number of rudimentary interfaces for gesture-controlled virtual synthesizers, mixing software, and instruments.
Lanier was doing all this 25 years ago.
In Silicon Valley in the late 80s, Lanier was a dreadlocked computer scientist with a hippie idealism about how a new shared reality could enrich the human experience. After a lucrative stint working at Atari, Lanier founded VPL Research, the first company to sell VR goggles and gloves. There, he developed the first multi-person virtual worlds and the first avatars and experimented with making music from within virtual reality.
One of the first VR music performances, The Sound of One Hand, entailed Lanier gesticulating on stage playing virtual instruments while wearing a magnetic-sensor “DataGlove,” which generated music notes from his hand movements.

Lanier playing music. Image: Ars Electronica archives.

Screens projected to the audience what Lanier saw in the 3D stereoscopic world he viewed through “EyePhones,” an early VR headset prototype. A virtual version of his gloved hand played virtual instruments like the Cybersax and CyberXylo by measuring his physical movements.

Image: Jaron Lanier

The DataGlove wound up being contracted by NASA Ames for the agency’s experiments with virtual simulations, and Mattel made a knockoff version, Power Glove, for Nintendo video games. But Lanier helped push VR’s creative applications.
He introduced the immersive headset to the Grateful Dead (Jerry Garcia famously remarked, “Well, they outlawed LSD. It'll be interesting to see what they do with this"), and the band used the DataGlove to create the animation in one of their music videos.
“Initially it was a live effect—we had a skeletal hand made of data from early surgical simulation research and someone backstage would wear a glove to animate it—it was a big hit,” Lanier recalled. “Later it was used in a video, or maybe more than one, but I haven’t the slightest idea which ones.”

* * *

In its early days, virtual reality was rooted in the techno-utopian ideology that powerful computers had the potential to create a new form of communication and expand human thought, even inspire people to question reality and gain a greater spiritual understanding. The idea was unsurprisingly embraced by the cyberpunk counterculture and psychedelics advocates who imagined a virtual shared world would let people show each other their dreams.
Although the charismatic Lanier was VR’s biggest champion, he’s since backed away from that hippie idealism. His last two books, You Are Not a Gadget and Who Owns the Future criticize technology for exploiting people’s privacy and argue that the digital economy is quashing the middle class.
But Lanier continues to have a deeply humanist, and ultimately optimistic view of emerging technology, which manifests through music. Lanier told me he’s always been interested in finding a way to think about technology that’s more positive and pleasant than the dismal metaphors people toss around, imagining some new super-intelligence will make humans obsolete, or that we’re headed toward the Matrix.
“If you think about technology as a musical instrument,” he said, “you start to think of technology as something that’s all about human expression and human connection and making things more beautiful and more meaningful, and filling the world with flavor.”
He’s fond of pointing out that although people often think of warfare and weapons as the drivers of new technologies, music also tends to be at the cutting edge of innovation.
“We can go all the way back to ancient times. The musical bow had to proceed the bow and arrow. And canons actually started off as bells that were turned upright. And oh, so many other examples. Hewlett Packard—the formative Silicon Valley company—it’s very first product was a synthesizer for Walt Disney to use in Fantasia,” Lanier said.
“And I think you can add virtual reality to that list.”
Of course, various forms of virtual reality existed before Lanier started calling it virtual reality, as a marketing term. But the first consumer-facing VR systems were indeed conceived as a new form of human connection.
Ironically, we now use the term to refer to anything viewed in a headset like Oculus Rift or Samsung Gear, which is usually a solitary experiences. But “originally, [the definition of VR] was the social or multi-user version of virtual worlds,” Lanier said. “I know that because I made it up.”
It’s no surprise then that Lanier now runs a lab at Microsoft that’s working on developing multi-person augmented reality experiences with the HoloLens holographic goggles. Researchers there are experimenting with environments where multiple people can interact with the same virtual objects, such as visualized sound waves moving around a mixed reality room.
Lanier’s not making music with the HoloLens (yet), but it’s easy to see augmented reality heading in that direction.

"I have my big modular synthesizer then I have a HoloLens sitting next to it. So why am I not interfacing the HoLolens to the synthesizer just yet? It’s totally doable."

“It’s a funny thing. OK, so I have my big modular synthesizer then I have a HoloLens sitting next to it. So why am I not interfacing the HoloLens to the synthesizer just yet?” Lanier said. “It’s totally doable. What is wrong with me?! I could do it like, I dunno, any minute.”
True to form, Lanier is wary of letting a novel technology usurp organic creative inspiration. “I think if music isn’t intuitive it isn’t anything. And sometimes if you nerd out too much, if you get too involved in the tech, it can kind of get a little dry,” he said. “I’m not going to do it unless it’s this perfectly organic thing where it really means something, and it’s flowing, and it has this beautiful feeling to it.”
Lanier’s latest composition is a four-part symphony written for Moogfest. He gave Motherboard a sneak peak of the piece, along with the below video of him explaining the composition.
Aptly, Lanier and Robert Moog, one of the inventors of the modern day synthesizer, were friendly when Moog was alive. The two used to bond over a shared love of the theremin, an instrument played without any physical contact, invented a century ago by Léon Theremin.
The theremin is considered one of the first electronic music instruments, and the precursor to the synthesizer. But it may have done just as much to inspire virtual reality.
“I used to build them when I was a little kid,” Lanier said. “I made these things where you wave your hand in front of one of Léon’s theremins and it would control what are called lissajous patterns on it… I made these weird visual things happen with my hands. And that was like my first VR thing when I was, I don’t know, nine or something.”
At Stanford many years later, Lanier had the chance to meet the inventor and give him an early VR demo.
“He was like, vibrating,” Lanier recalled. “I don’t even know how to describe it. Like if you could imagine some star about to go supernova or something. Like, he was just so filled was excitement… It’s one of the high points for me of the whole VR experience.”

* * *

An explosion of electronic music followed the advent of the synthesizer. But with software synths, making music became increasingly reduced to pushing a few buttons on a laptop. And now we’re seeing a bubbling interest in using virtual and augmented reality as a way to bring physical gestures back to electronic composition—just like Lanier’s cyber-instruments of the early 90s.
A few early experiments already exist. You can now use the popular DAW Ableton Live with Oculus Rift, using custom motion control gloves. Virtual interfaces like Soundscape, Playthings, Lyra VR, and Imogen Heap’s Mi.Mu haptic gloves let users control and manipulate digital music with physical gestures instead of knobs and clicks.

"The direction that excites me the most is making music physical again. Because we’ve had a way of making music more and more abstract lately."

“Personally, I think the direction that excites me the most is making music physical again,” said Lanier. “Because we’ve had a way of making music more and more abstract lately. Just people making music on laptops, and you sit there moving your finger like a half an inch or something, clicking. Not that there’s anything wrong with that!”
“I’m really interested in the physicality and, if I can say this, the sensuality of music,” he went on. “That it’s not just about ideas. It’s not just in your head. But it’s part of this whole flow of how you exist in the universe physically, biologically.”
If you continue with this train of thought, simply creating a copy of reality in stereoscopic 360 may be thinking small. The virtual world opens the door to redefining reality. You could be the guitar. Or, as the blog XLR8R puts it, “throw a handful of notes into a virtual whirlpool of synthesizer piranhas.”
This is the kind of thing Lanier has been interested in, since developing the first avatars at VPL. With powerful enough software, he imaged, avatars would not just move the way humans do naturally, but could, for instance, embody a musical instrument and use it to control every aspect of a virtual world—to essentially “play” reality.
This could take the form of actual music, or something else. Maybe it’s a language for creation, using music to create architecture in real time—”to play cathedrals with a piano, as it were,” Lanier said in a 1996 interview with Scientific American.
Lanier experimented with reality-bending in his 1996 album Music From Inside Virtual Reality, which starts to blur the audio and visual realms. Different tracks generate music in different ways. One uses a physical saxophone to control the virtual world; another uses Lanier’s motions to bend and twist the structure and in turn, sound of virtual flutes; another uses people dancing in VR to generate the rhythm of the track.
“If you think about virtual reality simulating instruments that are physical and making an air guitar, that’s cool. It’s fun. It’s worth it. I’m sure people will like that. But you can go so many other places. Like what if you turn into the instrument?” Lanier said. “All these things are possible.”