For awhile, “listening to music” was sorta my thing. Certainly in the way that other kids at school were known for, say, being good at throwing baseballs or having access to cigarettes, I rigorously self-identified as “the kid who listens to music frequently.” I took this title seriously, and spent a disproportionate amount of my time and spare money in its pursuit. Remember: an era in which 60-78 minutes of new music was a hard-earned commodity. Eminem was both popular and critically adored. I worked the system, gaming Columbia Music Club for maximum returns and figuring out which used record stores offered the best returns. My older brother, with whom I was engaged in a sometimes-venomous competition to acquire the most of these plastic discs, gamed the system even harder, getting a job at a record store and then lifting dozens of them at a time when he took out the garbage. He won the competition, in the long run. The prize was a cardboard box full of CDs to lug around every time he moved for the next decade.
But a weird outgrowth of all this effort, at the time, was that it required I also put aside a lot of time to enjoy my rewards: that is, to recline in a lamp-lit room and let the meticulous tones of James Iha’s guitar wash over me, jewel case pressed to my chest. I had a lot of free time in those days, so this wasn’t hard to accommodate. It’s hard to say which came first: the avaricious research and acquisition of these discs, or the long hours spent actively enjoying them. But regardless, as one fell away over time—first getting burnt mixes from friends, then carrying gigs around at a time on an iPod, and now tossing $10 at a streaming service that has everything ever—so did the other. I listen to music all day, every day, but it’s background sound, while I work or clean or sit on the bus. It’s a net positive for society that “the kid who listens to music frequently” isn’t something someone has to buy their way into anymore, but I can’t help but feel a pang of agreement when people grouse that the devaluation of music has lead to, well, a devaluation of a very different sort. We can listen to everything, and many of us earnestly attempt to—but when was the last time we really listened to it?
“Like, what if you’re too stoned to pick up the controller?”
Now, this line of thinking can be insufferable—it’s only a step away from talking about “social media nowadays”—but I’ve found myself quietly lamenting those long, leisurely hours spent listening. Harmonix’s Jon Carter has been thinking about this, too. His project, the directly named Harmonix Music VR, is designed to remedy this anxiety by using the all-encompassing hold of a VR headset on the user’s attention. It demands that you sit and listen and do very little else. The software will work with any song or album; you choose a track, then one of a handful of pre-set “worlds” designed by Harmonix. Something called “music information retrieval” technology (MIR) digs into the bones of the music to create a customized set of 360-degree visuals. “We’re exploring other aspects too, beyond just zoning out to the music,” Carter told me, “so calling it a music visualizer is something we’re still doing by necessity. But it’s more a way of using virtual reality to bring you closer to your music in ways that haven’t been possible before.”
This is something Harmonix is uniquely positioned to do. Carter has been at the company for almost a decade, and worked on some of their biggest titles, many of which have had exactly this effect: transitioning music from something we do alongside other activities to the actual focal point of our attention. I lived the results of their efforts, year in and year out. When Guitar Hero dropped in 2005, my long-held position as “guy who made the party’s playlist” was usurped as everyone waited around for a turn on one of the game’s signature plastic instruments. We played those initial songs to death, and over the years we picked our way through the vast encyclopedia of music that was eventually available through Rock Band (2007). Similarly, my neighbors are probably still quaking from the month or so after I got my hands on Dance Central (2010). “There’s definitely something that’s in our company DNA,” Carter told me. “Giving you tasks or giving you tools you can play, while you’re taking in music, is something that we’re super interested in exploring.” Harmonix Music VR, though, sits on the milder, more passive end of the spectrum. “Like, what if you’re too stoned to pick up the controller?” he said. “The active versus passive thing is just an interesting approach to music appreciation.”
While Music VR is generally more passive than Harmonix’s other games, its interactivity is rare in the history of music visualization. The technology’s existence gets at the human curiosity toward synaesthesia, a self-evidently awesome condition in which sensations toward one sense are felt in the others; i.e. seeing music, hearing foods, and so on. It’s common in artists: Kandinsky thought of his paintings as visual music, and Nabokov thought of his words as having colors. Those of us without such flexible neural pathways have sought it elsewhere; our parents and grandparents had lava lamps and black lights to zone out to, and computers have been used toward this application almost since their invention. The creator of Pong (1972) made a wonky, very-70s piece of hardware called Atari Video Music (1976), and as early as 1984 the game designer Jeff Minter was working on visualizers with names like Psychedelia and his 1990 light synthesizer Virtual Light Machine. More recently, music visualizers like WinAmp and MilkDrop have given kids something to look at after they discover pot. Music VR, then, is an evolution of those visualizers, not only because of its interactivity, but because of the very nature of its imagery. Harmonix is calling it a “music spatializer,” which is one of the rare bits of marketing jargon that is also entirely accurate, and not without some poetry.
In order to make these worlds for us to listen in, Harmonix had to first teach the computers themselves to listen closely. Carter’s first convincing VR experience, for example, was one of a handful of Harmonix’s early VR experiments, built around Pink Floyd’s “Time”; it was set in a cabin full of clocks, and when the track kicked in, the walls blew away. “It was very on the nose, but also very effective,” Carter said. That sold him on the technology, but the commercial version couldn’t be so directly connected to a song’s theme. This is where the MIR comes in, digging into the skeletons of songs, isolating individual elements like kick-drums and snares that are then mapped onto various visual patterns. “We actually break them into sections,” he said. “So we can do this high-level structural analysis to make sure that things transform on section boundaries. Then we’ve got this, what we call, energy detection, it’s sort of a way of identifying the energy or feel of a song.” So a song that starts with low energy might be connected with low lighting; if there’s a huge leap in intensity coming down the line, it knows that and plans accordingly. Weather patterns and color schemes might change based on mood.
But is a fancier visualizer enough to change our increasingly passive relationship to music? Carter has an affable attitude toward the current state of VR—”Anyone who pretends to know what they’re doing in VR at the moment is probably just talking themselves up,” he told me—but his ambitions for the project are high. “Eventually the idea of music VR, ideally it does grow into a platform,” he said. “The title is intentionally generic.” While VR is by its nature an isolating experience, and Music VR certainly takes advantage of that all-encompassing nature, its use as a platform could be social. We’re already listening to music together virtually; albums by Radiohead, Beyonce, and My Bloody Valentine, to pick a handful of high-profile examples, turned into all-internet listening sessions upon their release, and Kanye’s trainwreck fashion-show album-debut was the capstone event of a few weeks of gloriously messy creativity. Musicians like Bjork and the Arcade Fire are already dabbling in VR, and so the notion of a bespoke VR debut experience does feel like an inevitability, albeit one likely enjoyed by a monied few.
If you make the admittedly enormous leap, though, that VR will create within the user a sense of “presence,” Music VR’s possibilities are the techno-utopian hippie ideals realized. I’ve had glorious experiences listening to music with enormous crowds of people on handfuls of questionable drugs, and, while I would not recommend that exact process to anyone, Music VR promises a means of democratizing it. Describing how the technology sits along Harmonix’s spectrum of passive and active music listening, Carter told me, “There’s certain objects and hotspots that you can work on and focus in. There’s psychedelics that come out of that and you can just zone out, and it’s a little more intense—for as long as you want. You just look away and it stops and you’re back. That’s ideally a stepping stone for people if they’re really not comfortable with the psychedelics.” This is the transcendence of drugs without the transgression of illegality, and an attempt to solve one of the greatest problems of tripping: that you can’t turn it off when you want to. Whether or not that undoes the entire point of doing drugs in the first place—the giving in—I can’t quite say. But a vast and colorful attempt to mainstream this sort of thinking about and appreciation of art is one of the most noble uses of a new technology I can imagine.
