In his 11 years at Harmonix Josh Randall has worked on some of the Cambridge, Mass. company’s highest-profile hits, most recently as creative director for The Beatles: Rock Band. Over the past five months Randall has been tapping into his alter-ego—video artist and musician Robotkid—to head the development of VidRhythm, Harmonix’s new audio/video sampling and remix iOS app. We called Randall at Harmonix HQ and spoke about some of the strange and creative uses the app has found since its release last week.
/ / /
To kick this off, it is fascinating to me that although VidRhythm has been getting a lot of coverage in the games community, the thing that strikes me about the app is that it’s—you know—not a game at all.
[Laughs] Right!
And I tried to go into this conversation with an objective perspective, but I’ve got to say that it is the most fun I’ve had with an iPhone app. It’s the first one that I’ve ever used that has had me break out laughing while using it. And I wanted to touch both on how you personally, and you as a company, made this shift toward creating something that is obviously interactive, but definitely not a game.
I’ve been interested in gluing together audio and visuals since art school. One night I went to this show that was opened by this group called “Emergency Broadcast Network (EBN).”
EBN were from Providence, and were really early pioneers of what I’ve been calling video music. Their big thing was that they remixed George Bush Sr. saying, “We will, we will rock you!” And they basically did a lot of remixes from the first war in Iraq.
Was this live remixing?
They would actually sift through hours and hours and hours of videotape before shows. They would be constantly taping TV. And then they would find samples that they liked, and would drop those samples into an old Roland S-50 audio sampling keyboard and would be able to make a song. Then, after they were happy with how the song came out they would print that tape, and then go back and re-find all those video snippets and re-edit them on two VHS decks to line back up with the audio portion.
Wow. You really would have had to be committed to your art form back then.
Oh yeah. And so I became friends with those guys and worked with them a bit, and over time I made my own band that did synchronized videos. And then I got into VJing, then DJing, and over the years I’ve been performing in clubs and trying to do a blend of visual mashups where I’ll remix movies and music videos, but also try to DJ them like a regular DJ would.
After living and breathing this stuff for so long, I’ve always wanted to figure out how to bring that into an interactive space. I’ve been at Harmonix for 11 years now, and though I’m the creative director now, I started out as the music director on our first few projects. And when I found out that the iPad 2 would have a front-facing camera, I started thinking, “Wouldn’t it be cool if we could make an app that allowed people to sample themselves and create these mashups that would be really easy to do, and—importantly—would probably be really fun to make?” Because, typically doing this stuff with modern-day software, like After Effects or Final Cut—
Or even something like [commercial VJ software] Modul8 or Resolume…
Oh yeah, even that stuff. To get all of that stuff synched up and working, even the simplest thing, could be days of work. It is absolutely brutal. And I’ve been feeling that fatigue, because I really love this art form, but trying to get into it is just way too much work.
So I was thinking that it would be great to have a portable video remixer, and it just so happens that these iOS devices are totally suited for that kind of thing.
And around that time, it was right after Harmonix had become an independent studio again. And we were regrouping and trying to figure out what we wanted to do next with the company. And I’d told Alex Rigopulos, our CEO and cofounder, that I’ve been wanting to do this VJ app. He had some tweaks that he thought would make it a whole lot more fun for the people who used it.
What kind of tweaks were those? I’m interested in how this idea you had became something a little more consumer-friendly.
Well, my original concept was that we would make a remix tool, and people who were already into creating mashups would dig it, but most casual users won’t really dive deep into it. But Alex was thinking, “What if we made this a little more conceptually like Mad Libs,” where we would prompt the player to say certain things, or sing certain things, and then take that input and play it with MIDI. Alex’s tweak was that the user only had to do one thing, and then the app would automatically make the music for you.
Right, the startup cost is so low for the person who fires up the app. It is immediately clear to the new user what he has to do do.
I recognized that was a much better direction. And the app is also in tune with Harmonix’s philosophy. We’ve been trying to give people that sort of buzz of performing, or of creating music, even if they aren’t performers or musicians. This is very much along those same lines, even though it is certainly not a game.
The first thing that I did was to use shots of myself, using the front-facing camera to capture my own face—which was disturbing to my wife. And the video I created was this wall of screens with pictures of my face singing Beethoven, and she was saying, “I can’t take this, it is too much for me to deal with,” and then she left the room.
[Laughs]
But when things got really interesting for me was when I got the second level of interacting with the app, when I started pointing the camera at other people, and turning it on things in the environment. What are you seeing in these early days, as people start to discover it and interact with VidRhythm more?
When we were doing internal focus testing, the first thing they did was just follow the onscreen directions. But the second time, they started to get that creative spark. I’ve been checking YouTube like every five minutes for the past week, and it didn’t take that long for people to do cool stuff like, “Here are all sounds captured inside of my car,” or “Here are sounds from me having breakfast.” Or, some guy brought the app into his dentist’s office and sampled things like the dental drill.
I’ve seen people start to do the live-band thing, and take a guitar and sample one note, or take a drum kit. There was a balance we were trying to strike where, yes, it would be super-easy for anyone to step up to it and use it, but if you wanted to, you could go deep with it and explore your creativity.
There is obviously also a balance between providing a tool set for people to do this stuff, but also the structure where it is easy for them to do it. If you do release more content, where do you see the users’ input stop and Harmonix’s control over themes and songs begin?
We have tried to give people a base tool set so that if you wanted to make just a clean, straight-up, video-only thing, you could do that using the video stock theme called Video Wall. And there are other ones where if you want to get goofy, you can choose the cats theme, or the dogs theme.
If we wanted to let people get even more creative, one idea might be for us to add more themes that are more vanilla, so people can impose their own creativity on it—so that there isn’t so much, “There are pictures of other peoples’ cats on here already,” but pictures of their own cats, for example.
So the style or theme is one area for customization, and the song choice is the other. I was wondering how you guys came to the choices you made for the recognized songs—the Scott Joplin, the Beethoven, etc. And then there are the extra songs that were created specifically for the app.
Well, this was a really short project cycle for us. The team for most of the project was five or six people, and then we “ballooned” up to eight people. The Beatles: Rock Band was like 200-something people, and everything was planned out. So it was really interesting to say, “We are just going to bang this thing out and then see what the public’s response is.”
And how long did that take?
From start to finish, about five months. We knew from the start that we wanted to get in some recognizable songs. Early on I did a prototype where I grabbed a random classical MIDI file—Flight of the Bumblebee—off the web, and ran it through Resolume. The result was jus so absurd, it was so bonkers, that we all immediately thought, “Ooo-kay, this is hilarious!” The song was recognizable, and is public domain, and so we started to look for other public-domain, recognizable songs that we could take and wouldn’t get slowed down with music-licensing issues.
And then we looked at those recognizable songs and asked ourselves, “What is cool about these? And, how can we create new songs like that that will actually work in the app?” We opened the process up to the entire Harmonix audio team, and gave them simple rules: it should be about a minute long, and should be super-catchy.
I think you’ll notice that a lot of songs on the app are by Steve Pardo. And his stuff is just awesome; he really managed to figure out the aesthetic of the app went crazy with it. I think he somehow has a natural ear for this stuff.
I love the idea of someone having an innate capacity for composing VidRhythm-type music!
When people with a lot of spare time and drive to create these mashups start taking the templates you guys have created and replace, or don’t even pay attention to, the directions, I think they will start to come up with some really crazy stuff. There are not many things that allow you to do that kind of thing so easily. You have the tool set now, and there is little-to-no overhead to learn this stuff—unlike the commercial applications—which is just fantastic.
I hope that this is a gateway drug to using other tools. And if people get really psyched about making their own special video music, I hope that it excites them to go ahead and do more with this stuff. It’s actually a lot like the philosophy behind many of our games: here is a taste of how you can be creative with this instrument, and if you get into it, there is a whole world out there where you can go even deeper and deeper.
Mathias Crawford (@mfcrawford) is a Canadian living in San Francisco, where he designs games for Natron Baxter and is pursuing graduate studies at Stanford University. In his spare time he makes awesome VidRhythm videos like this.
Image of Megatron/Matrix by Nam June Paik, via nzebula. Interview edited for length and clarity