This article is part of a collaboration with iQ by Intel.
After the revelation that the NSA was using widespread data collection to keep tabs on pretty much everyone, the luddite defense force came out to crucify Big Data. Jaron Lanier penned for the New York Times a particularly alarming editorial. “The new class of ultra-influential computers come in many guises. Some run financial schemes, like high-frequency trading, and others run insurance companies. Some run elections, and others run giant online stores,” he wrote, explaining how usually the purpose of these super-servers is to tabulate ways to increase profit at the expense of you and me.
He makes a convincing if polarized case: Big Data is across the board a misanthropic disaster. But it may be rash to vilify a technology because companies want to use it to maximize their intake. There are plenty of good uses for Big Data, too. IBM’s Jeopardy!-winning Watson, for instance, is being used in Africa to diagnose problems like cholera outbreaks and water scarcity. Server farms are responsible for Google being able to catalogue the Web. And these same techniques have helped League of Legends become a perennial favorite among esport circles. The competitive multiplayer RPG, in which two teams of players pick one of a hundred “champions” to attack the others’ base, has a daunting 27 million daily users. Without a way to sort out the data, there’d be no chance to balance the game among its hundred-plus unique champions effectively.
“The whole point of League of Legends is to be on the Internet with this very rich, fat client and competing against other people,” says Bill Grosso, CEO of Scientific Revenue analytics group, explaining how the advent of online gaming has shifted game companies’ entire philosophies. Before the Internet, when a company shipped a product it was a done deal. But now online games are perpetually under development, constantly being upgraded and tweaked. Since these games are always-on, they might as well start storing data to improve the game. But, says Grosso, “This quickly becomes an overwhelming amount of information. It’s scary big!”
Thus, server farms were born. Lanier describes them in his article as “gigantic facilities, located in obscure places where they have their own power plants and some special hookup to nature, like a remote river, that allows them to cool a fantastic amount of waste heat.” The purpose of these unmanned architectural beasts gobbling up the hinterland is to give shelter to legions of computer servers performing calculations that were impossible previously. “When you have 20 million people playing a game, and you’re getting telemetry data, behavioral data, and game activity data and storing it somewhere, the old model breaks completely,” Grosso says. To put it another way, without server farms, there would be no DOTA, no League of Legends, no Diablo 3.
To effectively manage all those chain vests and chalices of harmony requires a change in the way that we think about computing. One might think that the key would be building a bigger, badder supercomputer, but that just proves slow and clumsy. Grosso elaborates that it’s vastly better to use a large numbers of smaller computers connected loosely. “Standard server class computers, like you can get from Dell. You have a whole bunch of these guys,” he says. Once in the network, the data analysis is farmed out to every computer, each doing its part. It’s that moment in technology’s infancy where it learns to share. That’s not so misanthropic after all.
Inline image via Henrik Bennetsen