How Amazon Is Using Trucks To Move Data Faster

Moving an exabyte of data over the web - about 250 million DVDs or 1 trillion 400 page books - takes a long time. 

 It turns out that transferring the data to a truck holding a massive storage device, downloading customer files to the truck, driving it to an Amazon facility, then transferring it to Amazon's cloud service, is faster. Which may be strange, but is another example of how the tangible and intangible are converging. JL

Jay Greene and Laura Stevens report in the Wall Street Journal:

Transporting data from companies to cloud providers has become immensely time-consuming as corporate data storage has ballooned from terabytes to petabytes to exabytes. Amazon plans to drive to customers, extract data, then to an Amazon facility where the information can be transferred to the cloud in far less time than it would for so much data to travel over the web, reducing the time to a little less than six months, from 26 years using a high-speed internet.
In Amazon Web Services, Amazon.com Inc. has built one of the most powerful computing networks in the world, on pace to post more than $12 billion in revenue this year.
But the retail giant proposed a surprising way to move data from large corporate customers’ data centers to its public cloud-computing operation: by truck.
Networks can move massive amounts of data only so fast. Trucks, it turns out, can move it faster.
To the sound of throbbing heavy-metal music and flashes of strobe lights, Amazon drove a big rig onto the floor of the Sands Expo & Convention Center during the company’s annual customer conference.
The tractor-trailer hauls a massive storage device, dubbed Snowmobile, in the form of a 45-foot shipping container that holds 100 petabytes of data. A petabyte is about 1 million gigabytes.
Transporting data from companies to cloud providers has become immensely time-consuming as corporate data storage has ballooned from terabytes to petabytes to exabytes, each step a factor of roughly 1,000 larger than the last.
“When we started AWS [in 2006], the notion of an exabyte of data just seemed completely out there,” AWS Chief Executive Andy Jassy said. “Today, an exabyte of data is much more common.” That is equivalent to about 250 million DVDs or one trillion books of 400 pages each.
Amazon plans to drive Snowmobiles to its customers’ offices, extract their data, then cruise to an Amazon facility where the information can be transferred to the cloud-computing network in far less time than it would for so much data to travel over the web.
The company, however, isn't promising lightning speed. Ten Snowmobiles would reduce the time it takes to move an exabyte from on-premises storage to Amazon’s cloud to a little less than six months, from about 26 years using a high-speed internet connection, by the company’s calculations.
During a press conference, Mr. Jassy declined to disclose the cost of each Snowmobile unit but noted that the company has “several” of them. The trucks are currently available and customers are using them, he said. Snowmobile will cost half a cent per gigabyte per month of use, or about $500,000 a month to use its full capacity.
This isn’t the first time Amazon has circumvented the web to move data faster. Last year it introduced a suitcase-sized data-transfer service appliance known as Snowball. Amazon on Wednesday upgraded that gadget, doubling its capacity to 100 terabytes.
But for customers who sought to move petabytes or even exabytes of data, the larger Snowball wouldn’t be enough.
“The first thing we thought was, ‘We’re going to need a bigger box,’ ” Mr. Jassy said.
The truck and the suitcase-sized device are part of the retail giant’s bid to woo large corporate customers, who have invested heavily in their own data centers, to move to Amazon’s cloud. Many have shifted key computing operations to cloud providers, but a significant number continue to run some jobs in their own facilities, even as they take advantage of public cloud providers, an arrangement known as hybrid cloud.
Relatively rapid data-transfer methods such as Snowmobile and Snowball may spur them to move faster to use Amazon’s services.
Nonetheless, Mr. Jassy acknowledged that the majority of companies likely would keep some computing operations in-house. “Most enterprises are going to operate in hybrid mode for many years to come,” he said.
That is one reason why Amazon announced a partnership last month with VMware Inc. to allow VMware customers to take advantage of AWS services without abandoning their data centers.
VMware Chief Executive Pat Gelsinger joined Mr. Jassy on stage Wednesday and called the response to the joint offering “overwhelming.”
At the conference, Amazon also debuted new services intended to help coders build web-based applications that tap into the retail giant’s artificial-intelligence capabilities.
Amazon Rekognition will let software developers write programs that detect the number of people in a photo, spot their gender and identify objects. It also can match faces, which could be useful in comparing two images to confirm, for example, a person’s identity.
The company also launched Amazon Polly, which converts text to speech. It lets programmers transform text input, such as “The temp in WA is 75 degrees F,” into spoken output that says, “The temperature in Washington is 75 degrees Fahrenheit.” This feature, available in 24 languages, could be used to build conversational applications.
Another new offering, Amazon Lex, affords access to the Alexa artificial-intelligence service that runs on the company’s Echo device. LEX understands spoken input, enabling developers to build software that answers questions.
A pizza company, for example, could build a web-based program that asks users what toppings they want when they order a pie. Such an application could keep track of previous orders so the system could ask customers if they wanted the same toppings again, Mr. Jassy explained.
“A lot of companies don’t realize the heritage that Amazon has in the machine-learning space,” Mr. Jassy said, referring to the area of artificial intelligence that makes such services practical.
The online retailer has thousands of employees focused on artificial intelligence, he noted. This could be used for services that show customers who bought a particular item; what other items they might like purchase or Echo’s Alexa AI assistant.
“We do a lot of AI at the company,” and Amazon Web Services customers wanted access to that, Mr. Jassy said.