The Big Data Future Has Arrived

Big data, as the following article explains, is beginning to deliver on its promise by providing insights not previously visible or understandable.

The challenge is in applying that knowledge intelligently given the pressures its discovery creates. JL

Michael Malone reports in the Wall Street Journal:

It is now possible to measure everything, from the movement of billions of stars to each heartbeat. When you can study billions, even trillions, of data points you begin to uncover forces and trends that until now have always been invisible to human observers. (But) big data is about to ask some cultural questions for which we have no ready answers, much less consensus. For example, who owns our personal data?
Big data, the tech story of a few years ago, is now beginning to show big results. The science of using powerful computers, ubiquitous sensors and the Web to crunch mountains of raw data to uncover previously invisible insights is increasingly used in businesses, universities and government agencies. It is transforming our understanding of everything from fetal development to cosmology.
Already, thanks to big data, we have learned that toddlers learn language not from repetition—which we’ve thought for centuries—but by hearing words used in multiple contexts. We’ve also found that premature babies are at greatest risk when their heartbeats are stable (healthy baby hearts are more erratic). Researchers are making inroads into understanding the external influences on autism (carbon-dioxide levels, room temperature) and how urban crime can largely be isolated to just a few blocks, even individual buildings. These and other big-data breakthroughs can be seen Wednesday night in the PBS documentary “The Human Face of Big Data,” based on a 2012 book of the same name, to which I contributed an essay.
We can now identify impending bouts of depression, even suicidal tendencies, by looking at the changing lifestyle (social media usage, diminishing movement) of potential victims. And, using Google search data, epidemiologists can spot an emerging epidemic before doctors do.
One of the most extraordinary features of big data is that it signals the end of the reign of statistics. For 400 years, we’ve been forced to sample complex systems and extrapolate. Now, with big data, it is possible to measure everything, from the movement of billions of stars to every beat of the human heart.
Big data is also making its way, almost silently, into everyday life. In Israel, 80% of the citizenry contribute traffic data to help each other commute via the WAZE app on smartphones, increasingly the platform of choice for big data. California is looking at using the accelerometers in smartphones, which detect sudden motion, to create an early-warning system for earthquakes that will be much faster than government-installed devices.
Farmers in the Midwest will soon have measurements taken in their fields nearly down to individual plants—and that will combine data on soil, water and weather to come up with trillions of scenarios for insurance coverage. Victims of natural disasters are now being pinpointed and given aid through real-time analysis of tweets, instant messages and phone calls. Even one of the oldest of arts, geography, is being revolutionized with maps that will change by the second.
But it is in the commercial world where big data has seen its greatest inroads to date. A recent survey by consultants NewVantage Partners has found that the number of U.S. firms using big data in the past three years has jumped 58 percentage points to 63% penetration—while 70% of firms now say that big data is of critical importance to their firms, an astounding jump from 21% in 2012. That’s one of the fastest tech-adoption rates ever. Meanwhile, the title of chief data officer—the C-Suite manager of big data—a title that until recently didn’t even exist, is now found in 54% of companies surveyed.
The commercial impact of this revolution can be found everywhere from products and services that can predict the unique needs of individual customers, to improved credit precision, to stores that adapt (through special discounts and deals that pop up on your smartphone) to each customer who walks through the door.
To date, much of this activity has remained hidden from sight. But soon it will burst forth much more publicly—and the experiences of daily life will be profoundly transformed into a new set of personalized, predictive and empowered experiences we can barely imagine today.
Perhaps nowhere will this change be greater than with our own bodies. With new generations of external and internal monitoring devices continuously gathering unimaginable quantities of health and medical data on billions of people, we will soon learn more about the human body in a year than we once did in a century. China is already working to do this on hundreds of millions of its chronically ill citizens. This knowledge will help transform everything from epidemiology to genetic research to individualized medicine and treatment programs.
Most of all, big data is about to ask some cultural questions for which we have no ready answers, much less consensus. For example, who owns our personal data? Today that answer is not clear—and it may take legal precedent, perhaps even the adoption of a Bill of Data Rights, to resolve that question.
These are major challenges, but they will be overcome because the fruits of big data are too valuable. None, it is now clear, is more valuable than what is called “metadata”—data that derives from, and provides information about, other data. One of the biggest lessons of technology revolutions is that they begin by solving a need you already have—say, tracking how a particular virus spreads in a population—and end by solving needs you never imagined. And that may be the real destiny of big data.
When you can study billions, even trillions, of data points you begin to uncover forces and trends that until now have always been invisible to human observers. What if that impacted wisdom tooth you had at age 10 shortens your life 70 years later? Or if that one fugitive, missed heartbeat last week is a harbinger of cardiac trouble to come next month?
It is the discovery of this metadata that may prove to be big data’s real destiny: to teach us to see both ourselves and the natural world around us in ways we never could before.