How Not To Be Mislead By Numbers

The more we are encouraged to rely on data, if only by the cheap and easy availability of the data itself, the more we need to be skeptical.

Not because data is any more inherently evil or merely misleading than any other type of knowledge, but because the more of it that everyone you know, with whom you work - or with whom you compete - uses it, the more its relative value changes. Source, context, timing and change are the defining characteristics that make numbers relevant, meaningful, or not.

And it may well be that - as the following article suggests - what it doesn't tell you may be more important than what it ostensibly does. JL

Alex Peysakhovitch and Seth Stevens-Davidowitch report in the New York Times:

The world is incredibly complicated. No one data set, no matter how big, is going to

tell us exactly what we need. The new mountains of blunt data sets make human creativity, judgment, intuition and expertise more valuable, not less.

BIG data will save the world. How often have we heard that over the past couple of years? We’re pretty sure both of us have said something similar dozens of times in the past few months.

If you’re trying to build a self-driving car or detect whether a picture has a cat in it, big data is amazing. But here’s a secret: If you’re trying to make important decisions about your health, wealth or happiness, big data is not enough.

The problem is this: The things we can measure are never exactly what we care about. Just trying to get a single, easy-to-measure number higher and higher (or lower and lower) doesn’t actually help us make the right choice. For this reason, the key question isn’t “What did I measure?” but “What did I miss?”

To see the dangers of big data untethered to any other kind of analysis, consider the story of Zoë Chance, a marketing professor at Yale. In a TEDx talk that has been watched hundreds of thousands of times on YouTube, she discusses her experience with a pedometer. She became so obsessed with increasing the count of her steps that she lost all proportion, taking walks at all hours and in all places. She told us that she even put the pedometer on her daughter so that her daughter’s steps would contribute to her number. She was able to “detox,” as she put it to us, only after she suffered an injury while walking in the basement, exhausted, in the wee hours of the night.

Or consider the numerous teachers in Atlanta and Chicago who cheated when they were judged based on improving students’ test scores. They spent their time worrying about gaming the test, not what was happening in class.

So what can big data do to help us make big decisions? One of us, Alex, is a data scientist at Facebook. The other, Seth, is a former data scientist at Google. There is a special sauce necessary to making big data work: surveys and the judgment of humans — two seemingly old-fashioned approaches that we will call small data.

Facebook has tons of data on how people use its site. It’s easy to see whether a particular news feed story was liked, clicked, commented on or shared. But not one of these is a perfect proxy for more important questions: What was the experience like? Did the story connect you with your friends? Did it inform you about the world? Did it make you laugh?

To get to these measures, Facebook has to take an old-fashioned approach: asking. Every day, hundreds of individuals load their news feed and answer questions about the stories they see there. Big data (likes, clicks, comments) is supplemented by small data (“Do you want to see this post in your News Feed?”) and contextualized (“Why?”).

Big data in the form of behaviors and small data in the form of surveys complement each other and produce insights rather than simple metrics. For example, it’s fairly obvious that clicks aren’t always the same — sometimes people click through to an article because they really want to see the content, but sometimes people are tricked by seductive headlines. Knowing this is useful only once we can go beyond just measuring clicks to actually differentiating one kind of click from another. With this enriched measure of high quality clicks in mind, Facebook can do a much better job of delivering the content that actually leads to a better experience and not just empty clicks.

Continue reading the main story

Because of this need for small data, Facebook’s data teams look different than you would guess. Facebook employs social psychologists, anthropologists and sociologists precisely to find what simple measures miss.

And it’s not just Silicon Valley firms that employ the power of small data. Baseball is often used as the quintessential story of data geeks, crunching huge data sets, replacing fallible human experts, like scouts. This story was made famous in both the book and the movie “Moneyball.”

But the true story is not that simple. For one thing, many teams ended up going overboard on data. It was easy to measure offense and pitching, so some organizations ended up underestimating the importance of defense, which is harder to measure. In fact, in his book “The Signal and the Noise,” Nate Silver of fivethirtyeight.com estimates that the Oakland A’s were giving up 8 to 10 wins per year in the mid-1990s because of their lousy defense.

And data-driven teams found out the hard way that scouts were actually important. During the height of the data revolution, the Toronto Blue Jays cut their scouting staff, but the team suffered through a series of poor drafts. It turns out that scouts’ ratings of high school and college players are often the best data point on them, because they play relatively few games, and the games they do play are against wildly different competition. In an interview with Mr. Silver, Billy Beane, the A’s general manager and the main character of “Moneyball,” said that he actually had increased his scouting budget through the years.

Human experts can also help data analysts figure out what to look for. For decades, scouts have judged catchers based on their ability to frame pitches — to make the pitch appear more like a strike to a watching umpire. Thanks to improved data on pitch location, analysts have recently checked this hypothesis and confirmed that catchers differ significantly in this skill.

Education, despite all the debate about test scores, the Common Core and value-added methods, is actually moving in a similar direction as baseball and tech companies.

It’s gotten much less press than the test score debate, but there is also a huge national effort to collect and evaluate small data. Student surveys have proliferated fast. So have parent surveys and teacher observations, where other experienced educators watch a teacher during a lesson.

Thomas Kane, a professor of education at Harvard, told us, “School districts realize they shouldn’t be focusing solely on test scores.”

A three-year study by the Bill and Melinda Gates Foundation bears out the value of both big and small data. The authors analyzed value-added models, student surveys and teacher observations. They tested how to best predict student outcomes on both traditional state tests and more cognitively demanding challenges in math and English. When they put the three measures together into a composite score, they got the best results. “Each measure adds something of value,” the report concluded.

As at Facebook and in baseball front offices, small data can find holes in the big data. If a teacher raises her students’ test scores but students say she wastes a lot of time, and outside observers rank her poorly, this raises big questions. Conversely, if a teacher does not improve test scores but students say she inspires them and principals think she is imparting profound lessons, we may give her the benefit of the doubt. Most important, while big data can tell us whether certain teachers are helping their students, small data gives us the best hope to answer a crucial question: How are they doing it?

We are optimists about the potential of data to improve human lives. But the world is incredibly complicated. No one data set, no matter how big, is going to

tell us exactly what we need. The new mountains of blunt data sets make human creativity, judgment, intuition and expertise more valuable, not less.

At 65, Bill James is considered the father of the data revolution in baseball. His irreverent newsletters and books in the 1970s and 1980s, which supported a more numerical approach to the game, were highly critical, even mocking, of the old guard.

When we talked to Mr. James, who now works for the Boston Red Sox, he emphasized an important point: The numerical revolution he spearheaded was never about putting traditional experts out of business. It was about acknowledging our ignorance, then gathering and testing data, whatever its source.

“Scouts understand — as I understand — that they don’t understand the world,” he said. “We all understand that none of us has this figured out. And we’re all working on the same problem.”