Big Data and the Big Idea: Integrating Intuition and Information

Nature, nurture. Make, buy. Online, offline. There was a time, in the long ago, when it seemed that everything important involved a choice. And those choices defined what we did, how we progressed, why we succeeded (or not), who we were.

But the world has become a lot more complicated, especially for anyone involved in the daily struggle to make a living from interaction with other members of the species.

We have 'frenemies,' our competitors who are also our suppliers and customers. Alliances work, then fade, then reignite. Apple disses IBM for decades - until they become partners. Alibaba, Amazon, eBay: everyone is doing business with everyone else.

In order to make sense of this maelstrom of inputs and outcomes we have come to rely on data, but often employed in order to stimulate emotions. Read a random sample of personal statements on Twitter and count the number of people who use words like 'passionate' and 'evangelist' - right next to their acronymical descriptions of their deeply technical expertise.

We still have to make choices, but they may no longer be sustainable, let alone defining. The factors driving the socio-economic environment are simply too dynamic, unstable and robust to be confined in that way. In fact, the vocabulary of biology is, to a great degree, supplanting that of engineering. This is fitting, given the rise of intangibles relative to tangibles as the means by which our work and lives are ordered - or disordered as the case may be. But the implications of co-evolution and sustainability and ecosystems and recombination in the economic context sometimes run counter to our deeply ingrained notions of win/lose, up/down.

The fact is that we need to integrate rather than isolate if we are going to manage the massive quantities of data hurtling our way - while employing our instincts to help make sense of them. JL

Greg Satell comments in Digital Tonto:

Most marketers would prefer to be like Leo Burnett and Steve Jobs.They take pride in their “marketing guts," which makes it hard for them to succeed in the era of big data. This is a false choice.  The future of marketing is not technology or intuition, but successfully integrating both.
Great marketers have great guts.  Leo Burnett didn’t need a legion of focus groups to come up with the Marlboro Man.  Steve Jobs, arguably the greatest marketing mind ever, famously eschewed market research because he didn’t think customers knew what they wanted until he showed it to them.
Yet big data and technology are clearly revolutionizing marketing.  Gartner predicts that CMO’s will soon be spending more on IT than CIO’s.  VentureBeat recently reported that marketing technology companies have attracted a hefty $50 billion in investment.
Still, most marketers would prefer to be like Leo Burnett and Steve Jobs.  They take pride in their “marketing guts” and want to follow their instincts, which makes it hard for them to succeed in the era of big data.  Yet too often this is presented as a false choice.  The future of marketing is not technology or intuition, but successfully integrating both.
The Problem With Guts
In Blink, author Malcolm Gladwell argued that snap judgments are often superior to studied, rational responses and there is a wealth of serious research that supports that view.  Antonio Damasio, a prominent neuroscientist, has shown that our biological response to information can often outpace our neurological recognition.  In other words, gut feelings are very real.
However, there is also evidence that shows that intuition can lead us astray. Philip Tetlock, who conducted a 20 year study of political pundits, found that their judgments were no more accurate than flipping a coin.  To see why that happens, take a look at what longtime observer Peggy Noonan had to say the day before the 2012 Presidential election.
 

All the vibrations are right. A person who is helping him who is not a longtime Romneyite told me, yesterday: “I joined because I was anti Obama—I’m a patriot, I’ll join up.  But now I am pro-Romney.” Why? “I’ve spent time with him and I care about him and admire him. He’s a genuinely good man.” Looking at the crowds on TV, hearing them chant “Three more days” and “Two more days”—it feels like a lot of Republicans have gone from anti-Obama to pro-Romney…
…Is it possible this whole thing is playing out before our eyes and we’re not really noticing because we’re too busy looking at data on paper instead of what’s in front of us? Maybe that’s the real distortion of the polls this year: They left us discounting the world around us.

Clearly, the “vibrations” that Noonan experienced had negligible predictive power, yet she felt them strongly nonetheless and, because she felt them, she had confidence in her judgment.  They seemed right, so she acted on them and ended up looking like a fool.
That’s the problem with gut feelings.  They are not only often wrong, but we also tend to have more confidence in them than we should.  They feel much more real than data and algorithms, so we are more comfortable acting on them.  When we go with our gut, we are putting our faith in the power of our own personal experiences and that feels good.
So we should not only distrust gut feelings, but also be skeptical of the confidence and passion they produce.  Intuition can be valuable when we experience events directly and have the opportunity to receive immediate and continuous feedback, but is fairly useless when our experience is indirect or infrequent.  That’s when data is not only useful, but crucial.
The Rise Of The Robots
Everybody in business today is struggling with data, but the challenges for marketers are especially daunting because there has been such an enormous shift in the amount and quality of data we work with.  Historically, we only had backward looking surveys to guide us, but now we collect data in real time and can process it with amazing efficiency and precision.
Perhaps even more importantly, the technology we have now is not only super fast, but super smart.  Computers today are not merely big calculators, but can recognize and learn patterns.  Often, they outperform humans in things like legal discovery, making medical diagnoses and even evaluating creative work such as music and screenplays.
Yet as powerful as they have become, computers are not all powerful, they perform much better when guided by humans.  For example, in a freestyle chess tournament combining both humans and machines, the winner was not a chess master or a supercomputer, but two amateurs running three simple programs in parallel.
And that’s gives us a clue to where marketing is going.

Racing With The Machines
As Brynjolfsson and McAfee put it in their book, The Second Machine Age, we need to stop trying to race against the machines and start racing with them. Yes, today’s computers are highly intelligent, able to analyze and learn from mountains of data in the blink of an eye, but they still need humans to design the curriculum that educates them.
BloomReach is a company that is at the forefront of the revolution in machine learning.  Its algorithms power some of the world’s most popular retailers, such as Neiman Marcus, Staples and Williams Sonoma, allowing them to create personalized customer experiences through the power of big data.
Recently, it announced the launch of BloomReach Compass a new application that puts humans at the center, instead of just letting algorithms run in the background.  As Ashutosh Garg, the company’s Co-founder and CTO told me,  “it’s designed for marketers rather than big data experts or a dedicated business intelligence team.”
The idea is to allow marketers to do what they do best—imagine new possibilities.  “Compass encourages professionals to follow their intuition and test unconventional ideas.  A crazy idea isn’t so crazy when you can calculate its effect without risking failure in the market,” says Joelle Kaufman, who heads up marketing for the company.
The Future Of Marketing is All Too Human
Leo Burnett was a magician.  The Marlboro Man, The Pillsbury Dough Boy and the Keebler Elves—all characters he created—revolutionized his clients’ brands.  Steve Jobs conjured up devices in his head that changed the world.  Neither of them needed machines to tell them what to do.
Yet both of them had their failures as well.  Many of Burnett’s campaigns bombed, just as many of Jobs’s products flopped.  Clearly, they would have been far better off if they could have used massive amounts of real time data to test their ideas before they went to market.
That’s what’s so exciting about the coming era of data driven marketing.  When done right, it puts humans at the center.  As Scott Brinker, CTO of Ion Interactive puts it, “big data makes it cheaper and easier to test concepts, but marketing is still about coming up with the big idea.  Algorithms are great at optimization, but terrible at imagination.”
And I think that gets to the heart of it.  Marketing technology does not mitigate the need for marketing guts, but it can help us tell the difference between an inspired idea and a bad lunch.