When a Simple Rule of Thumb Beats a Fancy Algorithm

A danger inherent in Big Data and our attempts to make sense of it is the propensity to assume that complicated information requires torturous analytical methods to unravel its mysteries.

We see this in the proliferation of statistical arabesques and baroque flights of rhetoric employed to justify the time and effort invested in teasing wisdom out of complexity. Which is not to say that there is no value in newer methodologies requiring experts with advanced degrees to effectively manipulate them.

But there is a reason, as the following article explains, why tried-and-true metrics have attained that status: because they are able to synthesize for the harried manager, what is essential to discern the direction, velocity and quality of a number at a given moment. In an attention-deficit-driven world, simplicity and speed may ultimately prove more valuable than a few extra degrees of accuracy. JL

Justin Fox reports in Harvard Business Review:

“In general, if you are in an uncertain world, make it simple. If you are in a world that’s highly predictable, make it complex.”
For a retailer, it’s extremely useful to know whether a customer will be back or has abandoned you for good. Starting in the late 1980s, academic researchers began to develop sophisticated predictive techniques to answer that question. The best-known is the Pareto/NBD (for negative binomial distribution) model, which takes a customer’s order history and sometimes other data points, then simulates whether and how much she will buy again.
Actual retailers, though, have tended to stick with simpler techniques, such as simply looking at how long it has been since a customer last bought anything, and picking a cutoff period (nine months, say) after which that customer is considered inactive.
This resistance to state-of-the-art statistical models has frustrated the academics. So, a decade ago, marketing professor Florian von Wangenheim (now at the ETH Zurich technical university in Switzerland) and his then-student Markus Wübben (now an executive at a tech incubator in Berlin) set out, in Wangenheim’s words, to “convince companies to use these models.”
To do this, Wübben and Wangenheim tested the predictive accuracy of Pareto/NBD and the related BG/NBD model against simpler methods like the “hiatus heuristic” — the academic term for looking at how long it’s been since a customer last bought anything — using data from an apparel retailer, a global airline, and the online CD retailer CDNow (from before it was acquired by Amazon in 2001). What they found surprised them. As they reported in a paper published in 2008, rule-of-thumb methods were generally as good or even slightly better at predicting individual customer behavior than sophisticated models.
This result wasn’t a fluke. “I’ve seen much more research in this area, many variables have been added to these models,” says Wangenheim. “The performance is slightly better, but it’s still not much.”
One way to look at this is that it’s just a matter of time. Sure, human beings, with “their limited computational abilities and their incomplete information,” as the great social scientist Herbert Simon put it, need to rely on the mental shortcuts and rules of thumb known as heuristics. But as the amount of data that retailers are able to collect grows and the predictive models keep improving, the models will inevitably become markedly better at predicting customer behavior than simple rules. Even Simon acknowledged that, as computers became more powerful and predictive models more sophisticated, heuristics might lose ground in business.
But there’s at least a possibility that, for some predictive tasks at least, less information will continue to be better than more. Gerd Gigerenzer, director at the Max Planck Institute for Human Development in Berlin, has been making the case for decades that heuristics often outperform statistical models. Lately he and others have been trying to define when exactly such outperformance is most likely to occur. This work is still ongoing, but in 2011 Gigerenzer and his colleague Wolfgang Gassmaier wrote that heuristics are likely to do well in an environment with moderate to high uncertainty and moderate to high redundancy (that is, the different data series available are correlated with each another).
Citing the Wübben/Wangenheim findings, Gigerenzer and Gassmaier (why so many of the people involved in this research are German is a question for another day), posited that there’s a lot of uncertainty over if and when a customer will buy again, while the time since last purchase tends to be closely correlated with every other available metric of past customer behavior. Ergo: heuristics win.
There are other areas where the heuristic advantage might be even greater. Financial markets are rife with uncertainty and correlation — and the correlations are strongest when the uncertainty is greatest (think of the parallel downward trajectories of lots of different asset classes during the financial crisis of 2008). Sure enough, while sophisticated financial models performed poorly during the recent financial crisis, simple market heuristics (buying stocks with low price-to-book-value ratios, for example) have withstood the test of time. Along those lines, Gigerenzer has been working with the Bank of England to come up with simpler rules for forecasting and regulating financial markets.
“In general, if you are in an uncertain world, make it simple,” Gigerenzer said when I interviewed him earlier this year. “If you are in a world that’s highly predictable, make it complex.” In other words, your fancy predictive analytics are probably going to work best on things that are already pretty predictable.