Model Manipulation: False Positives, Statistical Flukes and Twisted Strategies

What are we trying to prove? We have embraced data as the objective arbiter of fate, only to discover that numbers, let alone their meaning, can and will be manipulated when it suits whoever stands to benefit from the outcome.

This is hardly shocking since gaming systems is as old as mankind's agreement to be judged by other's standards, but given the investment - and belief - we are placing in the primacy of numbers, the realization that this data does not, in itself, constitute some sort of revealed truth could be just unsettling enough to end the party before it has barely gotten started.

There is some reason to hope that the market for good judgment will be self-correcting. Our increasing reliance on data, spurred in part, by the ability of internet-denominated performance is designed to be tracked, may be enhancing societal numeracy. 

The efficacy of these results and the systems and processes that produce them will not happen organically. Eternal vigilance is the price of sustaining many human values, including credibility. JL

Stephen Foley reports in the Financial Times:

Too many managers and advisers are purveyors of false positives, getting rich on statistical flukes.
An academic journal called the Notices of the American Mathematical Society may seem an unlikely periodical to have exposed fraud on a massive scale. The investigation, published in the current edition, is certainly not going to sit among the nominees for next year’s Pulitzer prizes. But a quartet of mathematicians have just published a piercing article in the public interest and in the nick of time.

In their paper, entitled Pseudo-Mathematics and Financial Charlatanism, they make the case that the vast majority of claims being made for quantitative investment strategies are false.*

By calling it fraud, the academics command attention, and investors would be wise to beware. With interest rates about to turn, and a stock market bull run ageing fast, there have never been such temptations to eschew traditional bond and equity investing and to follow the siren sales patter of those who claim to see patterns in the historical data.The (unnamed) targets of the mathematicians’ ire range from individual technical analysts who identify buy and sell signals in a stock chart, all the way up to managed futures funds holding billions of dollars of clients assets.
There will be many offenders, too, among investment managers pushing “smart beta” strategies, which aim to construct a portfolio based on signals from history.
There is even a worrying do-it-yourself trend: many electronic trading platforms now have tools encouraging retail investors to back test their own half-baked trading ideas, to see how they would have performed in the past.

Twisting strategy to fit data

The authors’ argument is that, by failing to apply mathematical rigour to their methods, many purveyors of quantitative investment strategies are, deliberately or negligently, misleading clients.

It is reasonable to want to test a promising investment strategy to see how it would have performed in the past. The trap comes when one keeps tweaking the strategy until it neatly fits the historical data. Intuitively, one might think one has finally hit upon the most successful investment strategy; in fact, one is likely to have hit only upon a statistical fluke, a false positive.

This is the problem of “over-fitting”, and even checks against it – such as testing in a second, discrete historical data set – will continue to throw up many false positives, the mathematicians argue.
Do not despair. The paper does not conclude that history is bunk, just that backtesting ought to require more statistical thought than investment managers need to display to make a sale to investors.
The perennial success of Renaissance Technologies, founded by code-breaking maths genius Jim Simons, suggests that some can separate signal from noise in financial markets.
At least the best quantitative hedge funds are attuned to the problem of overfitting. London’s Winton Capital published a paper last year warning that, even if individual researchers are scrupulous about calculating their probabilities, institutions risk “meta-overfitting”, because the tendency is to only submit the best fitting strategies for approval to the higher-up management committee.
It seems that finance may need the same overhaul as the pharmaceuticals industry did a decade ago.

Statistical flukes

Amid a furore over the safety of its antidepressant Paxil in 2004, it was discovered that GlaxoSmithKline had conducted numerous trials that failed to prove the drug was an effective treatment for children. However, a minority of trials did suggest efficacy, to a statistically significant confidence level, and these were the studies that got published. It wasn’t until scientists added together all the unpublished data that it became clear the drug increased the risk of teen suicides, for no offsetting benefit in treating depression, and it was banned for use by minors.

GSK responded by promising to reveal all its trials and to publish all its data, regardless of their outcome, and other large drug companies followed, more or less reluctantly. As a result, we continue to learn that large claims made for blockbuster medicines tend not to stack up over time, Tamiflu being the latest example.
When it comes to quantitative investment strategies claiming to have performed well historically, it is not good enough for managers to stamp “past performance is no guide to future performance” on to a marketing document. A crucial detail, almost never revealed, is how many discarded tweaks and tests led to the miraculous discovery of the strategy.
The authors of the Notices of the AMS paper are upbeat about the chances of banishing pseudo-mathematics from finance.
One of their number, Marco Lopez de Prado of Lawrence Berkeley National Laboratory, distributes open source software, at quantresearch.info, which can improve the modelling of mathematical probabilities and limit the risks overfitting. Another, David Bailey of the University of California, Davis, suggests that a regulatory body such as Finra could step in to promote best practice in the marketing of mathematical claims, just as the Food and Drug Administration monitors drug advertising. Together they have created a blog at financial-math.org to debate their ideas.
Raising the issue is necessary for raising the bar. Too many investment managers and advisers, it is claimed, are purveyors of false positives, getting rich on statistical flukes. If their methodologies do not improve in line with the improvements in academic thinking about backtesting and overfitting, then they really will deserve to be called out as frauds.