Is the Big Data Boom Already Over?

Like all tech projects, big data initiatives have to be integrated into the broader strategy of the enterprise.

Too many silos and too little evidence of significant or comprehensible returns may be cooling corporate ardor for more - until the impact is better understood. JL


Steve Ranger comments in ZDNet:

Too many big data projects have been poorly built, and lack return on investment - so companies are spending their money on other priorities. 48 per cent of companies invested in big data in 2016, up three percent from 2015. But the proportion who plan to invest in big data within the next two years fell from 31 to 25 per cent in 2016.
While investments in big data continue to rise, there could be trouble ahead as companies show signs of cutting back on projects.
Big data has been touted as a way for businesses to trawl through the vast amounts of information they own - from customer records to sensor data - to find trends that would otherwise remain hidden, but for many companies it has proved a difficult goal to achieve.
While investments in big data continue to rise, there could be trouble ahead as companies show signs of cutting back on projects.
Big data has been touted as a way for businesses to trawl through the vast amounts of information they own - from customer records to sensor data - to find trends that would otherwise remain hidden, but for many companies it has proved a difficult goal to achieve.
According to a survey of tech executives conducted by Gartner, 48 per cent of companies invested in big data in 2016, up three percent from 2015. But the proportion who plan to invest in big data within the next two years fell from 31 to 25 per cent in 2016. And while nearly three quarters of the 199 respondents said their organisation has invested or is planning to invest in big data, the majority of these projects are still stuck at the pilot stage: only 15 per cent of businesses reported their big data project was now in production - only a tiny shift from last year's 14 per cent.
One problem for big data projects has been a lack of proper management, along with thrown-together systems: "Too often, pilots and experiments are built with ad-hoc technologies and infrastructure that are not created with production-level reliability in mind," said Gartner.
And for many companies, big data projects are now less of a spending priority than competing IT initiatives, said the analysts. Only 11 per cent of respondents from organisations that have invested in big data described their big data investments as being as important, or more important, than other IT initiatives - and 46 per cent stated that they were less important.
Nick Heudecker, research director at Gartner, said this could be because many big data projects don't have a tangible return on investment that can be determined up front.
That doesn't mean the idea of analyzing large sets of data to find useful insights is going to stop, but that the approach is being used more broadly, he said, adding that another reason for the drop in funding could be that distinct big data initiatives are being folded into larger programs. "This will become more common as the term 'big data' fades away, and dealing with larger datasets and multiple data types continues to be the norm," he said.

"While organisations have understood that big data is not just about a specific technology, they need to avoid thinking about big data as a separate effort," Heudecker added.