US SEC Approves Vast Real Time Trading Data Surveillance System Covering 58 Billion Daily Orders, 100 Million Customer Accounts

This may arguably be the largest compendium of real time 'big data' in history. The goal is to gather enough information to be able - ideally - to prevent future market crashes.

The challenge is not just the cost and complexity of the initiative, but the reality that entrepreneurial efforts to game the system, as well as unintended consequences, may well render it ineffective, if not inoperable. JL


Nicole Bullock and Phillip Stafford report in the Financial Times:

The SEC sets in motion a timetable to bring to reality its Consolidated Audit Trail, a single data warehouse that collects the millions of orders and quotes that daily pass across US equity and options markets in real-time. (It) would be the world’s largest repository of securities transactions. It would have to be able to process more than 58bn records of orders and quotes a day, and maintain data on more than 100m customer accounts.
The main US markets regulator on Tuesday approved a plan to introduce a vast surveillance system to monitor trading on the $23tn US stock market, ending six years of delays.
The move by the Securities and Exchange Commission sets in motion a timetable to bring to reality its Consolidated Audit Trail, a single data warehouse that collects the millions of orders and quotes that daily pass across US equity and options markets in real-time.
“With the approval and ultimate implementation of CAT, the commission’s regulatory capacity strongly embraces 21st century technology,” said Mary Jo White, SEC chair. “Through the CAT, regulators will have more timely access to a comprehensive set of trading data, enabling us to more efficiently and effectively conduct research, reconstruct market events, monitor market behaviour and identify and investigate misconduct.”
The next step is to choose who will build the system. Three bidders, the Financial Industry Regulatory Authority, SunGard Data Systems and Thesys Technologies are in the running. The architects of the plan, a committee that includes exchanges, have two months to decide. The SEC estimates the system will cost $2.4bn initially and then $1.7bn a year to run.
US stock exchanges, Finra and the Options Clearing Corporation, which are known as self-regulatory organisations in the market, must begin reporting to the CAT within a year while large broker dealers have two years and small broker dealers have three years.
The idea was first proposed after the 2010 Flash Crash when stocks gyrated rapidly for 20 minutes. The crash revealed that authorities had little understanding of the fractured, complex nature of modern markets, where data and trading was executed in fractions of seconds across dozens of venues. The CAT will include data on the identities of the customers trading in the equity and options markets.
“The CAT would essentially be the Hubble Telescope for the securities markets,” said Kara Stein, SEC commissioner, in prepared remarks.
Still, the project has been beset by industry concerns over the cost, size and complexity of the project. The scheme is also likely to be one of the last pieces of business enacted by Ms White, who announced yesterday she would step down from the position in January.
Against that backdrop some market participants were still sceptical about the CAT.
“It is imperative if the SEC really wants to know who is doing what and when,” said Larry Tabb, co-founder of the Tabb Group, a capital markets consultancy. But he expressed concern that the CAT would not, for example, include information on the futures market where the Flash Crash is believed to have initiated.
“Will it also just be replaced with something else when Mary Jo White is replaced by a Republican,” he said.
Dave Lauer, chairman of Healthy Markets, an investor-focused non-profit group, raised concerns about the cost of the system, much of which will be borne by brokers who will in turn pass that on to investors.
“There is certainly plenty of controversy in how it is structured and who is paying for it,” he said.
Once completed, the CAT would be the world’s largest repository of securities transactions. It would have to be able to process more than 58bn records of orders and quotes a day, and maintain data on more than 100m customer accounts. Given the torrent of stock market data, the size of its database could grow to 21 “petabytes” — or 1,000 terabytes — of information within the first five years.