August 31, 2015    6 minute read

Big Data: The Holy Grail of the Stock Market

   August 31, 2015    6 minute read

Big Data: The Holy Grail of the Stock Market

Humans have recorded data for thousands of years, but only recently, we have amassed an unimaginable amount of information. We have always sought to collect and pass on knowledge as a community, whether this medium is hieroglyphics, spoken word or a written narrative. In more recent times however, this information has been stored as computer data. 90% of the current data in the world was created in the last two years, and we only need to look at our mobile phones to see our advances in data storage.

We have started recording data like never before. Our smartphones track our every location, our credit cards track our every purchase and our computers are recording our every keystroke. Now this information is no longer static, it is being used for some impressive purposes. There are now wristwatches that measure our entire biochemistry, there are self-driving cars and our computers can now provide personalise shopping suggestions, based on search history.

This rapid advancement in technology is demonstrated by machine learning, a branch of artificial intelligence, which is a computer’s ability to learn without being explicitly programmed to do so. It works on the principal that from more experience, a computer can improve it’s own performance. One example of machine learning is the finger print sensors on our phones. The first time we provide our fingerprint, the program begins to build up a mathematical image of the finger. The sensors take images from the sub epidermal layers of our skin, mapping out our unique ridges and contours. The more images are taken, the better the mathematical image. Eventually the fingerprint sensor is able to recognise the print to an incredibly precise detail. This example of machine learning is consistent with the definition above. The more data we give the machine, the computer is able to learn, and improve its detection. This is a very simplistic example, but machine learning is being used for much more complex, and beneficial ways for society.

Machine learning is at the forefront of transforming healthcare forever. Using big data we have been able to statistically analyse the relationships between our drugs and our genomes. Our progression in this area called pharmacogenomics, ties in directly with the future of precision medicine. By continuing to use big data in this way, scientists believe that we shall be able to personalise medicine to an individual. One example is the sequencing both the cancer patients genome and their tumour cells. This method instead of conventionally assessing the tumour by its tissue alternatively focuses on the biological makeup of the cells. The resulting medical treatment would therefore be precisely directed at the tumours biology, rather than a standardised treatment for all. These medical advances would not be possible without the use of big data and machine learning.

A developing application of big data is the technical analysis of stock markets. In order to use big data to predict stock market movements, I began with the study of Google. With over 40 billion searches per day, I believed that underlying this data was an aid to predicting the stock market, and that tracking the movements in search volumes of certain words, would correlate with movements in stock markets.

I initially began by tracking the movements in words and phrases used to describe macroeconomic performance in the media: debt, interest rates and recession. I discovered that the search volumes of these chosen words and phrases responded to events in real time. One major example was the falling prices of oil. During this period, there was hysteria within the markets and volatility was brewing. I saw search volumes for these words rise, and I could see a direct correlation with the stock market. As the search volumes increased, so did the volatility and the price of the S&P 500 fell. At the time of these movements in the markets, there was serious speculation amongst the media and academic economists, that the US economy could be adversely affected by falling oil prices, and the search volumes reflected this.

This established a general rule to help with share purchasing decisions: increased frequency of search indicated volatility would rise and prices would fall. Therefore, I was able to capitalise on the purchase of volatility futures contracts and a prediction of price falls in American stocks. The corollary of this was that when search volumes began to return towards their equilibrium level, I would exit my trades, as this would indicate that volatility would fall and prices would rise.

To generalise this particular application of the idea that search frequencies correlate with stock market performance, I developed an algorithm that assessed a large data set of words, the patterns of their correlating search volumes, and any correlation to the US stock market. After 6 months of testing, I had created my first trading strategy, which produced trading signals based only upon Internet search data.

Although my system was using big data to correctly identify potential trades, it lacked the autonomy of machine learning. After contacting several specialists in this area I was able to find someone willing to help me. I am now working on a way to get the machine to choose words autonomously, based on the data it already has of correlating strategies. The aim is that the machine will be able to test thousands of combinations of words per minute, across a number of different global indices. I have also incorporated conventional technical analysis alongside the Google data, and I believe that the outcome will be the ‘Holy Grail’ of the stock market. With each test covering over 500,000 lines of data, the working strategies have a certainty, unmatched by any other.

I believe a creation of the ‘Holy Grail’ will change financial trading forever. Anyone who would possess such a system would be able to exploit and arbitrage the stock market on a mass scale. With this form of big data testing, I believe we can also see the first sustainable risk free strategies. However if my predictions come true, it will be a very risky time for global economies. If all banks are using the same strategies to trade, then the markets will inherently be incredibly volatile. If you account for all investment banks placing the same trades, there is a potential for immense market failure. One potential failure is the short sell of a major currency. If all banks act together in predicting the depreciation in the value of the euro, for example, then simply by the nature of the trades, the currency will fall. When these trades are valued in the billions, there is a likely reality that this might even force an economy into meltdown.

Big data is changing the future at a rapid rate, and I believe it hold the keys to many unopened doors. Whether that is in health care, stock markets or technology than can enhance our lives. But we must use our data wisely. There will be obstacles, which will need overcoming, and privacy is essential. But in an age where our data is being collected so vastly, is it already too late? Nevertheless, this is an exciting time, and the benefits of big data will soon be prominent in all of society.

Get articles like this straight to your inbox each morning with our Breakfast Briefing. Sign up by clicking here!

Log in with your details


Forgot your details?

Create Account

Send this to a friend