The age old competition between brawn and brains continues in the computer age. Seemingly “intelligent” demonstrations of machines’ abilities to compete with humans, generally with games like chess, are really dependent on computer “brawn”. True intelligence in machines – artificial intelligence – will more than likely be the result of leveraging really clever algorithms that side-step the need for brute force calculations.

So, it wasn’t my intentions to have to write about this classic struggle when I decided over a week ago to do the simple calculation of correlation of some stock prices. Well, about 3000 stocks. Overly excited that I got a database populated with almost 3000 stocks, I set about writing a simple Java program that simply calculates the correlation of stock prices. Even more simply, I chose to only look at the last 100 days for each of the stocks.

Simple.

For instance, let’s say we want to find the correlation between Microsoft and Amazon. I pull the latest 100 closing prices for each stock and plug them in and out pops the correlation -> a number between -1.0 and +1.0. This is a mildly interesting number and it wasn’t an exercise that I meant to put a great deal of effort into – therefore I setup the program to calculate the correlation between all of the stocks in my database. All 3000. For a total of 9,000,000 calculations. Okay, at least I was smart enough to only calculate the 4.5 million unique combinations – the correlation of MSFT vs. AMZN is the same as AMZN vs. MSFT. Okay, so only 4.5 million calculations, but this number did not expose itself until late in the day AFTER I kicked off the job.

Brute force calculations – sometimes you just gotta do it. Off the top of my head I can’t come up with any neat short cuts to get around brute forcing the calculation of all correlations of the stock prices. Yes, I am sure I could have come up with some cute tricks within the code to speed up the process, as it was hacked together. This is one of those precious examples that distributed computing or parallel processing – brute force – would cut through the problem like a hot knife through butter.

Anyway, 200 hours later we have the correlation of 100 days of stock prices between all the symbols in the database. I might update it again in the future – I might not. I plan on looking at the data and writing a more interesting article about what I see in the correlation numbers, but for now – take a look and see if you find anything interesting.