You can see the basic info for any stock by going to http://www.deepmarket.com/stream/goog and replace “goog” with the symbol you are interested in. Like AMZN, SUN, ORCL, CSCO…

Yeah, kind of boring right now, huh? Once we get a decent amount of data in the database, then we can start doing some interesting analysis. I have my own ideas, but am very open to suggestions for tools that people are looking to see implemented.

Some motivating links for future reference…

As contenders continue the race to the Melbourne Cup, other Melbourne Cup Betting favorites, like Weekend Hussler appear to be in line for a victory in the running of the Caulfield Cup in October. Lloyd Williams – the owner of Efficient – has even recognized Weekend Hussler as the favorite and is debating whether to run another of his horses – Zipping:

Only bad luck can beat him (Weekend Hussler) in that so I can’t see any purpose in running Zipping. He would need to have a head start in the race to beat Weekend Hussler.

Zipping is not showing well in the latest 2008 Melbourne Cup odds. However, Weekend Hussler will compete with Efficient, who tops the odds in October. After which the Melbourne Cup might be the next run for Efficient – it seems Williams has not determined whether his title holder will run again before the cup.

Septimus is another high profile horse with a well regarded trainer – Aidan O’Brien. O’Brien’s Bet on Melbourne Cup would be Septimus and the odds are reflecting his confidence with Septimus following closely behind Efficient. Behind Efficient and Septimus – where the odds begin to get tighter – are Zagreb, Tuesday Joy and Weekend Hussler.

Mind you – not pure mathematics, but more on the compuational side. So, when I found that I was getting traffic from Ernie’s 3D Pancakes
it interesting to find a blog that I could (errr… should more often) read and understand (sometimes). Should read is the key. The links to other web sites and blogs are a great launching point for finding other worthwhile resources.

Language is one of those areas that sci-fi has to cheat, like it does with faster than light travel. Here on earth there are thousands – if not tens of thousands – human languages and dialects. Translating between all these languages is hard – really hard. So, it seems that some sci-fi just ignore the problem – like in the Serenity/Firefly universe (although they seem to be multi-lingual without any technological assistance). Or they have to come up with a technology that explains the incredible ability of multiple species to converse in multiple languages and still understand each other. Star Trek has their Universal Translators and Farscape has their Translator Microbes. These technologies allow the users to understand and communicate in languages that they had never encountered before.

Yet, this is one area that is difficult to truly suspend disbelief. I am sure anyone who has learned a foreign language would also agree – not to mention the poor scientists that are working on natural language processing today. It is hard enough to extract meaning and understanding from one language and the systems that do it well are not learning systems – they are generally programmed from the top down. It is just the way the domain is at this moment. Not to say that good learning systems can’t be developed in the realm of language – just pointing out that the top-down systems still have the edge for the moment. I invite NLPers to correct me – this is a difficult domain!

So, when a story has someone waiting for their handheld (if not invisible) universal translator instantly learning a language never encountered up to that time – it seems hard to believe. So, let’s move beyond sci-fi and back in the practical. One reason this universal translation/learning/understanding process seems so extraordinary is that there is no feedback loop. Some alien life form is talking and the technology is able to derive meaning from it – in an unsupervised manner. Unsupervised basically means that the technology does not get graded and has no idea that the translations it is offering up are correct. Granted, I am oversimplifying the concept of unsupervised learning and these techniques have a place in language learning, but I doubt an effective language learning system could be developed without some type of feedback.

This leads me to my practical application – stock chart patterns. Dare I say that chart patterns are a type of language – the language of the markets? Not unlike trying to decipher ancient hieroglyphics of some lost civilization, trying to understand the language of the markets without some form of feedback is a sure way to waste a good deal of time. However, it seems that is what chartists do when they are looking at patterns in charts of stock prices. There is an alphabet of the market – Head and Shoulders – Wedges – Triangles – Rectangles – are all patterns in charts that perhaps are better described as hieroglyphics. These patterns are supposed to describe and help the analyst to understand the future direction of the market. This is the clash of science and art. Describing the chart patterns are generally pretty straight forward – however, actually recognizing “valid” chart patterns is an art.

Or, is recognizing chart patterns an art? This week we will look at some chart patterns and see if we can come up with some methods of finding patterns in stock data and then determining the resulting price movement after the pattern was recognized. We are trying to put the feedback loop into the process and chip away a little at the art of chart pattern recognition.

DeepGraphica is a powerful graphics solution for image editing, conversion, viewing, and a ton of other functionality. A little something to remind you of DeepMarket when you are doing your deep stock analysis. No stranger to developing high quality software, Mystik Media is also a producer of all-in-one software for audio & video editing, converter, DVD & CD burning, and ripping.

Anyway, DeepGraphica is free. Download DeepGraphica and start annotating those stock charts!

The trend to find areas of opportunity for optimization – such as Graphics Processors (GPUs) and now the PhysX Processor – most likely will not just benefit the game developer and consumer. Not having dug deep into the SDK documentation, I can only imagine that there are opportunities to leverage the processing power of a dedicated physics processor for calculations of heavy duty stock market problems. Of course, the same could be said about leveraging the awesome power of graphics processors. The wonderful thing about general purpose processors is that you can throw anything at them and they will do what you need. Specialized processors like the GPU and physics processor have a much narrower view of the world and are effective on certain types of problems. But when you find a problem or calculation that can benefit from these processors, they can have a major impact on speed. And who wouldn’t want to solve some of their most complex problems faster?

Also, further improvements with the searching for symbols and company names have been added. On the right side of every page there are search boxes that should help you find any of the stocks in our database. Generally, the shorter your search phrase (”apple” instead of “apple computer”), the better the results. The searching will improve – I just need to fix it. For now we only have NASDAQ stocks and will add more stocks in the future, but will concentrate on these for now.

So, it wasn’t my intentions to have to write about this classic struggle when I decided over a week ago to do the simple calculation of correlation of some stock prices. Well, about 3000 stocks. Overly excited that I got a database populated with almost 3000 stocks, I set about writing a simple Java program that simply calculates the correlation of stock prices. Even more simply, I chose to only look at the last 100 days for each of the stocks.

Simple.

For instance, let’s say we want to find the correlation between Microsoft and Amazon. I pull the latest 100 closing prices for each stock and plug them in and out pops the correlation -> a number between -1.0 and +1.0. This is a mildly interesting number and it wasn’t an exercise that I meant to put a great deal of effort into – therefore I setup the program to calculate the correlation between all of the stocks in my database. All 3000. For a total of 9,000,000 calculations. Okay, at least I was smart enough to only calculate the 4.5 million unique combinations – the correlation of MSFT vs. AMZN is the same as AMZN vs. MSFT. Okay, so only 4.5 million calculations, but this number did not expose itself until late in the day AFTER I kicked off the job.

Brute force calculations – sometimes you just gotta do it. Off the top of my head I can’t come up with any neat short cuts to get around brute forcing the calculation of all correlations of the stock prices. Yes, I am sure I could have come up with some cute tricks within the code to speed up the process, as it was hacked together. This is one of those precious examples that distributed computing or parallel processing – brute force – would cut through the problem like a hot knife through butter.

Anyway, 200 hours later we have the correlation of 100 days of stock prices between all the symbols in the database. I might update it again in the future – I might not. I plan on looking at the data and writing a more interesting article about what I see in the correlation numbers, but for now – take a look and see if you find anything interesting.

INTC – Intel

No search functionality has been setup yet. If you know the ticker symbol of your stock, just replace SYMBOL (see example below) with the symbol of the stock you are interested in seeing:

http://www.deepmarket.com/stock/SYMBOL

I have some other numbers up there right now (correlation numbers), but the data for that is still being loaded into the database. Those numbers will be changing through the day and I will write up an article when that is finished.

I can calculate the motions of the heavenly bodies, but not the madness of people.

It is interesting to note that present day technical analysis has been inspired by Newtonian physics, like the concept of momentum of a stock’s price. Other areas in physics are drawn upon to help develop techniques using harmonics, waves, and cycles for the analysis of markets.

Are these techniques effective in understanding the movement of prices, ultimately determined by the “madness of people”? Practitioners of various methods will certainly have their opinions about the strengths and weaknesses of these techniques, but often it comes down to the artistic application of human judgment. This, of course, is the creative nature of analysis.

Artistry and analysis seem to be opposing concepts. However, using creativity to develop new techniques is good. Being artistic in the analysis of results from a technique – well… That is really up to the user. I am more comfortable being analytical in the application of the methods and saving my creativity for creating new methods.

So, can new inspiration be drawn from the world of Physics?

Before I continue to go on and on and on about walking to school uphill and in the snow and without shoes….

The point I want to make is – the first time I played around with stock price data was in a spreadsheet program. I didn’t have CCI, MACD, RSI or any other conventional technical indicator canned and ready for use. I explored and played with the data in the unconstrained (at least to my technical ability at the time – once I learned BASIC better, I grew out of Lotus quickly) environment that a spreadsheet can provide. Not that access to technical tools “constrains” the user, but the ease and ready access to these indicators can slow the creativity process. In fact, I imagine most technical analysis tools these days give the user a great deal of flexibility in designing new indicators of unlimited complexity. How many people take advantage of that capability? I imagine very few people go that route. I should say, take advantage in a creative and innovative way.

So, that being said – I am looking forward to using DeepMarket as a really big whiteboard to brainstorm interesting ideas, creative analysis and innovative tools. To the point of ridiculousness. Some ideas may seem (or will be) silly. That’s okay – this is all about the imaginative and cool new ways to look at the data.