Stock Market Analysis at DeepMarket

Remember, this blog is like a whiteboard - we are kind of brainstorming here…

I am beginning to play around with quantifying the predictability of patterns, such as the head and shoulders pattern. There is a lot of analysis, running of data and thinking that needs to be done, but I wanted to talk a little about what I am seeing so far. To really understand the usability of certain patterns, you really need to slice and dice the data in various ways to understand what tendency the market/stock/index/whatever might have after a pattern presents itself. As an experiment, I pulled closing data from about a thousand stocks that I had over 10 years worth of data. Presently, I am just trying to get a handle on the H&S pattern across ALL stocks, so I figured I would ignore individual stocks and just look at some global numbers.

I decided to be very simplistic (and practical) and look at the seven day H&S pattern. I am making a HUGE assumption in this next step - I am assuming the general tendency of the pattern will play out within seven days (again, seven to match the length of the pattern). In the future, I plan on looking at days one through pattern length X 2 - or something of that nature - to give use a better understanding of how a pattern evolves once it is recognizable. So, here are some early numbers:

- The average stock showed a 7 period head and shoulders pattern every 349 trading days
- That equates to about .4% of all stocks having a head and shoulders of length 7 at any one time

Okay, those make sense. It seems that patterns like this are relatively rare.

- It seems that only about 400 stocks out of 1000 had a tendency to move DOWN seven days after an H&S pattern

Ooops…

The “head-and-shoulders” pattern is believed to be one of the most reliable trend-reversal patterns.

Okay, let’s dig just a little here:

- Of the 400 odd stocks that showed a tendency to go down after an H&S pattern, the average seven day loss was 1.7%
- Of the 600 remaining stocks that showed a tendency to go UP after an H&S pattern, the average seven day gain was 2.45%

Okay, that is just a little taste. I have plenty of work ahead of me to really see what is going on. Also, if anyone sees any problems in my logic or has questions about how I am calculating things - let me know. I am skipping a lot for ease on my part and trying to get to the deeper issues, but details can be important and that is where the mistakes can be made!

Previously, I discussed the importance of feedback in understanding and learning and that is one nice aspect of blogs - getting feedback from readers. Michael wrote a nice entry about the last article over at Taylor Tree, a blog that discusses “the mechanics of system trading such as money management, position sizing, etc.” - good stuff! He describes an interesting pattern he calls Melba Toast, but I don’t think that the simple pattern matching I described would be the best way to identify this pattern. I need to look at more Melba Toast examples.

I got some really good questions from Thomas Czaszyski, who writes a stock investment and research blog. Thomas asked the perfect question:

This head and shoulders, I understand how to get it and everything now, but what does it mean?

Exactly! Now, I know the classic chartist answer, but I am not interested in regurgitating the same answer a chartist would give. Luckily, Scott pointed us to the book Encyclopedia of Chart Patterns, so I would recommend Thomas check that reference. Or do a quick look at head and shoulders described by chartists. Thanks for the pointer to the books Scott - I certainly want to take a look at how everything was quantified. My major concern is - how did they identify the patterns? Were humans finding the patterns?

Also, to Thomas - good luck with Princeton! Don’t be surprised if you find that most academics don’t truly understand the stock market and trading, let alone technical analysis.

It has been pointed out that in the article Oversimplified Method for Finding Patterns in Stock Charts the actual method used was not described in detail. The process is pretty simple - you might even say oversimplified. It was just an error of omission, not an attempt to hide any proprietary methods. In fact, just as a gentle reminder - this blog is for the open discussion and exploration of advanced techniques for analyzing the stock market.

So, let’s review. I am not a chartist or a technical analyst, but am interested in understanding patterns that seem to have “legendary” status among those who dabble in these arts. One of my greatest concerns is that there is little work on validating chart patterns in an objective manner. So, for instance - how reliable, profitable and practical are these patterns?

Well, before we can approach those questions, we needed a way to detect patterns in a repeatable fashion. Pattern matching is a fascinating field, but we just needed a quit and dirty solution. Correlation was that simple solution. What was not described was how we are using correlation to find patterns in stock charts. We are going to use some data that never changes (a stock in the database that has stopped trading) and Excel to show this simple process.

AMNT stopped trading on 12/12/2005 and just so happens to have a really classic head and shoulders pattern in the last seven days of trading:

Using Excel, we can describe the process that is used to find the head and shoulders pattern (or any other pattern - given a template of the pattern). First, we need a template of the pattern we are looking for in the chart data. I explored the charting and technical analysis literature and “eye-balled” the patterns people were describing as head and shoulders. This is far from scientific, but a good start. What I came up with as a template for head and shoulders is:

Or, very simply using numbers from 0 to 100:

0
60
20
80
30
70
20

Seven data points gives us a template for what I think a head and shoulders pattern would look like. This template is certainly open to discussion, but this is more about the process than the correctness of the template. What is important is that we can use the above template to easily and quickly identify that general pattern. We have a template of 7 data point, so we need to look at 7 data points from the stock data we wish to search for patterns. We are using AMNT and the last 7 data points (looking at closing prices) are:

10.22
10.25
10.24
10.28
10.24
10.26
10.24

Now, open up Excel and paste the two columns of data into columns A and B of your worksheet. It doesn’t matter which column you put the data - template in A or template in B:

Now, using the Excel function CORREL we are going to find the correlation between the template and the stock data. Put the following formula into any cell that isn’t already being used:

=CORREL(A1:A7,B1:B7)

The result is a correlation of .940712, which is a pretty good match. You look at the pattern’s template and the actual stock data and see that it is a pretty good match. Pretty simple, huh?

For what we are doing, we have this process written in Java and every morning we search for patterns that have a correlation greater than .85 - why .85? No reason, just an arbitrary point. For those who wish to do this for themselves in other languages or technical analysis packages, you just need a correlation function and the ability to use that function to compare two sets of data. Once you have that setup, then you can play around with any number of templates describing different patterns and correlation cut points.

Look, I don’t know anything about basketball (something about dribbling or something or other?) and the only March Madness that I am aware of is the snow we keep getting, even though it is spring already. I graduated from GMU back in 1993, so that makes good ol’ GMU my alma mater. So, when I heard on the news that GMU’s basketball team advanced to the Final Four (which, again, I am guessing is “a good thing”), I was pretty proud.

Perhaps this will help shine some light on the excellent Information Technology and Engineering school, which has recently been named the Volgenau School of Information Technology and Engineering. Hey - I used to work for that guy! Dr. Volgenau is a fantastic individual and I had the pleasure of working with him on a couple interesting projects involving - you guessed it - GMU.

george mason

I have been using StumbleUpon since late 2003. It is a wonderful tool that I can’t imagine being without these days. I have always gotten a couple of people visiting my sites from StumbleUpon everyday, but since last night I am getting a major stream of visitors from StumbleUpon. If there are any visitors from StumbleUpon who are visiting now, please leave a comment on how you got here - I would really appreciate knowing! I love the traffic and would like to know how I can keep it coming. Finally, my little web server is earning her salary…

Moments later…

Ah ha - could all this traffic be the result of DeepMarket making the list of Featured Stocks Sites at StumbleUpon? That is awesome!

I know - I am several years late to the Tag Cloud concept, but I figured I would try something out similar. For now (it may move it the future) we have a stock symbol cloud over on the right hand column of the blog. In standard cloud fashion, the font size gives you an idea of the popularity of that particular item. This is real simple - just looking at the number of people who visit that stock’s page in the past 24 hours. So, without even looking at the cloud, I can guess that stocks like Google and Yahoo will be on the large side. It may be interesting to see the symbol cloud evolve over the course of a very busy news day. Gee, this really makes me think about needing a new template for the blog and the rest of the site.

There is an interesting thread over at Elite Trader forum discussing the possibility of automatically trading the news feed. Some people claim that they do have such systems in place or know of such systems. Well, that is all fine and good, but it is a difficult problem. Language understanding is one aspect, not to mention what a system is to do once it does understand the news stream. I have some ideas in the back of my mind, but it is low on my priority list right now. Luckily enough, a forum poster directed us to a site that is experimenting with several algorithms and is posting the results daily - as well as the historical results.

The site - StockPrompts - looks very interesting. They are scanning various news sources to create daily predictions of what a stock will do during the next day - from open to close. A fun experiment! The site actually says “StockPrompts continuously scans news sources…”, but I have a feeling that it is actually a batch process. Recently, they have started tracking how the trades would actually work out with a $5000 paper trade, with curious results. For instance, paper trading for Apple Computer would give a profit of $2300 over the past 35 trades (again, from open price to close price).

Right now they are looking at a very limited universe of stock - mostly tech stocks. Again - this is experimental, so enjoy it while it is public, as it may dissapear at a moments notice. The real fun part is digging into how the algorithms work - but that is left to the reader’s imagination (much like the way I prefer it) ->

That’s a secret! But I’ll admit it involves scanning news articles and doing odd things with the Apache Commons Math Library.

An interesting experiment to keep your eye on.

Last week we put together a really simple way to identify Head and Shoulders patterns in stock charts and we have a daily list of stocks with potential Head and Shoulders - BUT, these are experimental of course. This where the human interacts with the technology. We have an algorithm that searchs for potential patterns, then it is left to the human to determine if it really is a Head and Shoulders formation. And what does it mean if it is a correct formation in the eyes of the human? That is left to the human to decide.

And in that spirit we put up a new page which lists stocks with potential Inverted Head and Shoulders chart patterns. To be honest, when I look at these patterns they do not seem to be as disctinctive as the regular H&S formations. Any chartists want to give us some feedback?

If you are looking for discussions or need answers to questions about anything and everything trading - join the Elite Trader forum and plug into almost 50,000 members. Zoinks - that is a lot of money being lost! Just kidding, as I am sure many of the members are doing really well, but one long thread basically enumerated every possible bad thing to do when trading. The poor fellow in the thread boasted that he was going to double his money in a month trading the FOREX. I am not going to ruin the ending of the story for you - read all 62 pages in the thread (no kidding!).

I like browsing the Trading Software, Harware, Automated Trading, and Technical Analysis sections. It seems like a good community that you can ask questions and generally get good feedback. Vendors - beware! Don’t try to spam the forums - people hate that, but I am sure you already know that…

This is an old story, but there is the semi-urban legend that if you mention Chris Pirillo in a blog posting, he will come and post a comment on your blog. This was known as the Chris Pirillo Effect, but the effectiveness of the CPE seems to have degraded over the last year. What the heck - it is worth a try.

However, Joe Anderson seems to want to replace Chris with his own Joe Anderson Effect. Joe writes the Webby’s World blog which is “about technology. It is generally focused on cool websites, hacks, piracy (and ethics) and open-source. Also, there’s quite a bit about the Mozilla programs and Mozilla spirit.”

EDIT
Well, someone claiming to be Chris Pirillo stopped by and posted a comment. Since I am feeling like “the cup is half full and not half empty” today, I am going to assume it was Chris - Thanks Chris! Sorry for doubting the power of the CPE…

Golly, how I miss TechTV!

I must admit up front that I am horrible with math. But I love it - an unfortunate paradox. I am constantly tracking down and trying to learn the latest and greatest algorithms in the data mining and machine learning fields. Unfortunately, blundering through acedemic papers is often what I end up doing - and unless there is some example software with source code - I am lost. So, having said that - I love reading about math.

Mind you - not pure mathematics, but more on the compuational side. So, when I found that I was getting traffic from Ernie’s 3D Pancakes
it interesting to find a blog that I could (errr… should more often) read and understand (sometimes). Should read is the key. The links to other web sites and blogs are a great launching point for finding other worthwhile resources.

The first blog entry is titled Making a difference, which I guess is exactly what Google would expect of “lots of world class Ph.D.s (and a few non-Ph.D.s)“. It is interesting that they made a distinction between engineers (”who do the real work”) and Ph.D.s. Golly, the blog was launched almost a month ago and I had not seen anything about it anywhere else.

It is good to hear that Google may be stepping into Bell Labs’ old shoes and working on problems that have very long time frames and not just short-sighted quaterly results. They even mention the year 2020 for some goals. So, be certain to check out the Official Google Research Blog every now and again - it seems to be update infrequently.

Before we begin to determine the predictive capabilities of patterns in stock charts, we probably should come up with a method to identify patterns. To do this we are going to come up with a really simple method to start exploring the nature of pattern recognition. Just as the Naïve Bayes algorithm ignores certain assumptions made by the Bayes Theorem, we are going to work towards a practical way to pull patterns from streams of data.

To be honest, I have not worked with chart patterns in any detail before this week. Looking into the study of chart patterns has always been on my To Do list. I have been interested in Sequence Patterns – not dissimilar with chart patterns – but more one the discovery of new patterns, rather than detecting patterns that are well known – like the Head and Shoulders pattern. Over the years I have heard people mention Head and Shoulders and although the pattern’s name is pretty descriptive, I was not 100% certain what it looked like. After some browsing on the web and looking at real and prototypical examples, I came up with this simple graphic that gives us a template of the classic Head and Shoulders pattern.

In fact, this is the actual template that we are going to use. Notice that we are looking at 7 price points. The first three points form the left “shoulder”. There is a peak in the center above the two “shoulders” – this is the “head” of the pattern. Finally, the last couple of points give us the right “shoulder”. Again – remember that I am not a Chartist – an expert in chart patterns or even a believer in chart patterns. That is – until I get some interesting evidence that the patterns are meaningful.

We have our prototype Head and Shoulders pattern that we can use as a template, overlay the template on a stock chart and decide if the pattern is present in that chart or not. Being that we are very lazy and have technology at ready hand, we decide to come up with an automated routine for finding the Head and Shoulders pattern in our stock data. Luckily, we have been using a simple measure for finding stocks that move together – correlation.

Correlation is simple, fast to calculate and we will see how good it is at finding patterns in data. Basically, we are taking the seven numbers that were used to create the template graphic and then doing a correlation with the closing price over the last seven trading days. So, we are able to find patterns like this:

Our template is below for comparisonon. The graphic above is AMNT closing prices over a 15 day period and we have a seven day Head and Shoulder pattern highlighted in red. That is pretty distinctive, so that seems to work pretty well. Hmmmmm… Are all Head and Shoulder patterns 7 days long? I don’t think so.

So, we need to come up with a way to stretch the pattern out and compare across longer time dimensions. I came up with a morphing routine that stretches the pattern to twice its size, three times, four times on up to a time frame of about 100 days in length. Although the routine is kind of crude, it does give us a way to automatically find perspective Head and Shoulder patterns that are up to one hundred days in length. At this point, the artistic part of chart analysis comes into play. We have the routine run every morning on the previous day’s closing prices and present the stocks which seem to show the Head and Shoulders pattern. Take a look at the Head and Shoulders pattern experiment page and see what you think.

Look, there is no denying it – we love science fiction here at DeepMarket. Mind you, we are more than willing to allow for certain suspension of disbelief when it is required to make the story more interesting or even possible. For instance, most space related sci-fi has some type of faster than light propulsion systems. Golly, who wants to watch a story where it takes several generations for a ship to just get to the nearest star beyond good old Sol? We suspend our disbelief and enjoy the story.

Language is one of those areas that sci-fi has to cheat, like it does with faster than light travel. Here on earth there are thousands – if not tens of thousands – human languages and dialects. Translating between all these languages is hard – really hard. So, it seems that some sci-fi just ignore the problem - like in the Serenity/Firefly universe (although they seem to be multi-lingual without any technological assistance). Or they have to come up with a technology that explains the incredible ability of multiple species to converse in multiple languages and still understand each other. Star Trek has their Universal Translators and Farscape has their Translator Microbes. These technologies allow the users to understand and communicate in languages that they had never encountered before.

Yet, this is one area that is difficult to truly suspend disbelief. I am sure anyone who has learned a foreign language would also agree – not to mention the poor scientists that are working on natural language processing today. It is hard enough to extract meaning and understanding from one language and the systems that do it well are not learning systems – they are generally programmed from the top down. It is just the way the domain is at this moment. Not to say that good learning systems can’t be developed in the realm of language – just pointing out that the top-down systems still have the edge for the moment. I invite NLPers to correct me – this is a difficult domain!

So, when a story has someone waiting for their handheld (if not invisible) universal translator instantly learning a language never encountered up to that time – it seems hard to believe. So, let’s move beyond sci-fi and back in the practical. One reason this universal translation/learning/understanding process seems so extraordinary is that there is no feedback loop. Some alien life form is talking and the technology is able to derive meaning from it – in an unsupervised manner. Unsupervised basically means that the technology does not get graded and has no idea that the translations it is offering up are correct. Granted, I am oversimplifying the concept of unsupervised learning and these techniques have a place in language learning, but I doubt an effective language learning system could be developed without some type of feedback.

This leads me to my practical application – stock chart patterns. Dare I say that chart patterns are a type of language - the language of the markets? Not unlike trying to decipher ancient hieroglyphics of some lost civilization, trying to understand the language of the markets without some form of feedback is a sure way to waste a good deal of time. However, it seems that is what chartists do when they are looking at patterns in charts of stock prices. There is an alphabet of the market – Head and Shoulders – Wedges – Triangles – Rectangles – are all patterns in charts that perhaps are better described as hieroglyphics. These patterns are supposed to describe and help the analyst to understand the future direction of the market. This is the clash of science and art. Describing the chart patterns are generally pretty straight forward – however, actually recognizing “valid” chart patterns is an art.

Or, is recognizing chart patterns an art? This week we will look at some chart patterns and see if we can come up with some methods of finding patterns in stock data and then determining the resulting price movement after the pattern was recognized. We are trying to put the feedback loop into the process and chip away a little at the art of chart pattern recognition.