Correlation Pattern Matching Explained
It has been pointed out that in the article Oversimplified Method for Finding Patterns in Stock Charts the actual method used was not described in detail. The process is pretty simple - you might even say oversimplified. It was just an error of omission, not an attempt to hide any proprietary methods. In fact, just as a gentle reminder - this blog is for the open discussion and exploration of advanced techniques for analyzing the stock market.
So, let’s review. I am not a chartist or a technical analyst, but am interested in understanding patterns that seem to have “legendary” status among those who dabble in these arts. One of my greatest concerns is that there is little work on validating chart patterns in an objective manner. So, for instance - how reliable, profitable and practical are these patterns?
Well, before we can approach those questions, we needed a way to detect patterns in a repeatable fashion. Pattern matching is a fascinating field, but we just needed a quit and dirty solution. Correlation was that simple solution. What was not described was how we are using correlation to find patterns in stock charts. We are going to use some data that never changes (a stock in the database that has stopped trading) and Excel to show this simple process.
AMNT stopped trading on 12/12/2005 and just so happens to have a really classic head and shoulders pattern in the last seven days of trading:
Using Excel, we can describe the process that is used to find the head and shoulders pattern (or any other pattern - given a template of the pattern). First, we need a template of the pattern we are looking for in the chart data. I explored the charting and technical analysis literature and “eye-balled” the patterns people were describing as head and shoulders. This is far from scientific, but a good start. What I came up with as a template for head and shoulders is:
Or, very simply using numbers from 0 to 100:
0
60
20
80
30
70
20
Seven data points gives us a template for what I think a head and shoulders pattern would look like. This template is certainly open to discussion, but this is more about the process than the correctness of the template. What is important is that we can use the above template to easily and quickly identify that general pattern. We have a template of 7 data point, so we need to look at 7 data points from the stock data we wish to search for patterns. We are using AMNT and the last 7 data points (looking at closing prices) are:
10.22
10.25
10.24
10.28
10.24
10.26
10.24
Now, open up Excel and paste the two columns of data into columns A and B of your worksheet. It doesn’t matter which column you put the data - template in A or template in B:
Now, using the Excel function CORREL we are going to find the correlation between the template and the stock data. Put the following formula into any cell that isn’t already being used:
=CORREL(A1:A7,B1:B7)
The result is a correlation of .940712, which is a pretty good match. You look at the pattern’s template and the actual stock data and see that it is a pretty good match. Pretty simple, huh?
For what we are doing, we have this process written in Java and every morning we search for patterns that have a correlation greater than .85 - why .85? No reason, just an arbitrary point. For those who wish to do this for themselves in other languages or technical analysis packages, you just need a correlation function and the ability to use that function to compare two sets of data. Once you have that setup, then you can play around with any number of templates describing different patterns and correlation cut points.
I have no idea how I got here….but I am using Stumble!
I want to start learning more about how to invest - not doing much but a 401(k)….and my house…..right now.
My stock blog seems childs play compared to yours.
Your blogs very great. This head and shoulders, I understand how to get it and everyhing now, but what does it mean?
Don’t you want to get into the stock before it gets to the head? How would you predict the head and shoulders before it we can even find a correlation.
Aswell how did you learn about all this. I want to be as knoweldgeable as you one day. Did this all come from university? I myself am hopefully going to get accepted into Princeton with in the next week, where I can embrace all these technical analysis of stocks.
Thanks for the great blog.
You might be interested in these books by Thomas Bulkowski:
Encyclopedia of Chart Patterns, Second Edition
Trading Classic Chart Patterns
http://mysite.verizon.net/resppzq7/mybooks.html
From the description of Encyclopedia of Chart Patterns:
In each chapter of Encyclopedia of Chart Patterns, Second Edition you’ll learn the following about each pattern:
-Results Snapshot—A statistical summary of pattern behavior, including its performance rank, breakeven failure rate, average rise or decline—all separated by breakout direction and market type (bull or bear)
-Tour—A broad introduction to the pattern
-Identification Guidelines— Characteristics to look for
-Focus on Failures—What failed patterns look like, why they failed, and how to avoid them
-Statistics—The numbers and what they tell you, separated into bull/bear markets and breakout direction, including average rise or decline, failure rates, volume shapes, performance by size, and busted pattern performance
-Trading Tactics—Strategies to increase profits and minimize risk
-Sample Trade—Puts it all together, showing the chart pattern in action, with hypothetical or actual trades using real data
-For Best Performance—A table of selection tips to boost performance
I realize that my previous comment looks like an ad for Bulkowski’s books. Actually, your last few posts reminded me of his books where he painstakingly quantifies each pattern, well, pretty much like the descriptions for the books say. If you haven’t seen these books before, you might like to take a look at them.
The posts on correlation are fascinating and I’m particularly impressed by the tool on the site that works out correlations between stocks. In fact I’d made some posts to the trade2win boards to attempt to find out information on precisely this subject a few days ago. In actuality I spend a lot of time researching these kind of effects both as an author (Profits Without Panic, 1999) and as a research fellow at Columbia University - and not as a finance specialist but as a psychologist specializing in investment! Yup - even psychologists are getting in on the act! The fact is that while correlation is an important method there has been very little work in its use as an indicator. Take a look, for example, at a range of websites and platforms and while there are many indicators on offer from moving averages to elliot waves and so on, few if any offer the basic correlation. Surprisingly sites such as Yahoo! give both an r and r-squared value for any stock against a major index so its not like there isn’t a use or a demand. Nevertheless, there is a problem, while your tool nicely provides the time period Yahoo! doesn’t, same as many others. Although many of them use one year of data.
The point is that a tool that works out correlations is vital to an understanding of market behavior. Information can be downloaded into Excel as you suggest but it is slow and laborious. Indeed your tool can go further, I believe. What’s needed is the ability to run correlations on stocks and indexes (I found this didn’t actually work on your site by the way) and most importantly to be able to alter the time period. So for example, we can then ask questions concerning whether different time periods have different associative qualities. Moreover, a correlation based on a year of data might be useful but surely it depends on whether the investment is intended to be long term or not. My feeling is that for long term holding periods a long term correlation may be fine but for short term holding periods a short term correlation is more suited. I’d be interested to know your thoughts on this and if you have any intentions to expand the functionality of the correlation tool.
[...] Oh - that was just an article to explain the process for finding correlation and Excel is great for simple demonstrations like that. Indeed, running correlations on 10,000 stocks would be very laborious in Excel (yes - yes - I know - you can write VBA to do it easily…). Indeed your tool can go further, I believe. What’s needed is the ability to run correlations on stocks and indexes (I found this didn’t actually work on your site by the way) and most importantly to be able to alter the time period. [...]
my personal experience has been that it is easy to pick a head and shoulder from the data. But a trend reversal is hard to confirm. However, it is more reliable to use mathematics based stock predictions. http://www.magicta.com conatins analysis and prediction history for nearly 8000 stocks traded in the USA.
[...] Here’s a couple interesting lists that Eric over at DeepMarket is adding using the StockTickr Email API. The top 10 Head and Shoulders Patterns (RSS) and Inverted Head and Shoulders Patterns (RSS) are being added at the end of each trading day. I asked Eric to explain a little about how his process works: As an experiment I am trying simple correlation to help bring up a list of “possible” head and shoulders patterns. I describe it in more detail in my articles Oversimplified Method for Finding Patterns in Stock Charts and Correlation Pattern Matching Explained. The process is very simple and the results are mediocre at best - I get a lot of nasty email from technicians - but very little insight about how they find them. I standard case of “I know it when I see it” syndrome. What I was trying to do is show how programs can help humans reduce their work load (looking for patterns), but generally it is still up to the humans to determine if the patterns are valid or not. [...]