Stock Market Analysis at DeepMarket

It is pretty obvious that DeepMarket upgraded to a new WordPress theme - finally! We have Colleen from Mystical Twilight to thank for developing our custom WordPress theme. If you have a need for a new theme, be sure to contact Colleen or check our her free WordPress themes. Right now you see the new theme in the “blog” section of DeepMarket, but not in the data areas - such as Stock and Correlation areas. We will get everything integrated over time. We had also upgraded to the most recent version of WordPress, so we are juggling implementing new features and balancing with possible performance issues.

I wanted to highlight one of the WordPress plugins that was added today. The quality of comments on the posting here has been pretty high. Since WordPress defaults to “nofollow” tags on links in comments, I decided to remove that with the Dofollow plugin for WordPress from Semiologic. I appreciate the excellent feedback I get via comments and want to encourage good behavior with live links back to contributors - thanks!

I don’t want to beat these guys up too much, but I blundered onto the web site for Proteom Capital and I suddenly felt like I was thrust into a Philip K. Dick short-story.  I love buzzwords - no doubt.  I sprinkle them into the posts here and other places way too often.  This is, of course, aweful - because I have become that which I hate.  So, back to Proteom Capital’s web page.  Proteom runs several private equity funds, which means you have to be an accredited investor to get in on the action.  The buzzword intensive copy made me wonder - which came first?  The buzzwords or the business model?

“Genetic algorithms, neural networks and computer clusters — the basis for new decisions.”

From the front page of Proteom’s site, these technologies are near and dear to my heart - but don’t really mean that these techniques are better decision makers.  No matter how good they are, the human can be the weak link - or the salvation.

“Using the power of computational biology to navigate financial markets”

And they go on to explain that “Proteom creates high performance securities portfolios with a rare combination of algorithms, hardware, and software tools derived from computational biology.”

What it really comes down to is fund performance - no matter how much computational biology you throw at a problem.  What I see looks okay, but does not seem to reflect the generation-leaping technology they describe.

At least they left out references to Swarm Intelligence and Fractal algorithms.

Since there was some new interest in shorter correlation time periods, I finally got around to adding 10 day correlation data for all the stocks. A friendly reader - Jon - had asked about looking at multiple periods:

Time frames of, say, 365 days, and 250 days, are useful but more importantly perhaps, the shorter time periods of 100 days, 30 days, and 7 days really need to be looked at.

So, what I did was set everything up to display 200 days, 100 days, 30 days and 10 days of stock correlations. For each stock, the top 5 positive correlations for each time period and the top 5 negative correlations for each time period. I decided to go with a 10 day period instead of the 7 day period because we are working with trading days, not calendar days - 5 trading days in a week. Not really important, because obviously the 10 days cuts across calendar weeks - it just seemed nicer to be looking at two trading weeks at a time.

I had also mentioned that DeepMarket doesn’t have index data and Jon asked:

Not having index data is a problem, as sometimes it would be useful to look at these correlations in the context of the market. Is it not possible to use another source? What about Yahoo! they have a lot of available historic and recent data.

Well, this is a question of MAY I vs. CAN I. There is no question that I CAN, but I MAY NOT display that data here on DeepMarket. I don’t have the rights to display that data - sorry! To show that data is really expensive and I only display the data that is in my feed. The exchanges (who own most of the indexes) are really weird about it and I was looking for data with my “path of least resistance” algorithm - so, what you see is what you get. Don’t even ask about mutual fund data…

However - you can look for proxies. A fantastic proxy - and probably the most famous is the triple Q or QQQ or QQQQ - which is the NASDAQ 100 Tracking Stock - check the link for a better understanding. Bottom line - its moves are correlated with the NASDAQ 100 Index - and an excellent proxy for the index. Of course, in this case the movements are not only correlated, but causal in relationship to each other. So, use our little search tool in the upper right corner and see if you can find a proxy for the index you are looking for.

Okay - that is a ton of data out there for all the stocks. I am not sure if it is helpful - data is always the first step. Now that we have all this data, I hope people are able to come up with some interesting strategies. Jon gave us some clues and hopefully others are willing to share their ideas and insights - I welcome comments!

Social bookmarking has had a great deal of growth in the past several years, with sites like (loosely categorized as Social Bookmarking sites) StumbleUpon, del.icio.us, Digg and Furl leading the way. These services are great for exchanging, highlighthing and discovering web sites. It was only a matter of time before the same paradigm was applied to picking stocks.

DigStock is using the basic format that Digg popularized - but with a good number of additions. Just like Digg, DigStock allow you to submitt stories that are voted on by the website membership. The popular stories appear on the front page. The user can also rank stock and indexes - give readers an idea of what other stocks people are interested in. This feature seems to need some work, as it is a little slow. A great deal of caution should be exercised when looking at what people have voted up and voted down. Bookmarking systems are prone to manipulation and DigStock is no exception. They also have a specific trader blog dig section with stories from stock market related blogs - again, allowing you to vote on those stories.

Another site that is in an open Alpha test - socialPicks - describes itself as a place to “Share your stock picks and research.” They seem to be leaning more on the tracking and validation of picks by individuals - kind of like a universe of amateur stock analysts. If it sounds interesting, go over and sign up for an invitation to test out the system.

I just saw a post by Jason over at InvestorGeeks where he gives his First Impressions of SocialPicks and StockTickr - which introduced me to another site I was not aware of - StockTickr.

Upgrade the server to Fedora Core 5 and finally got WordPress upgraded to the latest version.  Not sure if my feeds are working correctly, so this post is not exactly user friendly or very interesting.

Okay, we now have three time periods covered for every stock in our database - 200 days, 100 days and 30 days. It took a little while to get things optimized so that we could re-run the calculations every days. Remember - we have well over 8000 stocks in the database, so that is (8000*8000)/2 = 32 million correlations that need to be calculated for each time period.

What the heck is the deal with IBM? It seems odd that no other stock has a positive correlation over 200 days that is above .80. It seems that MSFT is also weird - nothing over .80 for 30, 100 or 200 days.

Please excuse the presently slow site - we are trying out some suggestions in regards to the display of correlation of stocks. Major parts of the correlation database is in the middle of being re-calculated, so many of the stocks will have blank pages for the next couple of hours. The results should be interesting!

Google seems to have hit the wall when it comes to creative and exciting new ideas. Proof of this is the new Google Related Links from Google Labs. Their tag line is “Provide your website’s visitors with links to useful information”. Isn’t that basically the job of the website developer/writer/blogger? Just for giggles, I have placed Google’s related links boxes in the right column of the blog and on the stocks pages.

What is a little concerning is the “safety” of such an idea. Google got to where it is by being bold and pushing the envelope, but this seems to be one of those ideas that is “good”, but really not great. Don’t even get me started on Google Talk.

I made a big mistake. I was under the impression that Head and Shoulders was a “bearish” pattern, when in fact it may depend on the preceeding trend. Jon, from the Fickle Trader pointed this misunderstand (okay - let’s be honest - my ignorance):

If there is no preceding trend, then why should we expect a trend reversal? If you are looking for price declines after the head and shoulders pattern completes, you need to identify the conditions where the stock has a preceding uptrend on the same timeframe as your h&s pattern.

Oooooooops. He also says:

For the purposes of head and shoulders analysis, I recommend we define a prior uptrend as a series of higher highs and higher lows (you can identify this with correlation too).

Dang it - I messed up that last experiment in looking at head and shoulders, so we will re-do it to see what difference it makes. Thanks for pointing out my, uh, ignorance!

Honestly, I didn’t think much about correlations of stock prices when I first put together the section that showed highly correlated stocks - it was more of an interesting exercise. I had a bunch of stock data and wanted to see what it looked like on the web and showing stocks that were correlated was a straight forward experiment to try. For instance, you can see what stock are correlated with Google, Apple Computers, Yahoo!, Microsoft and IBM (curious side note - sometimes IBM has not correlations over .8) over the past 250 trading days.

Well, Jon gave us some really interesting feedback and brought up some questions about the correlation data we are showing. It seems that Jon is a psychologist specializing in investment and the author of Profits Without Panic : Investment Psychology for Personal Wealth. Let me parse out Jon’s comments and questions:

The fact is that while correlation is an important method there has been very little work in its use as an indicator. Take a look, for example, at a range of websites and platforms and while there are many indicators on offer from moving averages to elliot waves and so on, few if any offer the basic correlation. Surprisingly sites such as Yahoo! give both an r and r-squared value for any stock against a major index so its not like there isn’t a use or a demand.

Jon is correct - it seems that correlation isn’t a big part of the stock trader’s toolbox. Should it be? I am not sure yet. It seems that many people are interested in the correlation of indexes to stocks and look for divergence and such, but looking across all the stocks to get a sense of what stocks are correlated with other stocks - not a top priority for traders.

The point is that a tool that works out correlations is vital to an understanding of market behavior.

Cool - I had not thought about this too much yet. I always seem to be looking for the next problem to be solved before I finish the one I am working with! I definitely need to get some more ideas about this.

Information can be downloaded into Excel as you suggest but it is slow and laborious.

Oh - that was just an article to explain the process for finding correlation and Excel is great for simple demonstrations like that. Indeed, running correlations on 10,000 stocks would be very laborious in Excel (yes - yes - I know - you can write VBA to do it easily…).

Indeed your tool can go further, I believe. What’s needed is the ability to run correlations on stocks and indexes (I found this didn’t actually work on your site by the way) and most importantly to be able to alter the time period.

DeepMarket does not have indexes - sorry! The stock data we have now is pretty expensive as it is - for display on a website. To display indexes is cost prohibitive. You would NOT believe what the exchanges want independent website owners to pay to display index quotes. So, as a matter of practicality, we do not have indexes. Also, the correlations are run once a week. Finding the correlations between 10,000 stocks is a decent computing job and we certainly don’t do any of that on the fly. Altering time frames - adding time frames - is a great idea and we most certainly can do that. What time frames would people like to see?

My feeling is that for long term holding periods a long term correlation may be fine but for short term holding periods a short term correlation is more suited.

Okay - no problem. We can do shorter term correlations. For those, we may run daily computations - depending on the time frames.

I’d be interested to know your thoughts on this and if you have any intentions to expand the functionality of the correlation tool.

Heck yeah! I am interested in any further feedback from all readers. What correlation periods are you interested in? Remember, all I have is daily data.

Thanks for the feedback Jon!