Stock Market Analysis at DeepMarket

It was simple to program, but I kept putting it off. Now you can see the “Find Symbol” button located on the right side of all pages. Just type in the symbol of the stock you would like to see information about in the space above the button and hit “Find Symbol”. Type in a partial symbol and it will bring back all the matched symbols. Pretty simple.

DARPA (the Defense Advanced Research Projects Agency), for the second year in a row, is running the DARPA Grand Challenge on October 8th. The Grand Challenge is to develop an autonomous vehicle that can navigate tough terrain - in this case, a desert in Nevada. To win, the vehicle has to complete a 150 mile course in under 10 hours. Remember - this is not a remote control vehicle - it is completely autonomous - no human interaction or help. Why would anyone compete in this challenge? Other than bragging rights - how about a $2 million prize to the winning vehicle’s team - not bad!

Of course this is not just an artificial intelligence problem - it has other challenges that make this a true Grand Challenge. In 2004 none of the contestants came close to completing the course, but I am certain a great number of lessons were learned and the feedback into this year’s competition will certainly move everyone closer to the ultimate goal of autonomous vehicles.

This is a complicated task to be sure, but one team is using a motorcycle. The BlueTeam made up of students from UC Berkeley and Texas A&M have an excellent entry in this year’s competition. Last year they made it to the final race, but didn’t get too far - actually fell over at the start line. This year the team has created a much improved vehicle - check out the videos of autonomous motorcycle test runs.

In All the President’s Men, Deep Throat told Bob Woodward to “Follow the money” as part of his investigative efforts surrounding the Watergate break-in. So, is it any less reasonable for Deep Market to advise you to “Follow the Mad Money”?

Well, if you were referring to Jim Cramer’s “Mad Money” on CNBC, then you may want to do some serious due diligence before spending a great deal of time following the Mad Money. Jim Cramer is a wonderful personality. He is a smart guy. He is fun to watch. In fact, my 14 year old son refers to the show as the “Mad Monkey” show.

So, it was pretty interesting when I found the BOO-YAH BOY AUDIT! website. This site has been tracking Jim’s recommendations since the end of July (2005 - the show started this year). Heck, that is hardly long enough to make any judgments. But, what I will say is that there are a LOT of recommendations made by Jim - even within one show. As 23 Sept 2005 - they have tracked a total of 153 recommendations. Wow - that is a goodly amount of commissions. What is the performance of the portfolio (if you were to follow ALL of the recommendations)?

2.11%

Okay - is that truly “Mad Money”? I will leave that to the reader to decide. There are also forums, like CramersMadMoney - unauthorized discussion forum, to discuss the daily ranting of Mr. Cramer. Here you can continue to mix it up with other Cramericans after Jim has gone off the air.

So, as with all potentially hyped information sources - Deep Market says to “Follow the numbers” and keep checking Jim’s record, but enjoy the show.

The age old competition between brawn and brains continues in the computer age. Seemingly “intelligent” demonstrations of machines’ abilities to compete with humans, generally with games like chess, are really dependent on computer “brawn”. True intelligence in machines - artificial intelligence - will more than likely be the result of leveraging really clever algorithms that side-step the need for brute force calculations.

So, it wasn’t my intentions to have to write about this classic struggle when I decided over a week ago to do the simple calculation of correlation of some stock prices. Well, about 3000 stocks. Overly excited that I got a database populated with almost 3000 stocks, I set about writing a simple Java program that simply calculates the correlation of stock prices. Even more simply, I chose to only look at the last 100 days for each of the stocks.

Simple.

For instance, let’s say we want to find the correlation between Microsoft and Amazon. I pull the latest 100 closing prices for each stock and plug them in and out pops the correlation -> a number between -1.0 and +1.0. This is a mildly interesting number and it wasn’t an exercise that I meant to put a great deal of effort into - therefore I setup the program to calculate the correlation between all of the stocks in my database. All 3000. For a total of 9,000,000 calculations. Okay, at least I was smart enough to only calculate the 4.5 million unique combinations - the correlation of MSFT vs. AMZN is the same as AMZN vs. MSFT. Okay, so only 4.5 million calculations, but this number did not expose itself until late in the day AFTER I kicked off the job.

Brute force calculations - sometimes you just gotta do it. Off the top of my head I can’t come up with any neat short cuts to get around brute forcing the calculation of all correlations of the stock prices. Yes, I am sure I could have come up with some cute tricks within the code to speed up the process, as it was hacked together. This is one of those precious examples that distributed computing or parallel processing - brute force - would cut through the problem like a hot knife through butter.

Anyway, 200 hours later we have the correlation of 100 days of stock prices between all the symbols in the database. I might update it again in the future - I might not. I plan on looking at the data and writing a more interesting article about what I see in the correlation numbers, but for now - take a look and see if you find anything interesting.

The constantly evolving and growing backend of DeepMarket is starting to peek out from behind the curtain. Finally, got some stock data loaded into a database. For now, just looking at stocks on the NASDAQ exchange. Explore a little if you get a chance. For example, here is some information on Intel:

INTC - Intel

No search functionality has been setup yet. If you know the ticker symbol of your stock, just replace SYMBOL (see example below) with the symbol of the stock you are interested in seeing:

http://www.deepmarket.com/stock/SYMBOL

I have some other numbers up there right now (correlation numbers), but the data for that is still being loaded into the database. Those numbers will be changing through the day and I will write up an article when that is finished.

Sir Isaac Newton said, having participated in the South Sea Bubble and losing about 20,000 pounds of his own investment, that:

I can calculate the motions of the heavenly bodies, but not the madness of people.

It is interesting to note that present day technical analysis has been inspired by Newtonian physics, like the concept of momentum of a stock’s price. Other areas in physics are drawn upon to help develop techniques using harmonics, waves, and cycles for the analysis of markets.

Are these techniques effective in understanding the movement of prices, ultimately determined by the “madness of people”? Practitioners of various methods will certainly have their opinions about the strengths and weaknesses of these techniques, but often it comes down to the artistic application of human judgment. This, of course, is the creative nature of analysis.

Artistry and analysis seem to be opposing concepts. However, using creativity to develop new techniques is good. Being artistic in the analysis of results from a technique - well… That is really up to the user. I am more comfortable being analytical in the application of the methods and saving my creativity for creating new methods.

So, can new inspiration be drawn from the world of Physics?

Creative energy can be burned out trying to implement a neat, new idea. Progress can be hindered by an interface or constraints imposed by a machine - this is the dilemma computer programmers face daily. But, to be fair - that is their job. Other, more mundane tasks, such as using a calculator also impose extraordinary impositions on “non-technical” people. Considering how long the calculator has been around - compared to the “home computer” - it is pretty amazing that the interface has not really progressed all that much. Students learn how to do mathematics on paper one way - then have to learn how to solve the exact same problems in a different way on calculators.

It looks like Will Thimbleby and his father Harold have been working hard to come up with creative solutions to this mismatch of human interfaces. Pen and paper interface has one way of doing things - calculators require a context switch. Several papers - including A novel pen-based calculator and its evaluation - have led to a working prototype. Try out the Java applet - the instructions say “Write mathematics as you would on paper” - and it works - very cool!

Really put it through its paces - try all kinds of stuff. Granted, working with a mouse to write numbers is a pain, so imagine using this interface with a graphics pad or an interactive white board. Now, think about this as an interface to a trading system testing framework. Or as an interface to a spreadsheet. A lot of interesting possibilities!

The sub-title of this data mining blog is “Ruminations on computational geometry, algorithms, theoretical computer science and life”. I enjoy capturing links to blogs that have a great number of entries to browse through for new ideas. This blog has been around since early 2004 and it is updated quite often.

For instance, I found this article about Streaming Algorithms especially interesting and pertinent to the field of stock market analysis. Remember, stock market data is inherently “streaming” in nature - if you wish to deal with it in real-time.

Anyway - that is one of many articles that are excellent insights into data mining on their own and interesting for their potential application to the analysis of market data.

The Machine Learning (Theory) blog is an excellent place to read about various aspects of Machine Learning. John Langford is the evolutionary based intelligence behind this blog and he has quite a list of research interests and projects. Read and generate new ideas!

Okay, I have given up on Ping-o-Matic. It has never worked for me.

The best alternative I have found is Pingoat. Less than two months old, I have had little to no trouble with it yet. Pingoat is having growing pains - mostly with trying to filter out all the spam blogs. Of course there is The Goatlog, which is where you go to get your immediate updates and a newly opened Pingoat forum - should this be the Goat Pen?? Thanks Kailash Nadh (he is Pingoat’s shepherd) - great job!

There are a lot of stock market related blogs out there. I have looked at many of them and have been impressed with the effort that some people put into providing (potentially) valuable analysis for free. Of course, there are a lot of splogs - spam blogs - also. However, in looking through the universe of stock blogs, I found Stock Picks Bob’s Advice website. From what I can tell, it looks like Stock Picks Bob has been doing a great job documenting his ideas, background information, and charts since 2003. Golly, and what a history of postings to read through. Did I mention the pretty pictures? Not just the charts, but also nice photos.

I am guessing that Bob has an interest in Point and Figure charting - most of the charts I saw use P&F.

He also tracks his trades, but I don’t see an update since June 2005. If you have a question, Bob seems to be more than happy to provide answers - he has “A Reader Writes…” sections sprinkled liberally through his other postings. He begins his answers to questions with the following caveat:

As always, please remember that I am an amateur investor so please consult with your professional investment advisor prior to making any decisions based on information on this website.

Truer words have not been spoken, but I doubt most “professional investment advisors” would be willing to have their trading records in an open forum like Bob has his. In the world of brooding and negative blogs, Bob provides a useful and positive source of information.