One of the major problems in dealing with large amounts of data is the eventual presentation to a human being. Humans - constantly making things harder to develop and deploy. Even worse is the eventuality that the data needs to be interactive. The phrase “that looks too busy” is not very encouraging when you are trying to compress complex data into bite size and easily digestible pieces. Edward Tufte is famous for trying to do just this - make complex data immediately make sense. My first introduction to Tufte was an interesting poster (not created by him, but by Charles Joseph Minard) that depicts Napoleon’s army in the failed Russian campaign of 1812. I had not actually looked at any of Tufte’s work in many years, but recently found the Sparkline PHP Graphing Library. Tufte had come up with Sparklines in his book (still unreleased) “Beautiful Evidence” and called them “intense, simple, wordlike graphics”. Wow - those are pretty cool.
I had recently re-run the correlation between all the stocks in the NASDAQ over the previous 250 days. This seems to be an interesting way of presenting the data. As it stood, I had a list of the top positively correlated and top negatively correlated stocks with a cutoff of about + and - .75 for every stock in the NASDAQ. No graphics and it really didn’t highlight the fact that over the past 250 days, these stock looked like they moved together (no implication of causation - just simple correlation). So, in addition to the list of stocks correlated with a particular stock, I added a nice little Sparkline with 250 days of the closing price. Now you can look at a stock like Google and immediately see the stocks with positive correlation have a similar 45 degree slope upwards and the negatively correlated stocks are quite the opposite. The QQQ has a generally positive move upwards, but a bumpy ride - as do those stocks correlated with it. It is interesting to note that Intel has no stocks correlated (plus or minus) higher than .80, but with the Sparklines you can easily see the stocks that had movement closest to INTC.
I have to thank James Byers for putting together the Sparkline project.
