Let me start by saying this: I am a data junkie. I love data. I love making spreadsheets, trend lines, projections, and trying to eek conclusions out of data. I used to love Excel, but now that I’m a Mac guy, I love Numbers. I love importing data and generating beautifully elegant graphs that sumarize findings. There, I’ve said it. Hopefully that provides a small amount of context for this post.

I’ve been thinking a lot about all the data I gather related to my business. Every morning when I get up, I check the following list of analytics data:

• Dapple sales stats from the previous day
• Dapple Lite download stats from the previous day
• Dapple and Dapple Lite world-wide rankings in the App Store
• Google Analytics for Streaming Colour and App Treasures
• LinkShare reports for click-throughs to the App Store from various places on the web
• Google AdWords campaign data
• YouTube channel view data
• Bit.ly account for click data on some important links

There is an incredible amount of data available to us as iPhone game developers and web site hosts to micromanage aspects of the business. The staggering thing is that all of this data is available on a daily basis. Some of it is even available on an hourly basis! What I’ve started wondering recently, however, is if checking this data every day is necessarily a good thing.

Take a look at what happened with the iTunes App Store last week when Apple rolled out iTunes 9 at their iPod event. The store was broken for a few days, in that games sub-categories disappeared. The issue was fixed in a few days in the US store, and this morning it looks like the Canadian store has been fixed. But developers, myself included, went insane! We were able to watch our sales plummet every day, and watch our rankings plummet by the hour.

The important thing to note is that, for myself, I had one bad day of sales as a result. After the 2nd day, my sales were basically at where they were before. I’m sure for some other developers who have larger daily sales, it had more of an impact. But it really got me thinking that maybe I worry about the short-term too much. Maybe we all do.

Don’t get me wrong. In an industry that moves as quickly as ours; where a game can go from the #200 ranked game to the #10 ranked game in a matter of hours, this kind of granularity definitely has its place. It’s important to follow your trends and pay attention to what works and what doesn’t.

However, I think it’s equally important to remember that not everything can be tracked at this level of granularity. For myself, even, in my first Numbers Post, I talked about the small spike in sales I received from the Kotaku review of Dapple. But what the data and the graphs don’t tell me how many people saw the review but didn’t buy the game right away. How many people saw the review, but didn’t buy the game until they read two other reviews of Dapple? When we look at the data in such a narrow window, we lose sight of the bigger picture.

After Dapple’s release, I spent the next several months promoting it, advertising it, coding updates for it, and watching all the data roll in. I tweaked what I was doing every day in response to the data. While this approach taught me an incredible amount about what seems to work and not work when promoting a game, I think there are also several big problems with this:

• It takes too much time – I’m a one-person studio. Every hour I spend on one thing means I’m not spending that hour somewhere else. While marketing and PR are an incredibly important part of launching a game (the most important, perhaps), spending that time wisely is also important. At the granularity I was looking at data and reacting to it, I think I wasted a lot of time. I will learn from that with my next game.
• You lose sight of the big picture – Early on, when I’d have the occasional day of 0 sales, it would suck the wind right out of my sales. It was like someone slapped me across the face. But what I’ve learned is that a day of 0 sales now and then doesn’t matter. If every day has 0 sales, that’s a different story. But the occasional bad day is going to happen. What I try to do now is look at bigger trends more often. I look at sales over a week or two and look at whether they’re trending generally up, down, or stable. This allows me to make smarted decisions about what to try next.
• You don’t have enough data - My university degree is actually a Bachelor of Mathematics, so you think this lesson would have set in sooner. I took a lot of stats classes, and they told us this a lot, but sometimes you have to remind yourself. There’s a lot of noise in data. This is especially true when your sample size is small. This relates to the previous point. If you’re selling between 3 and 10 copies per day of a game, it’s impossible to draw any kind of conclusions from daily data. There just isn’t enough of it there. However, if you have 10 sales one day, and 3,000 the next, that’s data you want to pay attention to. Looking at data over the longer term helps to eliminate noise and show more important trends that you can draw real conclusions from. If I change my app description text one day and my sales go from 2 to 4 the next day, I can’t conclude that my description change doubled my sales. However, if my sales went from 14 sales per week to 28 sales per week, now I can feel slightly more confident in saying that it helped.

So my point is really this: daily data is important. We shouldn’t discount the usefulness of this data. It has the ability to show us big changes due to immediate events (like being featured by Apple, for example). But, let’s not forget that data over the long term is also extremely important, especially when you’re looking at building a long-term business strategy. We can’t rely solely on the daily data to derive our conclusions. There’s just too much noise in that data most of the time. Plus, worrying and agonizing about your data every single day will just drive you nuts.

For my part, I will keep checking that data every day, because I do need to be aware of daily changes. But I’m not going to let that daily data run me and my business; I’m going to try to draw conclusions from data over the longer term. When I see big changes from one day to the next, that’s when I’ll dig in deeper to find out why. Hopefully this will allow me to concentrate on what’s really important: making and selling great games.

Owen

This entry was posted on Tuesday, September 15th, 2009 at 1:47 pm and is filed under Business, iPhone, Misc. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.