I spent yesterday tracking the nastiest bug I’ve had on the project. Two of my testers reported a random crash in the game that only happened once in hours of playing. I had them send me their crash logs so that I could look at the callstacks. Both of them had had the game die trying to play a sound effect. Nothing else looked out of the ordinary about the callstacks.

(Aside: whenever an iPhone/iPod touch app crashes, it generates a full crash log on the device that gets transfered to the user’s computer the next time they sync. As a developer, this is incredibly useful. The callstack won’t have any symbol data, unless you gave them a debug build, but as long as you kept your dSYM file for the build you sent out, you can “symbolicate” the crash. There are full details here: iPhone Crash Logs. There are also ways to get symbol data for crashes, if symbolicatecrash doesn’t work, using GDB: Apple Tech Note: CrashReporter — I might write up a tutorial on this if I get some time)

Back to the bug…the first thing I needed to do was figure out how to reproduce the error. I’ve played Dapple for dozens of hours and I had never seen the crash. This worried me. My first reaction was to panic. Then I calmed down. A friend of mine happened to IM me while I was trying to figure out how to tackle the problem. He’s a really smart guy, so I laid out the problem for him. He suggested building some kind of debugger app that I could distribute with Dapple to my testers. Dapple would write out debug information and if it crashed, my testers would launch my debugger app that would pull the debug information out of Dapple and email it to me. This seemed like an interesting idea, but complicated. I started looking into sharing data between apps and then figured there must be something simpler.

I hit up the developer forums and searched the line my game was crashing on (the function was OALSource::Play(), for those who might be experiencing the same thing). Lo and behold, a few other people had the same problem. This is the problem with using Apple’s example code. Granted, they explicitly stated that the code was given “as is” and they made no assurances that it was bug free. I guess the problems we were all having was the reason they pulled that example code off the site.

Someone on the forums had said they thought it was a threading issue with the sound engine code, so they had converted the sound engine to run single thread and the bug seemed to have gone away. Someone else mentioned they thought it was happening when too many sounds were playing. This seemed like a good hypothesis, so I decided to test it.

I inserted a loop into my main update function that would play 2 sounds, 100 times each, every frame. This meant that about 6000 sounds would try to play every second. Then I let that run. After about 30-120 seconds, the game would crash with virtually the same callstack my testers had sent me! Yay! Reproducible! This also seemed to confirm that a high volume of sound could be the cause of the crash. Or, more likely, that starting and stopping the same sound file multiple times per frame was not a good idea.

I went back into the sound engine code and added in some new functionality that does the following: each sound effect now has a flag that says whether or not it has been played during the current frame. When the sound is played, that gets set to true. If another request for the same sound comes through in the same frame, it gets ignored. This doesn’t affect the sound quality of the game, since playing the sound twice during the same frame would sound the same. At the beginning of the next frame all the sound flags are wiped clean again.

I ran my test again (6000 sounds per second) and I let it run for 30 minutes without incident. So, while that’s not 100% guaranteed fixed, it’s enough to convince me that it is. Well, it’s either fixed, or the bug has become so infrequent as to become a non-issue.

After I had the bug fixed, I decided to do some more mem leak testing and discovered a leak in the sound engine when locking my device. This was because of some code in there during sound engine teardown that was doing something silly (I’m simplifying the code to show the problem):

for (UInt32 i = 0; i < soundEffectMap.Size(); i++)
{
    soundEffectMap.removeIndex(0);
}

See the problem? Yeah, that loop only executes through half the map. Say there are 4 elements in your map, after the first time through the loop, i = 1, but there are now 3 elements left. Next time through the loop i = 2, but there are only 2 elements left, so it exits…with 2 elements still in the map!

So I thought I’d be clever and do this instead:

UInt32 mapSize = soundEffectMap.Size();
for (UInt32 i = mapSize - 1; i >= 0; i--)
{
    soundEffectMap.removeIndex(i);
}

But found that my loop never exited at all! Whoops! Dang UInt32, ruining all my fun. See why? i can never be < 0, so the loop never exits. The next time through the loop after i = 0, i = -1, but it’s a UInt, so i = 2^32 – 1, which is clearly greater than 0…and on and on it goes. Changed that to an int. Bob’s your uncle, Fanny’s your aunt!

Now my game is, once again, memory leak free (as far as I can tell), and crash free (as far as I can tell). I sent out a new build to the testers with the fixes last night, so now I’m waiting to hear back on the results.

Today I’m going to create a new gameplay video for Dapple. It should be nice to do something not related to bug fixing.

Owen

This entry was posted on Wednesday, February 4th, 2009 at 7:49 am and is filed under iPhone, Technical. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.