Post-mortem: The Beta Bug

Around 11AM today, I uploaded our first beta build. There was just one problem: it didn’t work.

This was something I could not understand. Everything worked perfectly in our development environment, so what could the problem possibly be? All of the error messages and sanity checks in our code were for naught: no errors were being displayed. Even the Flash Debug Player was of no help: there were no NPEs, no errors or stack traces of any kind. So there was no indication of what was going wrong, and yet something was definitely going wrong: attempting to create a new user or login caused the loading spinner to spin indefinitely.

I didn’t have time right then to figure out what the problem was, so I fired off some e-mails and briefly discussed the issue with Matt over IM. The only things we knew were that beta1 did not work, but alpha6 worked perfectly. Around 6PM I parked myself behind a monitor and started looking for what I had nicknamed “The Beta Bug”. I started by performing a binary search: since alpha6 worked, some change between then and now obviously had to be the culprit. So I started testing revisions to try to determine where the problem was introduced. At some point in this process, I left the browser window open while preparing a new revision and discovered much to my amazement that it worked – it was just really, really, really slow. As in, slower than dial-up slow.

So then I wondered if this was affecting other revisions. I tried alpha6 again, and lo and behold, it was now experiencing a speed problem as well. So I tried beta1 again, but that still seemed to hang indefinitely. We never had any speed issues before, so I figured these things had to be related somehow. Since there were no error messages to go on, I decided to use Wireshark to look at the raw traffic that was being sent through the tubes. One capture run later and I’d found the problem: there was a PHP warning before the XML schema. In the past, this rightly caused Flex to complain. Loudly. Apparently, somewhere along the way, this became a situation in which Flex just fails silently. Great job, guys.

With the text of the problem in hand, I easily fixed it in just a few minutes. I uploaded the new PHP script and beta1 started working, albeit very slowly, like the other builds. I informed Matt of my success a few minutes later, and when he tried it, he told me it was much faster. So I tried it again, and the situation seemed to be pretty much back to normal. Exactly what happened there is anyone’s guess. Maybe the PHP errors were being logged somewhere and it was slowing down the works?

In any case, while we’re pretty close, we’re not done yet. The beta1 build has five known bugs, and Matt is still restyling stuff. My hope is that we’ll have the bugs fixed by Monday or Tuesday, at which point we’ll upload a second beta. That build will become the official version 2.0 if no other bugs are discovered. At least, that’s the plan, but we all know how my plans usually go…

Advertisement