Okay, in more detail, here's the inside scoop on why all of the Escape Artists sites were bouncing up and down (mostly down) between Thursday night and about 5 AM this morning:
* For the last year, we've been on a VPS (virtual private server) hosting plan for our Web sites. If you don't know what this means, it's not that important; the upshot is supposed to be guaranteed resources and much more power than you get with one of the cheesy $8/month shared hosting sites like Dreamhost (i.e., where we were before that.)
* I had pre-paid for a year on that plan, which ended on September 13. For various reasons, I wasn't too happy with the place where the VPS was hosted. They had odd little technical limitations which I thought were making the server unstable, and although it was a good deal at the time, there've been better deals since then. So I decided to move over to a newer, more professional VPS provider (
http://slicehost.com) and get more resources for less money.
* As a test, a couple of weeks ago I moved just the Escape Pod site over to Slicehost, leaving PodCastle, Pseudopod and the forums alone. Things went well with that. We had one or two crashes over the two weeks, which made me start to think "Huh, maybe the VPS provider isn't the problem," but it was still better than before.
* Thinking things were good, on the night of September 11 I moved PodCastle, Pseudopod and the forums over to our Slicehost server.
* All of the sites went straight to hell.
* I mean crashes every few hours, no matter what I did to free up memory and keep things smooth. I have a monitoring service that pings my cell phone when a site goes down, and the damn thing wouldn't shut up.
* So I spent too much time Friday trying to fix things. And then way too much time Saturday. It was seriously interfering with my emotional state. Worst day I've had in months. Mostly because I considered myself pretty competent at this stuff, and I felt I was being proven wrong. (Most people don't have these problems!) And if you think that's bad, try crawling server logs for PHP fatal errors and tracking the code down while a three-year-old is crawling all over you and wanting his fair share of attention.
* I did find some stuff that was killing things. Some Wordpress plugins that were broken (notably Podpress). I actually found myself fixing other people's code, wondering why it was only now screwing me up so much. But there's still something mysterious somewhere in PHP that's causing infinite hangs. It's like the dark energy in the universe that can't be detected except by its gravitational effect. Only for "gravitational effect," substitute "locking up the Web server and making Steve feel rage."
* But whatever. Having taken it as a law of the universe that PHP sucks (gravity again!), I chose to work around it. Uninstalling the rough slouching memory beast that is Apache, I installed Nginx and custom-configured a pool of PHP FastCGI processes with failover balancing. Again, if you don't know what that means, it isn't important. Think of it as, "I took apart my car's engine, looked at all the pieces on the floor, threw them all away and replaced it with a fusion reactor and some pine cones."
* This still didn't solve the crashing problems. But I installed a shiny new monitoring program which will watch for failed processes every few seconds and restart them. I also wrote some code for that program to retrieve Web pages from the site frequently and fixes things if *that* fails. Amusingly, the monitoring program is called "God." I thought this was a very clever name. Until I tried to Google for information about it. Then I found it annoying.
* And that's where we are now. Moving to a better host with more resources has, for mysterious reasons, made our Web sites unstable. I haven't fixed it, but it's monkey-patched to keep working anyway. On the upside, response times should be faster.
* I have also come to really, really hate PHP and the PodPress plug-in. I feel so strongly about this now that I am getting ready to push up one of the projects I've had in the back of my mind -- to write a simple, but elegant, podcasting platform in Rails that'll do the things I need but won't piss me off. I'll probably need to make this a small team project or it'll never get done. If anyone wants to step forward and volunteer, feel free, but I'm not asking for that right now. (I will later.) And yeah, I know I have no time to do this. I may do it anyway. Because I am angry, and anger can sometimes drive good software development.
* And now I will sleep.
