Sunday, March 05, 2006

Every Webmasters Worst Nightmare

Here I sit working on my website late at night, we're talking LATE at 3am, so I can test some new bot blocking technology in the middle of the night when the traffic is low so impact will be minimal if I screw up and knock everyone offline.

Ok, upload some code, bug, quick patch, bug, something weird going on so hop onto SSH and look at server and something is chewing up CPU like crazy and website isn't responding properly. Whew, it's an automated nightly update that only lasts a couple of minutes and everything is back to normal.

Test some code, find another minor bug, upload some new code, run the page, and it goes and goes and goes. What the heck, did I just blow something big time? Check SSH again, staggers a little and stops, then generates a socket error and closes.

Holy Fuck! Server Down!

So I click here and there in my browser, EVERYTHING is down!

YAY! I didn't crash the server, the cable modem is offline.

30 harrowing minutes later the damn cable modem comes back online and everything on the server is running just fine except I left a debugging message that's displaying on all the pages.

Remove that message, test it again, everything seems to be OK, time for bed.

Whew!

Bracing for nightmares tonight....

1 comment:

Jon said...

What's funny is how coincidental those events are with each other. You're making changes, then stuff quits working, but the obvious reason why it's probably not working isn't the reason -- it's the stupid Internet connection. That happens to me all of the time.