That moment our website went down during a flash sale in Chicago

Our site crashed at 10am during a 50% off promotion we'd been planning for 3 months, lost about $4k in the first 12 minutes before we realized it was a bad caching plugin. I had to manually refund 30 orders because the payment processor kept charging people even after it errored out, has anyone else dealt with a plugin update blowing up at the worst possible time?

2 comments

2 Comments

the_sandra12d ago

I mean, "it was a bad caching plugin" - you're really throwing that plugin under the bus. Maybe it's just me but that update probably fixed a security hole or something important and you just didn't test it right. If you're running a flash sale that's been planned for 3 months, you should have had a staging environment to check the update before pushing it live. I dunno, seems like the plugin didn't blow up at the worst possible time, your deployment process did.

reese_patel12d ago

@the_sandra I'm with you on the staging environment point, but what if the plugin's changelog didn't flag that as a breaking change? Not everyone has time to regression test every single plugin update, especially on a tight timeline.