A side note on service availability

Jul 26, 2019

About three weeks ago, there was a span of a week when I experienced more service outages than usual from multiple providers. I’ll try to list all of the events:

  • Facebook, Instagram, and WhatsApp had an outage on July 3rd: Mashable.
  • Slack had a seemingly bigger outage, although I did not experience as much of it as some people reported: Slack outage.
  • The Cloudflare Web Application Firewall (WAF) had an issue that resulted in 502 errors. As it turned out, a seemingly innocent regular expression was resource-hungry: Cloudflare WAF outage blog. It looks like regular expressions are something to watch out for, since a bad one took down Stack Overflow as well back in 2016: Stackoverflow regex down 2016.
  • Cloudflare’s situation is really tough because they mitigate DDoS (Distributed Denial of Service) attacks on a daily basis. Since protecting against DDoS attacks is their core service, their code needs to scale exceptionally well. So I’m not surprised at this Cloudflare outage map.
  • Shortly after and during the Cloudflare issues, I also received reports that certain sub-services of Azure were having problems. When we look at maps like this, we need to keep in mind that these are compound maps. The one I cite is the July 03, 2019 07:04 UTC, WARN, about 1 hour, Information Diagnostic logs, Autoscale, Classic Alerts (v2).
  • Linode is very good at communicating scheduled maintenance, and most of the time it doesn’t result in any outages. However, the biggest box I have (16 CPU, 768MB+ SSD) experiences problems sometimes. We had a surprise restart where the production site was down in the middle of the day for 1.5–2 hours. The VM host had some physical problems, and the migration to another host took that much time. We also had a surprise restart two weeks before that one, but that was overnight.

The bottom line is: service outages happen. It’s best if the architecture of the system is redundant enough to survive these. Sometimes that is not possible, though.

Comments loading...