A few years ago DynDNS experienced a DNS outage that caused a massive disruption in the internet. It feels like forever ago when I had a fresher face (partly true) and more hair (sadly, not true). While I have learned to not worry about my age, what have organizations learned about their DNS since then? The problem is, not enough and we are seeing it again with this latest problem caused by an Akamai outage that brought down many large tier 1 websites including the likes of Playstation, Airbnb, Steam, and more.
This article on ZDNet did some analysis to show that even many of the impacted organizations from the 2016 outage haven’t done anything to fix this catastrophic failure point. In fact, across the internet the problem has only grown worse because of the concentration of the service in a few cloud providers.
Many wrote about how to help mitigate this problem with a secondary DNS backup, including ourselves. We have a very strong opinion about this because adding redundancy to a domain is super inexpensive considering that your domain runs everything. Don’t believe me? Here is a list of 20 applications that rely on your domain that took me about 2 minutes to come up with.
And it is a big deal. With our DNS Firewall users alone we logged 6 million fails during this recent outage and if they weren’t backed up then they didn’t resolve.
For instance, here is a Canadian police force that was returning NXDOMAN and on looking closer we see that they only use a single nameserver provider. If it is just a short-term web outage then maybe someone doesn’t find an address or phone number, but if it is also impacting applications then operations can be affected.