A few years ago DynDNS experienced a DNS outage that caused a massive disruption in the internet. It feels like forever ago when I had a fresher face (partly true) and more hair (sadly, not true). While I have learned to not worry about my age, what have organizations learned about their DNS since then? The problem is, not enough and we are seeing it again with this latest problem caused by an Akamai outage that brought down many large tier 1 websites including the likes of Playstation, Airbnb, Steam, and more.
This article on ZDNet did some analysis to show that even many of the impacted organizations from the 2016 outage haven’t done anything to fix this catastrophic failure point. In fact, across the internet the problem has only grown worse because of the concentration of the service in a few cloud providers.
Many wrote about how to help mitigate this problem with a secondary DNS backup, including ourselves. We have a very strong opinion about this because adding redundancy to a domain is super inexpensive considering that your domain runs everything. Don’t believe me? Here is a list of 20 applications that rely on your domain that took me about 2 minutes to come up with.
And it is a big deal. With our DNS Firewall users alone we logged 6 million fails during this recent outage and if they weren’t backed up then they didn’t resolve.
For instance, here is a Canadian police force that was returning NXDOMAN and on looking closer we see that they only use a single nameserver provider. If it is just a short-term web outage then maybe someone doesn’t find an address or phone number, but if it is also impacting applications then operations can be affected.
Full disclosure, Akamai is a great partner of ours for cybersecurity and also a supplier to our web properties. But we use both their DNS and our DNS services to ensure our properties are reachable. This means that our own website CIRA.ca and all our apps that need it did not fail despite being an Akamai customer. In short, we are extremely happy with their service, but we still add depth to our DNS.
On the malware and phishing front, CIRA Cybersecurity Services uses Akamai technology as part of our CIRA DNS Firewall and CIRA Canadian Shield services. Straight up – this recursive DNS technology is NOT impacted in any way by this authoritative DNS outage. If you are a customer of CIRA DNS Firewall or a free home user of CIRA Canadian Shield, your security is still up and running.
What has CIRA done to help our cybersecurity customers be resilient to outages:
- We have the CIRA Anycast DNS service that is used as a secondary backup to your primary DNS. If you aren’t a customer it is a recommended best practice to back up the authoritative DNS and so check it out.
- If you are a CIRA DNS Firewall customer, what you probably don’t know is that we configured these servers not use the TTL on the DNS record if the server is non responsive – at least for a short time. Rather than fail we will continue to deliver the older DNS record. What this means is that, for customers of these services, 99.9% of the internet DNS entries will still resolve. It is only very recent changes that may not. Given that DNS can take a full day to propagate around the world, this is <almost> a normal scenario. If the outage is short and since you should already be assuming your new records aren’t propagated instantly in every recursive resolver on the planet then waiting is normal.
The DNS is pretty awesome in how redundant and reliable it can be in the hands of full-time managers. Leveraging cloud service providers like CIRA can help to keep it up and running because if there is one thing we know about technology, there is always risk that needs to be mitigated.
Rob brings over 20 years of experience in the technology industry writing, presenting and blogging on subjects as varied as software development tools, silicon reverse engineering, cyber-security and the DNS. An avid product marketer who takes the time to speak to IT professionals with the information and details they need for their jobs.
