How to Recover in the Aftermath of a DDoS Attack

You were hit by a DDoS attack. Now it has subsided. That’s the good news. But you probably have a big mess to clean up.

While much of our research has been focused on what happens during a DDoS attack, we haven’t written much about what happens afterward. And while many of you will breathe a sigh of relief when the onslaught stops, those who need to restore their site or service have a lot of work left to do.

I thought about writing this post after reading a recent study from the Ponemon Institute and Emerson Network Power. It revealed that 22 percent of all data center downtime was caused by DDoS attacks. A staggering number, up from 18 percent last year. DDoS attacks are on par with UPS system failures (25%) and human error (22%), and well above cooling (11%), weather (10%), and equipment failure (4%).

Thinking of DDoS as just one of the many ways a data center goes down got me thinking about the aftermath. Here are four time-consuming tasks that will keep your IT staff busy and your customers waiting. Understanding these and having a recovery plan in place will help set proper expectations (Are we up yet?) and help your IT team more efficient (No one asked me to do that).

Reestablish Your BGP Connections — Odds are that if you’re hit with a Layer 3 or 4 DDoS attack, connections with your transit providers and peering partners will be dropped. The BGP protocol uses what are called keepalive messages to let a peering partner know that a route is still up. Every provider will configure differently, but to illustrate, by default these are sent every 60 seconds. Failing to send three in a row means that a route will be dropped by your providers and partners in only a minute and a half. You will be considered down and routes from you will be flushed. Again, exactly how long depends on your providers and their configurations, but that only highlights the uncertainty of how long it will take to recover.

Once the attack is over, you will need to announce your network again. Transit providers will likely accept your connection request right away (typically in a few minutes). Peering partners may take longer. Meaning that the peering connection that cost you the least will not be available. This will increase the overall cost of the DDoS attack as you will be on more expensive routes for the first hour or so after you are back up.

Check/Restart Firewalls and Other Appliances — As you bring network devices back online, another risk you run is that the sudden surge in pent up traffic will cause a flood — like a secondary attack — as those connections attempt to reestablish themselves. Bring up equipment in the wrong order and you could potentially be setting yourself up to come down again as the load will appear all at once. The only way to do it is to know your application and have a plan for an orderly restoration.

Get Unblocked by Your ISP — Many, if not all, ISPs will cut off and not offer connectivity to customers who are hit by DDoS and consume bandwidth needed by other customers. We sometimes call this the “noisy neighbor problem.” The DDoS attack on your site is costing them business and what you pay them may not be worth it.

So, you may need to convince your provider to let you back on its network, and they may ask you to prove to them that it won’t happen again. If you suffered a volumetric attack, you’ll need to demonstrate some kind of DDoS attack mitigation. Otherwise, you’ll be shopping for a new ISP, and negotiating a contract and reconfiguring an entire network will take several days at least.

Application Recovery — When your network is back online, your customers may try to connect all at once. They may have been trying to connect for the time you were down, and that pent up demand coming all at once could be a problem, potentially creating an application layer DDoS effect with thousands of sessions reconnecting.

To prevent this, devise a strategy for gradually reconnecting customer sessions. There are several ways to do this, and it may depend on your business. You could, for example, intelligently route to different data centers based on IP address range or geography. Or, you could also simply meter the number of connections that can be established.

There may be other things to clean up too. If you use a cloud service like AWS, you may find yourself with a large bill to pay. This is because application layer DDoS attacks that use lots of CPU can trigger additional instances to be spun up. You may need to work with Amazon on settling your bill. It’s possible you may need to clean up or purge your logs.

We encourage you to create a plan to deal with these specific issues, as well as how you communicate within your organization. To help, we’ve created the DDoS Response Playbook for Network Operations.

The hardest conversations to have are the ones with your customers. They have an expectation that you have this covered in your operating plan. When they find out that you haven’t and now have a new expense, that could trigger a need to discount or offer credits to your customers to keep them.

In short, the calm after the storm may not be as restful as you hope. The cost of a DDoS attack often extends beyond the incident.