Yesterday (Tuesday, Feb. 17) we had a nasty DDoS attack on the server ams117. Instead of a standard ‘blunt’ attack, we faced a very targeted scenario that our standard security did not have an immediate answer to.
Everything has since returned to normal, and the server is fully operational again. We apologize for any inconvenience you may have experienced from this incident.
I'd like to take you through exactly what happened, how we resolved it and what we learned from it to avoid this next time.
Why the attack could not be blocked quickly
Normally, our network security immediately catches large spikes in traffic that normally occur during a DDoS attack. This attack was just very cleverly done, using different IP addresses each time. It was small enough not to be filtered out by network security, and large enough to bring the server to its knees.
The attackers used a huge pool of IP addresses (eventually we had to block 5500 of them). They only allowed a handful of IPs to connect at a time each time. As soon as we blocked those, the attackers immediately switched to the next set. Because the IP addresses came from all over the world and were not yet on blacklists, we could not simply block an entire country or a known botnet.
This is very striking; since IPv4 addresses are relatively scarce. This also indicates that it must have been an expensive attack for the attackers.
Attrition of IPv4
The attack focused on keeping the server's network interface occupied. Countless requests per second were coming in, leaving the connection ‘open’. This simply ran out of capacity for new IPv4 connections; the server was ‘full’ in terms of connections, preventing legitimate visitors from getting through.
In addition to IPv4, the attackers were also targeting us from IPv6 addresses. Since there are an almost infinite number of IP addresses available there, manual blocking was impossible. We therefore decided to temporarily disable IPv6 completely to restore peace of mind.
How we solved it
To get things back online, we took a few necessary steps;
- Blockages: We blocked all IPv4 addresses that visited the specific site under attack that day. In the end, that was about 5,500 unique IP addresses.
- Server tweaks: We have adjusted the server settings so that more concurrent connections are allowed to be open and ‘hanging’ connections are broken much faster.
- Domain isolation: The target of the attack was temporarily removed from the server. Even after that, the attackers continued to bombard the server's direct IP address (they then ended up with an ‘account does not exist’ page), but the pressure on the Web server and PHP thankfully eased as a result. The domain has since been brought back online via CloudFlare.
How to move forward?
This incident shows that we must and can prepare better for such small but targeted attacks. We are going to further evaluate this incident internally, and list all the ‘lessons learned’ so that we can act more quickly and effectively next time.
In addition to a faster response to recover from an attack, we would prefer to be able to prevent such situations from now on. We will work on that too to devise and implement measures. Here you can think of:
- Wider deployment of a CDN network, which can block malicious traffic “on the edge.”.
- Monitoring on domain names that suddenly get many requests in a short period of time, so that we can take quick and targeted action even before a DDoS attack starts.
- A kind of ‘under attack’ mode, where we can quickly make certain configuration adjustments at the server level to give normal traffic as much space as possible.
