Disclosure: VNKB.com at the time this article was published is hosted by Linode.
After almost 2 weeks of becoming under a siege of DDoS attacks. VPS provider Linode has managed to defend most of the attacks. Datacenters that were attacked by the DDoS included: Dallas, Singapore, Atlanta, London, Newark, Fremont, and Frankfurt.
The first signs of the DDoS attack began to surface on December 25, 2015 with the Dallas datacenter experiencing connectivity issues. The Linode Manager and Linode.com website were hit with the first wave of DDoS attack. The next day it was the Atlanta datacenter. On the same day the hosted DNS was also attacked and the Dallas and Newark datacenter were also hit. On December 27th the Atlanta & Newark datacenter were under attack again. The Fremont and London datacenter was also targeted this time around.
On December 28th the Dallas datacenter was also hit again and the main Linode website and Linode Manager was under attack again. More DDoS attacks began to hit Dallas, London, Atlanta, Frankfurt, & Fremont for the reminder of the month of December. Singapore and Frankfurt were also targeted on January 1st. Since January 1st the main Linode website and datacenters in Dallas, Atlanta have received continued DDoS attacks.
DNS service provided by Linode, the Linode Manager, the Linode.com website, and API were hit with DDoS during the attacks. Atlanta was one of the datacenters that were hit multiple times that resulted in the upstream provider limiting connectivity from parts of the world from connecting to migrate the attack.
Linode also took security precautions and had customers change their Linode Manager passwords:
Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.
A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.
This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.
Linode’s description of the attacks:
It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business. The following is a partial list of attacks we have received in no particular order:
– Multiple volumetric attacks simultaneously directed toward all of our authoritative nameservers, causing DNS hosting outages
– Multiple volumetric attacks simultaneously directed toward all of our public-facing websites, causing Linode Manager outages
– Layer 7 (“400 bad request”) attacks toward our web and application servers, causing Linode Manager outages
– Large volumetric attacks toward our colocation provider’s upstream interconnection points, overwhelming the router control planes and causing significant congestion/packet loss
– Large volumetric attacks toward Linode network infrastructure, overwhelming the router control planes and causing significant congestion/packet lossAll of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change.
As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. We are working extremely closely with all of our technical partners, including our network equipment vendors and our colocation providers, to prevent future attacks.
Once these attacks stop, we plan to share a complete technical explanation about what has been happening. Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency.
We would like to apologize for the lack of detail in some of our recent status-page updates. Please know that we are dedicating all resources from multiple departments to stopping these attacks. We acknowledge the amount of downtime we’ve been experiencing is completely unacceptable, and we appreciate the understanding and support we have received over the past several days. We will share more information as our investigation continues.
Alex Forster
Network Engineer at Linode