Attack Postmortem

As you may have noticed, our site went down for a few hours today.

This post will explain why, and some of the things we're done to correct the problem.

First, what happened?

• Between 5am and 8am today, CloudFlare (our DDOS mitigation service) detected a surge in 'threat' against our site. "Threat" in this context means connections from what appear to be unfriendly bots. In this case, the bots originated from Canada. CloudFlare handled this by itself.
• Between 10-11am, we started getting 10x our normal traffic, 90% of which was made up of bots flooding us with requests.
• These requests were specifically designed to bypass caching, and hit a script on our end that created database logs of requests. By sending hundreds of thousands of requests to this script in a short space of time, they filled up a portion of our database with junk data, which caused our site to stop working temporarily.
• Once the attack stopped, automatic cleanup code successfully removed the junk data and the site started working again.
• The second attack appears to have originated from an IP in Poland.

So, in summary:

• Someone attacked us multiple times, trying different techniques until they succeeded in bringing us down by targeting a flaw in the vbulletin software that we use.

OK, what are we doing about it?

• First, we've upgraded our core software (vbulletin) to the most recent branch version, applying a bunch of fixes to make attacks like these against our forum software more difficult.
• Second, we've raised the security level on CloudFlare, our DDOS protection service, to make it screen out potential attacks like these more aggressively.
• Third, we've started a review process to assess what further actions we could take.

From your point of view, the site is back up, and should remain that way.
We apologize for the inconvenience.

Attack Postmortem