Ever feel this was a good solution? by aa o’carroll

There are few things I hate more than looking at my binging phone and seeing an alert from Pingdom. We use this service to constantly monitor our servers and let us know immediately within 5-minutes when something goes wrong. This month we’ve gotten 5 alerts, compared to our normal zero.

For some things, zero is really the best number.

The issues have been 3 fold (which is not zero).

1. The forums got hacked. One of our long term, very skilled and totally trusted, forum administers had his password compromised. This could have happened to anyone. Unfortunately, as a result of this, our forums were brutally hacked and a backdoor was added (along with all sorts of other nasty code). In the process of ousting the evil doers, we upgraded everything to the latest version… which is how we learned the VBulliton license we have doesn’t support the most up-to-date version of PHP. This led to a massive downgrading of the upgrades… which was remarkably harder to do because downgrading is not a built-in function 🙁
2. Upgrading things broke our WordPress themes. One of our WordPress plugins underwent an automated upgrade that caused our theme to no longer load any of our content pages. We don’t know which plugin it was, we’re not going to point fingers, we’re just going to upgrade our themes. This is going to be a multistep process, and things are going to be a little ugly for a bit, while we develop things on a test server that is running the most recent version of PHP (more on that below).
3. And then there is the denial of service attack. Yeah, I’m not going to go into that. It’s over now.

The fix is straightforward but tedious: We are going to move the forums to their own server, where they can have their preferred version of PHP at forum.cosmoquest.org domain. Cory has been working his fingers off making this happen for the past week (send coffee!), and the end is in sight, but he’s triple checking everything as he goes, to make sure the hacker code is removed. There will likely be a few things that we find broken when we spin things back up, but getting the forum healthy is our highest priority. Once it’s back up, we’ll upgrade this server to the latest PHP version.  While Cory is doing that, I’m working on slowly making WordPress not look so ugly. I currently have the main blogs usable, and I’m going to focus on making 365 Days of Astronomy awesome, and then we’ll have our newest graphics and user experience grad students work with us to make the rest of the site awesome (it’s their first week of classes).

It’s going to get better. It’s just slow going if we want to do things right.

We deeply appreciate all your patience.

The forum is back up, and here is a thread on this situation.