Forum Downtime - Power Outage at the webhost

**Fraser** · 2012-Jun-30, 05:02 PM

Sorry for the downtime last night, our webhost, Amazon EC2 suffered a massive power outage at its eastern storage facility. That took out Netflix, Instagram, Pinterest... and Universe Today/BAUT.

We rebuilt the server and got it running again as soon as we had power.

It was indeed, a bad night.
http://www.theverge.com/2012/6/30/31...tagram-go-down

**Jeff Root** · 2012-Jun-30, 05:42 PM

I noticed. I figured it was you doing some kind of
maintenance.

Can you explain briefly why you would need to rebuild
the server after a power outage? I'd think a reboot
would be completely automatic, and fairly routine, not
requiring any human intervention, and certainly not by
anyone not onsite.

Also, I just asked a couple of questions in the name-
change thread.

-- Jeff, in Minneapolis

**Van Rijn** · 2012-Jun-30, 05:51 PM

Presumably it's a regeneration of the server image. This is in a cloud environment, and apparently all the power was lost (I thought they were supposed to have redundant backups) so it shouldn't be routine.

**slang** · 2012-Jun-30, 06:20 PM

Originally Posted by Jeff Root

Can you explain briefly why you would need to rebuild
the server after a power outage? I'd think a reboot
would be completely automatic, and fairly routine, not
requiring any human intervention, and certainly not by
anyone not onsite.

For an ordinary reboot, where all processes are warned about the impending action, yes, it usually is a quite routine event. Things change when there is a severe powerfailure, with virtual servers and storage even more so. It is much more complex than the old days, when every server was a machine with a couple of harddrives.

**captain swoop** · 2012-Jun-30, 07:38 PM

Even the simple servers we set up for Doctors and Dental PRactices to store their patient records and Digital Xray Images have to have a consistency check on the Database if it stopped all standing.

**Fraser** · 2012-Jun-30, 07:42 PM

Normally, you're able to just reboot the server, but in this case the main server wouldn't restart. So we had to create a brand new server instance matching the old server. And then I couldn't attached the active drives to the new server as they were attached to the corrupted server and wouldn't detach. So we actually had to create a new server, pull the most recent backups from a snapshot and use that instead. We might have lost a few minutes. I think we take snapshots every 5 minutes.

So this event was very different from a normal service interruption, and there was a lot of hairpulling and fretting involved. I would think that Amazon would be more insulated from a major power outage, but I guess not.

**Ara Pacis** · 2012-Jun-30, 08:05 PM

Originally Posted by Fraser

So this event was very different from a normal service interruption, and there was a lot of hairpulling and fretting involved. I would think that Amazon would be more insulated from a major power outage, but I guess not.

Yeah, I thought that was the whole point of the cloud.

**Fraser** · 2012-Jun-30, 08:38 PM

You would think that "the Cloud" means that you can just run your programs in some nebulous scaling server infrastructure, but we're not there yet. You have to tell Amazon what size server you want to run, and that's that. The servers themselves are actually very fragile and run from a specific place. I really wish I could just pay a fee per pageview or something, but that's just not possible right now. A huge opportunity for Google, I think.

**Trebuchet** · 2012-Jun-30, 08:47 PM

Isn't this the second such instance in the last year or so? Or am I thinking of some other site I visit?

**Fraser** · 2012-Jun-30, 09:20 PM

It's the second instance. The previous one was bad, though. It lasted for a couple of days. This was only about 8 hours. Some day we'll have a real cloud.

**Swift** · 2012-Jun-30, 09:24 PM

Originally Posted by Fraser

It's the second instance. The previous one was bad, though. It lasted for a couple of days. This was only about 8 hours. Some day we'll have a real cloud.

And I always assumed astronomers hated clouds....

**slang** · 2012-Jun-30, 09:37 PM

Originally Posted by Fraser

You would think that "the Cloud" means that you can just run your programs in some nebulous scaling server infrastructure, but we're not there yet. You have to tell Amazon what size server you want to run, and that's that. The servers themselves are actually very fragile and run from a specific place.

Hmm.. the virtual environment that I'm currently having "my" servers put into is like that: specify server size, some storage, and the resulting server instance can be running on any of the host servers in "some nebulous scaling server infrastructure", in any of the datacenters. The hosting software itself decides where it runs, and it may move a running instance from one datacenter to another, and the attached storage may or may not move with it. It sounds real scary, but it does seem to work.

**Fraser** · 2012-Jun-30, 10:21 PM

@slang - It's the "specify a server size/storage size" that I have a problem with. I'd really only like to pay for what I use, and be able to scale up smoothly when there are spikes.

**Tobin Dax** · 2012-Jun-30, 10:41 PM

Originally Posted by Swift

And I always assumed astronomers hated clouds....

I was under the impression that real clouds were the problem, unless the storms were virtual, too.

**slang** · 2012-Jun-30, 10:44 PM

Originally Posted by Fraser

@slang - It's the "specify a server size/storage size" that I have a problem with. I'd really only like to pay for what I use, and be able to scale up smoothly when there are spikes.

Ah, I see what you mean now. I'm sure even that can be done, within limits, but it will probably be expensive. You'd have to balance peak usage against lower usage on other boxes.. You've got me curious now, if there's an opportunity I'll ask our virtualisation guys, or the storage people.

**Jeff Root** · 2012-Jun-30, 11:44 PM

Originally Posted by Fraser

It's the "specify a server size/storage size" that I have a
problem with. I'd really only like to pay for what I use, and
be able to scale up smoothly when there are spikes.

I don't understand. There are spikes in numbers of users
trying to connect, but it sounds like you are talking about
spikes in storage size, which I wouldn't think would ever
happen except when you are doing something like you just
did and briefly have multiple copies of everything stored.

-- Jeff, in Minneapolis

**captain swoop** · 2012-Jun-30, 11:52 PM

More users means more processor power, more memory being used etc.

**slang** · 2012-Jul-01, 12:19 AM

Originally Posted by captain swoop

More users means more processor power, more memory being used etc.

For us, yes, probably mainly those. And bandwidth. Other types of clients buying hosting might also have a more dynamic need for storage. And if you can't dynamically adjust your usage to spikes, you must acquire enough resources to deal with spikes, even if 75% of those resources are idle 90% of the time.

**Jeff Root** · 2012-Jul-01, 02:01 AM

I assumed that when you go from 10 users doing stuff
simultaneously to 100 users, the number of connections
needed would be multiplied by 10, but the disk storage
needed would only increase by maybe 1%.

-- Jeff, in Minneapolis

**Gillianren** · 2012-Jul-01, 04:13 AM

Originally Posted by Trebuchet

Isn't this the second such instance in the last year or so? Or am I thinking of some other site I visit?

The last one was a year ago Easter. By coincidence, I was at a con that weekend and didn't know about the severity until I got home Sunday. This has not stopped me from using it as an example of why I'm not a fan of getting rid of physical copies of things.

**Solfe** · 2012-Jul-01, 04:26 AM

Can't you ask for a server far, far away from Netflix and the other companies that attract bad luck and trouble?

(I love Netflix, but they are the first people mentioned when the cloud goes down.)

**Moose** · 2012-Jul-01, 05:26 PM

Originally Posted by Solfe

Can't you ask for a server far, far away from Netflix and the other companies that attract bad luck and trouble?

The power failure that caused this still has a few million people out of power, by the last news reports. Maybe let's all keep this in a little perspective, huh? BAUT is BAUT, but it's still just an internet message board.

**Solfe** · 2012-Jul-03, 01:18 AM

<Tongue>They won't know we are back up for at least a week. </cheek>

**ToSeek** · 2012-Jul-03, 02:09 PM

Originally Posted by Moose

The power failure that caused this still has a few million people out of power, by the last news reports. Maybe let's all keep this in a little perspective, huh? BAUT is BAUT, but it's still just an internet message board.

I'm still waiting for the server for my primary email account to come back up - it's in Rockville, MD, northwest of DC, which is right about where the worst of the storm hit, at least in this area.

Meanwhile, my parents - who live 250 miles to the southwest - are still waiting for their power to come back on after going through the same storm.

**Gillianren** · 2012-Jul-03, 07:17 PM

Well, we know whose problems are more important . . . .

**ToSeek** · 2012-Jul-04, 04:38 PM

Originally Posted by Gillianren

Well, we know whose problems are more important . . . .

Well, my parents found out yesterday morning their power was back and promptly emailed me to let me know - at the account that wasn't working. So it took me a while to find out.

**Gillianren** · 2012-Jul-04, 06:42 PM

Okay, that's funny. And very much a modern problem.

Thread: Forum Downtime - Power Outage at the webhost

Thread Tools

Display

Forum Downtime - Power Outage at the webhost

Similar Threads

BAUT Outage

"Star" power? (Fusion power reactor experiment)

Power outage in Eastern US

Posting Permissions