"Server too busy" message

[clint]

I get this message a lot around mid-morning CET
(Central European Time - that should be around 1-2 AM on the West Coast or 4-5 AM East Coast)

Is this due to maintenance routines on the BAUT servers?
(there shouldn't be much traffic at those hours, right?)

[Moose]

Yes. The server is getting backed up at that time. About 5:15am EST, and it takes about a half hour to get everything finished.

[01101001]

Watch my bautbot log episodes of slow serving (when I remember to keep it running).

That will give you an idea of when and how long the daily lassitude is, and also answer the occasional question: BAUT was slow for me and it wasn't the usual maintenance time; I wonder if anyone else saw that slowness, too, or if it was more likely on my end?

[slang]

The Server too busy page takes less than 10 seconds to load, usually.

[01101001]

The Server too busy page takes less than 10 seconds to load, usually.

Not in my experience, but I haven't seen that message for a long time. When I did in the past, I recall it was accompanied by very long stretches of lack of response, on the order of minutes.

I can dial down the threshhold for my robot's reporting though, or look for that specific too-busy response and report those instances slightly differently. If I look for those words, "server too busy" (amongst a small amount of screen data, so a full main page with one topic entitled "server too busy" won't give a false positive), will that be good? Anyone ever captured the wording that comes quickly and indicates lack of service?

[slang]

Can't say from memory... I'll try to grab the page tomorrow morning.

[01101001]

Probably:

The server is too busy at the moment. Please try again later.

If I believe the comment I found in my bautbot script. Apparently I was thinking of doing it once. Good thing I write messages to my future self.

I'll go with that text and see what happens.

Edit: Message to my past self and anyone currently reading. I've got a private copy running with the extra "server is too busy" check. I restarted the public version with a 3-second threshold just for fun. That's probably too sensitive, but it will show slight hesitance from the server.

[DrRocket]

Probably:



Good thing I write messages to my future self.

.

It is also comforting that you don't seem to write messages to your past self.

[Moose]

Generally what happens is that there'll be ten-twenty minutes where the server simply doesn't respond, and a last ten minutes where it'll serve up the "too busy" page.

[clint]

Ok, that explains everything, thank you

[slang]

Nice. Ready to save the page and now the forum is quick and responsive.

[HenrikOlsen]

Just had a slow server situation again.

It may be just my impression but it seems to have gotten worse recently, with actual timeouts on connection rather than just slow responses.

[01101001]

Edit: Message to my past self and anyone currently reading. I've got a private copy running with the extra "server is too busy" check.

Edit: I implemented it. Over the last half-day the bautbot saw:

bautbot: Beginning at Sun Nov 15 01:07:56 UTC 2009
Watching page http://www.bautforum.com
Reporting instances of "The server is too busy at the moment."
Reporting instances of rendering taking more than 7 seconds
Reporting each new day
Sleeping 2 minutes after a report, otherwise 12

Sun Nov 15 05:07:45 UTC 2009 45 seconds
Sun Nov 15 10:21:28 UTC 2009 748 seconds
Sun Nov 15 10:26:23 UTC 2009 The server is too busy at the moment.
Sun Nov 15 10:28:00 UTC 2009 The server is too busy at the moment.
Sun Nov 15 10:30:01 UTC 2009 The server is too busy at the moment.

I also added a daily report, not encountered in this run yet, of a day change, so one can tell the difference between perfect server performance (no reports but the daily one) and a dead robot (no reports).

[mugaliens]

The Server too busy page takes less than 10 seconds to load, usually.

That checks on this end.

There are two servers involved: The web server, and the database server. It's the database containing all of BAUT's messages, forums, threads, PMs, etc. that's being backed up.

The web server remains free to do it's job, one of which is to spit out that message if it doesn't receive a response from the database server within a realtively short time frame (a few seconds, and my guess is between 5 and 7).

The database won't respond during backup because it's locked before backup begins. This is to protect the integrity of the data. Having a thread updated in the middle of a copy operation, for example, would corrupt the thread.

Once the backup operation is completed, the system unlocks the database, and you're back in business.

As a matter of course, anytime I'm making a post between 2:45 am and 3:45 am (or until after the backup is complete) MST, I simply copy my post. That way if I receive the timeout, I simply use the backpage button, paste it, and hit Submit Reply after 3:45 am.

[slang]

That checks on this end.

Yep, and 01101001's bot timings from this morning/last night match what I observed. My less than 10 seconds turns out to be 3-5 seconds, today at least, in other words: normal delays, perhaps slightly slow.

The web server remains free to do it's job

(unless cpu or other load on the server prevents it)

The database won't respond during backup because it's locked before backup begins. This is to protect the integrity of the data. Having a thread updated in the middle of a copy operation, for example, would corrupt the thread.

This is probably true, it depends on which database server is used and how it's set up. But, for completeness sake, there are ways around the lockout for data integrity, I've worked on such systems. It involves some things like keeping the database read-only, and keeping all mutations in a temporary store. After completing the backup the mutations are "rolled forward" into the database. At all times is the integrity of the database itself guaranteed, but it's not necessarily a complete set of data (not counting the temporary store). Such setups can be complicated, cost performance, and are not typically used for web sites.

[HenrikOlsen]

One alternative is if the file system allows atomic snapshots, then a file system dump of the snapshot can get a consistent backup that can be recovered at the point where the backup was made.

[mugaliens]

One alternative is if the file system allows atomic snapshots, then a file system dump of the snapshot can get a consistent backup that can be recovered at the point where the backup was made.

By date/time stamp? I'm familiar with that term as it's used in memory, but never with disc-based databases.

vBulletin has the option of locking the database or not. I've always preferred to lock it because in the earlier days, not locking it occasionally meant a corrupted message forum, which really ticks people off!

[HenrikOlsen]

By date/time stamp? I'm familiar with that term as it's used in memory, but never with disc-based databases.

The files backed up will be a consistent image of how they were at the time the snapshot was taken
as if the database was killed uncleanly at the time of the backup.
so a restore from backup of the database files
Not all file systems have the capability (UFS1/2 on FreeBSD and ZFS are the ones I know that do) and the details vary a bit between those that do, but essentially the state of the file system at a specific point in time can be saved and subsequently mounted(readonly) as if it was another file system.
Backing up this instead of the original means all the database files are in a consistent state, as if the db process had been killed at the time the snapshot was taken, so restoring the files and starting the database and letting it go through the usual recovery of unrolling uncommitted transactions will get it back to the state it was when the snapshot was taken.
Normally the snapshot is deleted after backing it up so the old version of any subsequently changed files can be forgotten.

[mugaliens]

The files backed up will be a consistent image of how they were at the time the snapshot was taken
as if the database was killed uncleanly at the time of the backup.
so a restore from backup of the database files
Not all file systems have the capability (UFS1/2 on FreeBSD and ZFS are the ones I know that do) and the details vary a bit between those that do, but essentially the state of the file system at a specific point in time can be saved and subsequently mounted(readonly) as if it was another file system.
Backing up this instead of the original means all the database files are in a consistent state, as if the db process had been killed at the time the snapshot was taken, so restoring the files and starting the database and letting it go through the usual recovery of unrolling uncommitted transactions will get it back to the state it was when the snapshot was taken.
Normally the snapshot is deleted after backing it up so the old version of any subsequently changed files can be forgotten.

Thanks for taking the time to pen that out, Henrik!

While reviewing ZFS, I came across Snapshop (computer storage), which itself includes a link to Atomicity (database systems), for more details beyond those you provided.

Thanks again!

[Glom]

Been getting this message a lot lately. Looks like the board has grown too big for its own good.

[Moose]

Glom, you've been getting it because you're attempting to post/read during the half hour the board is performing its backups and automated scripts. You won't see that message any other time of the day.

[Glom]

Glom, you've been getting it because you're attempting to post/read during the half hour the board is performing its backups and automated scripts. You won't see that message any other time of the day.

See it's too big. If this forum was like the forum at NIOF, then all that would take half a second.

[mugaliens]

So... We... start over?

[Glom]

How often do these backups happen? This problem is happening a lot. And the perverse thing is I'm actually here for legitimate work reasons this time.

[slang]

Every day. Same time. 10:00am UTC. About 30, 40 minutes. Check bautbot's output.

[Glom]

Every day. Same time. 10:00am UTC. About 30, 40 minutes. Check bautbot's output.

Well that's consistent at least.

[01101001]

bautbot

Sun Jan 3 00:04:01 UTC 2010 New day
Sun Jan 3 10:17:45 UTC 2010 465 seconds
Sun Jan 3 10:18:29 UTC 2010 29 seconds
Mon Jan 4 00:04:01 UTC 2010 New day
Mon Jan 4 09:51:22 UTC 2010 382 seconds
Mon Jan 4 09:55:09 UTC 2010 189 seconds
Mon Jan 4 09:59:10 UTC 2010 190 seconds
Mon Jan 4 10:12:59 UTC 2010 779 seconds
Mon Jan 4 10:14:17 UTC 2010 77 seconds
Mon Jan 4 10:34:46 UTC 2010 1186 seconds
Mon Jan 4 10:37:20 UTC 2010 140 seconds
Mon Jan 4 10:55:47 UTC 2010 767 seconds
Mon Jan 4 11:08:43 UTC 2010 763 seconds
Mon Jan 4 11:09:00 UTC 2010 The server is too busy at the moment.
Tue Jan 5 00:00:01 UTC 2010 New day
Tue Jan 5 01:42:07 UTC 2010 367 seconds
Tue Jan 5 01:43:49 UTC 2010 49 seconds
Tue Jan 5 01:44:01 UTC 2010 The server is too busy at the moment.
Tue Jan 5 01:46:14 UTC 2010 74 seconds
Tue Jan 5 01:52:01 UTC 2010 The server is too busy at the moment.
Tue Jan 5 01:53:00 UTC 2010 The server is too busy at the moment.
Tue Jan 5 01:54:01 UTC 2010 The server is too busy at the moment.
Tue Jan 5 10:12:12 UTC 2010 132 seconds

January 3 was normal.

On January 4, activity started about 20 minutes earlier than normal and ran about 30 minutes longer. Could it have been an annual extra-special backup?

January 5 had 10 minutes slow service about 8 hours before maintenance time, and then an extremely short slowdown at maintenance time (circa 1015 UTC).

What will January 6 bring? A return to normalcy? Maybe not. It already did the unusual 10 minutes of slow service about 8 hours before maintenance time.

Wed Jan 6 02:12:09 UTC 2010 189 seconds
Wed Jan 6 02:13:05 UTC 2010 5 seconds
Wed Jan 6 02:14:01 UTC 2010 The server is too busy at the moment.
Wed Jan 6 02:17:59 UTC 2010 179 seconds
Wed Jan 6 02:18:01 UTC 2010 The server is too busy at the moment.
Wed Jan 6 02:19:01 UTC 2010 The server is too busy at the moment.
Wed Jan 6 02:20:00 UTC 2010 The server is too busy at the moment.

Curious.

Edit: The rest of January 6 was normal: nearly 20 minutes of unresponsiveness starting about 1010 UTC.

Wed Jan 6 10:20:38 UTC 2010 578 seconds
Wed Jan 6 10:22:15 UTC 2010 75 seconds
Wed Jan 6 10:28:01 UTC 2010 The server is too busy at the moment.

[Fazor]

Heh, I was confused there for a minute (more than usual, that is). I was looking at the "Jan 6 10:XX:XX" times, and your post's timestamp, and going "How did he come up with that?" . . . I missed the little line that said "Edit: . . . " heh.

Yeah, I couldn't get around BAUT last night before bed, which would have been around those first group of times you posted. Oh well, not that I had anything interesting to say last night anyway.

[Gillianren]

I've been having trouble around maybe six PM PST.

[01101001]

I've been having trouble around maybe six PM PST.

The last two days have been that way for bautbot. I wonder what's going on, and if we'll see it again this day, January 7, circa 0200 UTC (about 2 hours from now). Anticipation. Maybe dread. We'll see.

Edit, 3 hours later: It's quiet. I mean, it's noisy and not quiet. I mean, there's no circa-0200 slowdown today. It's just like BAUT 2009.

Edit: The January 7 maintenance slowness was right on schedule, about 30 minutes long, starting about 1010 UTC. All normal.