:: 26-May-2000 02:38 (Friday) ::

Well, there’s good news and bad news. }:8)

Statsbox once again seems to be stable running on one CPU. The bad thing
is that it’s pretty damn slow with only one CPU… statsrun is taking 6-8
hours.

The other bad news is that the CPU with the screwy heat sink is apparently
fried… the box won’t even POST with it in.

So, stats will probably be a bit slow for awhile until we can take care
of the CPU situation. But they should be UP at least. }:8)

Thanks to everyone for their understanding.

Moo!

:: 25-May-2000 06:08 (Thursday) ::

Well, I don’t want to get anyone’s hopes up, but we **MAY** have found
the problem with statsbox. Seems one of the CPUs had a loose fan. We’ve
got that CPU out of the box right now, and I’m doing a statsrun. We’ll
see how well the box holds up. Don’t hold your breath though, there’s
still plenty of other things that could be wrong.

:: 23-May-2000 06:51 (Tuesday) ::

Me and my big mouth. No sooner did I finish that last
.plan update than I noticed that my ssh sessions to the
box weren’t responding. We’re working on it right now, but I’m guessing
that the box is going to need a good kick in the backside, which won’t
happen until tomorrow.

*sigh*

More info as available….

:: 23-May-2000 06:45 (Tuesday) ::

[01:39:30] .plan update, .plan update!!

Very well, since #distributed asked for it… }:8)

We’ve dropped the FSB speed on statsbox (no, it was *not* overclocked
before) and have also turned off swapping on the questionable drive. After
that, we got a backup and a half (the backup to Birmingham was cut short
due to a router failure, but we think we got most of it) of what’s on the
box right now. So, even if the box blows up, we should be ok for the most
part.

I just started a statsrun… the webserver is off right now but we’ll turn
it on in a bit. If we’re lucky, these two changes have solved the problem,
and we’ll just have to figure out which one did the trick (we’ve got a
little pool going… my money is on the swap partition }:8) ).

Anyway, with luck we’ll have stats back again in a few hours.

Sorry for the inconvenience.

:: 23-May-2000 02:01 (Tuesday) ::

Statsbox update….

The box has definitely degraded… it’s been down more than it’s been up
today. At this point, our priority is just to get a solid backup before
we’re totally dead in the water.

Based on the few useful error messages from /var/log/messages, this seems
to be a virtual memory issue, and we can also tell that one of the drives
is failing, so a safe bet would be that the swap partition on that drive
is giving us fits. A donated RAID controller should be on it’s way, which
would solve this *IF* it was in fact the drive that was to blame, but
that’s a pretty big *IF*. There’s also been discussion of flat-out
purchasing a box from a major manufacturer, with on-site tech support,
etc. But it would take some time before such a box could be put into
production, so hopefully we can stabilize statsbox in the meantime.

More info as available…

:: 22-May-2000 05:05 (Monday) ::

Statsrun died tonight. Well, to be more accurate, the box itself died
during statsrun. It’s been rather unhappy the last 24 hours.

We’re working the problem, but it’s possible that statsruns will be late
until we figure out what exactly is going wrong (it seems to be an issue
with swap getting cowrupted, but that’s not a definite at this time). I’m
hopeful that we will be able to continue running updates, but it’s possible
that things may degrade to the point where that won’t be possible.

More info as available.

:: 20-May-2000 01:06 (Saturday) ::

/usr on statsbox got too full for today’s statsrun, so I had to do some
cleaning. Stats will be an hour late as a result. Sorry for the delay.

:: 09-May-2000 01:34 (Tuesday) ::

For some reason, statsbox puked on decompressing one of the logs tonight,
even though the same exact log is working fine now. End result is that
stats will be ~1.5 hours late tonight. Sorry ’bout the delay. }:8/

:: 26-Apr-2000 01:50 (Wednesday) ::

Statsrun will be a bit late tonight. For some bizzare reason, the first
block for Apr 26 ended up in the last logfile for Apr 25. Statsbox doesn’t
like seeing this kind of thing. :) Sorry for the delay.

:: 24-Apr-2000 06:46 (Monday) ::

It’s way past my bedtime so this’ll be short. I just fixed the broken
images on stats… I’m not sure how they got broken, though it seems to
have been a CVS issue.

Many thanks to Bovine for helping me with CVS, especially in my current
cranky, sleep-needed-right-now state. :)

Moo.