:: 23-May-2000 06:51 (Tuesday) ::

Me and my big mouth. No sooner did I finish that last
.plan update than I noticed that my ssh sessions to the
box weren’t responding. We’re working on it right now, but I’m guessing
that the box is going to need a good kick in the backside, which won’t
happen until tomorrow.

*sigh*

More info as available….

:: 23-May-2000 06:45 (Tuesday) ::

[01:39:30] .plan update, .plan update!!

Very well, since #distributed asked for it… }:8)

We’ve dropped the FSB speed on statsbox (no, it was *not* overclocked
before) and have also turned off swapping on the questionable drive. After
that, we got a backup and a half (the backup to Birmingham was cut short
due to a router failure, but we think we got most of it) of what’s on the
box right now. So, even if the box blows up, we should be ok for the most
part.

I just started a statsrun… the webserver is off right now but we’ll turn
it on in a bit. If we’re lucky, these two changes have solved the problem,
and we’ll just have to figure out which one did the trick (we’ve got a
little pool going… my money is on the swap partition }:8) ).

Anyway, with luck we’ll have stats back again in a few hours.

Sorry for the inconvenience.

:: 23-May-2000 03:07 (Tuesday) ::

First off, thanks for all your patience with statsbox’s instability the
past few days. It looks like we’re losing the 9gb drive that’s in the
machine. getting scsi bus errors, timeouts, and system panics. (panics
are, we think, because that drive housed some swap space in addition to
database data).

We’re in the process of doing a backup of all the data to a few off-site
machines and then we’re going to do some more detailed diagnostics to
determine the source of the problem. The symptoms are still a bit sketchy
at this point and we don’t have a solid theory. Data integrity first,
then we’ll worry about replacing the faulty hardware.

It’s unlikely that we’ll bring stats up tonight, most likely it’ll be
tomorrow at the earliest.

Look for further updates from myself, decibel, or peter as we hone in on
the problem.

:: 23-May-2000 02:01 (Tuesday) ::

Statsbox update….

The box has definitely degraded… it’s been down more than it’s been up
today. At this point, our priority is just to get a solid backup before
we’re totally dead in the water.

Based on the few useful error messages from /var/log/messages, this seems
to be a virtual memory issue, and we can also tell that one of the drives
is failing, so a safe bet would be that the swap partition on that drive
is giving us fits. A donated RAID controller should be on it’s way, which
would solve this *IF* it was in fact the drive that was to blame, but
that’s a pretty big *IF*. There’s also been discussion of flat-out
purchasing a box from a major manufacturer, with on-site tech support,
etc. But it would take some time before such a box could be put into
production, so hopefully we can stabilize statsbox in the meantime.

More info as available…

:: 22-May-2000 05:05 (Monday) ::

Statsrun died tonight. Well, to be more accurate, the box itself died
during statsrun. It’s been rather unhappy the last 24 hours.

We’re working the problem, but it’s possible that statsruns will be late
until we figure out what exactly is going wrong (it seems to be an issue
with swap getting cowrupted, but that’s not a definite at this time). I’m
hopeful that we will be able to continue running updates, but it’s possible
that things may degrade to the point where that won’t be possible.

More info as available.

:: 20-May-2000 01:06 (Saturday) ::

/usr on statsbox got too full for today’s statsrun, so I had to do some
cleaning. Stats will be an hour late as a result. Sorry for the delay.

:: 09-May-2000 01:34 (Tuesday) ::

For some reason, statsbox puked on decompressing one of the logs tonight,
even though the same exact log is working fine now. End result is that
stats will be ~1.5 hours late tonight. Sorry ’bout the delay. }:8/

:: 05-May-2000 05:29 (Friday) ::

Within the last few days we’ve been getting reports from some users that
their virus scanners have suddenly begun detecting the presence of the
Win32.Weird.12288 virus within DNETC.EXE, DNETC.COM, and DNETC.SCR of the
v2.8007.458b client. This false report is known to occur with the May 1
version of the VET Virus scanner, and we have been assured that the error
has been corrected in their most recent virus signatures. Most users
would probably have updated to the corrected signatures if they had also
updated to signatures that include detection of the ILOVEYOU worm anyways.

:: 04-May-2000 12:58 (Thursday) ::

An updated client source tarball has been made available. It contains
all of the improvements that we have made in the recent beta series of
clients, including an optimized Athlon RC5 core.
http://www.distributed.net/source/

Additionally, a few new utilities have been added to the Add-ons page in
the last few weeks, including a new MooSounds perl script that is usable
on many UNIX platforms. You might want to take a brief look at our download
page again if you haven’t done so in awhile. There might be a useful
utility or two that you didn’t know existed.
http://www.distributed.net/download/addon.html