:: 31-Aug-2003 22:39 GMT (Sunday) ::

Paul’s done a lot of work today, and except for a few bugs I think we’re ready
to roll out mostly functional stats. I’d do it tonight but I want to make sure
he’s around in case of problems.

All of the lookup functionality exists; what is missing is participant editing.
We should be able to get that done fairly soon though.

:: 30-Aug-2003 18:04 GMT (Saturday) ::

The PostgreSQL copy of stats is up-to-date now, although it does need to
process August. The HTML side still needs to be setup, which paul will work on
in 10 hours or so (it’s midnight his time right now).

Unfortunately this means another day without stats, but I’m really
uncomfortable with trying to get data back into Sybase properly. There might
be some information that has been lost going into PostgreSQL, but it should
only be some changes that were made in a window of about 8 hours. The Sybase
database is currently 6 weeks out of date, so there’s much more room for
problems.

While paul is getting the HTML stuff setup there might be periods where you can
not get to any stats web pages at all. Don’t worry, this is just him working on
stuff.

I want to thank everyone for their patience; the end is in sight.

:: 30-Aug-2003 13:21 GMT (Saturday) ::

Quick update…

Sybase is up and running again, but with month-old data. Sybase’s BCP routine
doesn’t handle embedded delimiter characters very well, so I’m a bit uneasy
about trusting the data pulled out of the corrupted database, especially for
loading in a month of changes impacting 2600 participants.

Because of this, I’m concentrating on getting the data loaded into PostgreSQL,
since there should be very few changes (the data is PostgreSQL is only a few
hours older than when I shut down stats).

In any case, progress is being made and we’re much closer to working stats now
than we were yesterday.

:: 30-Aug-2003 00:03 GMT (Saturday) ::

I just talked with MattR; he’s not going to be able to recover the existing
database. We’ll be restoring from a backup as soon as it’s done bunzipping.

:: 29-Aug-2003 23:58 GMT (Friday) ::

MattR was able to get the database online again, so I now have copies of the 3
tables. This means that no data should end up lost out of all of this.

There are 3 options we have right now. First, we can attempt to repair the
existing stats database. MattR’s attempting this right now. The second option
is to drop the existing database, restore from the July 12th backup, and bring
in the updated information. The third option is for us to cut-over to stats
running on PostgreSQL, which is 95% done right now.

I’m playing it a bit by ear before deciding which way to go. Going to
PostgreSQL is very tempting, since we’ll need to do it in the near future
anyway, but I don’t like the idea of going to production when it’s not complete
and hasn’t been beta-tested.

Whatever happens, stats definitely won’t be up until tomorrow afternoon at the
very earliest.

:: 29-Aug-2003 20:09 GMT (Friday) ::

Blower is up and running again; apparently it died because the raid controller
freaked out after a disk failure. I’m still hoping to recover any data that was
modified last night, but if that doesn’t happen soon I’ll just go with what we
have.

:: 29-Aug-2003 15:59 GMT (Friday) ::

Looks like all the trouble is being caused by hardware issues. Moose is working
on getting blower up and running again, at which point I’ll have a better idea
what’s left to be done.

:: 29-Aug-2003 11:45 GMT (Friday) ::

As I feared, the Sybase stats database is corrupt. I have copies of all
participant data (team membership, retires, team and participant settings) as
of ~8/29/03 0:00 UTC. If I have to restore using this data, any changes people
made last night will be lost. I’m still trying to find a way to at least read
from the corrupt database, since only one table is corrupt. If I can do this,
no data will be lost during the restore.

Unfortunately, no matter which route I take, stats will be down for most of
today at a minimum.

More info as available.

:: 29-Aug-2003 02:40 GMT (Friday) ::

RC5 stats are still wrong, but I’ve turned access back on. Hopefully I’ll have
it fixed tomorrow.

:: 29-Aug-2003 00:38 GMT (Friday) ::

I’ve added a couple new clients on the pre-release page (NeXT and
NetBSD) and moved two earlier NeXT clients onto the release page.
http://www.distributed.net/download/prerelease.php
http://www.distributed.net/download/updates.php

There have also been a couple of reports of unauthorized client
installations in combination with a root kit, so I’ve added another
entry to: http://www.distributed.net/trojans.php

Naturally this type of behavior is very unacceptable and causes a
great amount of effort for those who need to rebuild machines that
have been compromised in this way. So please restrain yourself to
only running out client on machines that you have proper authorization
and permission to do so on.

In other news, as of this weekend the keymaster is now filtering old
OGR blocks from clients from version v2.8014 or earlier (as per my
previous plan).