:: 07-Nov-2000 18:58 (Tuesday) ::

Quick status update… RC5 stats are running now; Nov. 4th is nearly done.
As soon as RC5 stats are done, I will start the OGR stats. Peter and I
will be working on some hardware changes to the box tonight though, so
don’t expect stats to come back on until that’s completed. If things go
well tonight, stats should be back on in the morning.

:: 07-Nov-2000 02:43 (Tuesday) ::

I’ve started running RC5 stats again… it’s working on 11/01 right now.
I’m still getting an error from dbcc checkalloc though, which I’d really
like to fix before bringing everything back online. Nugget was also able
to get me a BCP of the teams table from May of this year… that should
drop the number of affected teams down to 20-30 or so.

More info as it’s available.

:: 06-Nov-2000 07:18 (Monday) ::

I just realized that I hadn’t posted an explanation for what exactly
happened to stats. Here’s the rundown:

The OGR statsrun aborted Thursday when the audit script detected some data
inconsistencies. This script makes sure that the total amount of work
reported for a given project matches in a bunch of different tables. (For
anyone interested in the gory details, see
http://cvs.distributed.net/cvsweb.cgi/stats-proc/daily/audit.sql).

The script was indicating a disparity between how much team work was in
the master table, the team members table, and the team ranking table. It
turns out that a number of records (3000 is the number I recall) had
vanished from the table that contains team information (the team name,
contact person, motto, password, etc.) This missing information meant that
a lot of work didn’t get credited to teams for that day.

Once I found this out, I shut off http access. I didn’t know why these
records went *poof*, but I wasn’t going to rule out bad hardware, and if
it was bad hardware, there’s no telling what other damage could occur
while we were pushing the box.

I pulled in a copy of this table from a backup only to discover that the
backup was incomplete/incompatible. The backup had been done using a
Microsoft version of the BCP tool, talking to our Sybase database. The
BCP out to a file (the backup) seemed to work fine, but on this table, it
didn’t like working in the other direction. Before everyone assumes this
is Microsoft’s fault, keep in mind it could just as easily be an issue
with Sybase. Also, Nugget originally used a Microsoft BCP tool to get the
data into Sybase, and it worked fine back then.

Although the backup wasn’t what it should be, it did contain 99% of the
missing info. While working to find out what teams were still MIA, I
discovered some issues with some of the team data in the master tables
not matching the information in the table that tracks the history of what
team everyone has been on. I had to correct that issue before I could
accurately determine what teams were still missing.

Once that was done, I determined there were still 154 teams missing records.
Bringing in an older copy of that table brought the MIA number down to
124. We’re looking to see if there’s any more copies of that table tucked
away someplace. In the meantime, I re-created records for all the teams
that were missing them.

I should point out that no actual stats data is missing; the audit script
would have caught it if that was the case. This will only affect the
appearance of some team’s team summary pages.

I’ve been unable to determine exactly why tally dropped some data on the
floor. I’ll be working with the person who is colo-ing the box for us to
see if we can discover any hardware related issues.

The database is currently being backed up (using Sybase’s BCP utility),
and as soon as that’s finished and I’ve verified that the backups are
good, I’ll start processing logs again. We’re 5 days behind right now, so
it will probably take about a day to catch up, assuming I leave http access
turned off. I hope to have everything back to normal by Wednesday.

Thanks (as always) for your patience.

:: 04-Nov-2000 17:22 (Saturday) ::

Have you wished someone happy birthday today? Maybe someone who’s now
3/4ths of the way to officially being over the hill? }:8)

:: 04-Nov-2000 14:08 (Saturday) ::

Did I happen to mention that our beloved president turned 30 today?

:: 04-Nov-2000 12:45 (Saturday) ::

Today I turn 30, so I guess it’s no longer safe to trust me.

Since the “stats are down” page points here, I thought it would be prudent
to redirect folks to decibel’s plan.

http://n0cgi.distributed.net/cgi/dnet-finger.cgi?user=decibel

Due to a variety of real-life demands being made on my time, I’ve been
less involved in stats development over the past few months. The present
crisis, at least, is in decibel’s capable hands.

:: 04-Nov-2000 08:00 (Saturday) ::

Happy birthday Nugget! }:8)

:: 02-Nov-2000 17:26 (Thursday) ::

Bad news. tally has started doing some odd-ball things. Until I can ensure
that we have a couple of good, reliable backups, I don’t want to stress
the hardware any more than necessary. I’ve been awake for the past 24
hours now, so I think working on it any further without some rest would
be unwise. So, stats will remain off for the immediate future. This is
to ensure that we still have stats to come back with once the hardware
issue is sorted out.

:: 01-Nov-2000 19:01 (Wednesday) ::

Fixed. }:8)

:: 01-Nov-2000 05:10 (Wednesday) ::

Ok, things didn’t go exactly as planned. The number of team members is
off for all of OGR. It’ll have to wait until tomorrow, as I’m far too
tired to mess with it now without strong odds of really mucking things
up.