:: 05-Jan-2000 07:04 (Wednesday) ::

I discovered/realized that the changes made to the psearch code
will end up requiring some changes in the statproc code as well.
This is always a bit of a hassle, since there are two copies of
statproc code, one for each contest. Instead of making the changes
in both places, I decided to make the changes necessary to allow
the two scripts I changed to work for both contests.

Unfortunately I didn’t realize the RC5 statsrun was still happening.

Net result is that the ranking portion of the RC5 code is now re-
running. Unfortunately, I lost yesterday’s overall email ranking
in the process, so everyone will show up as having moved 0 places
overall in tommorow’s stats. The ranking will be back to normal
tomorrow.

Until I get these changes finished, the participant search function
may stop working for a short time after the rc5 statsrun (it will
only stop working for rc5 folks). CSC *should* be ok.

The good news is that I am working on (and making progress on)
allowing Nugget, Bruce, and myself to have small-scale copies of
the stats database that we can tinker with. This should help
eliminate some of these problems that we’ve been having as of late.

Again, thanks for you patience.

:: 05-Jan-2000 03:06 (Wednesday) ::

Well, we’ve finally got the load on statsbox under control again
*knocks on wood*

The problem seemed to be with the participant search script again.
(This is what caused the massive slowdown last time too). The
problem is that in order to allow the most flexible search
possible, the script would search for the string entered anywhere
in the email field. (“*string*”). Unfortunately, Sybase can’t
optimize this query, and it was killing the box. (The data-servers
were using over 80% CPU almost constantly… when the psearch
scripts were down, usage was rarely over 60%).

So, the script has been redone a bit. First, if there’s an @ in
the search term (such as when looking up a complete email
address), the script will attempt to do a “string*” search first,
which Sybase can optimize. If that fails (or if there’s no @),
the script will do a “*string*” search like before, but it will
only retrieve a list of IDs, which it will then use to look up
participant info for the specific project. This is a somewhat in-
efficient way to do things, but it’s brought the load under
control. The only functional difference will be if you try to do
a partial search using a string with @ in it. You’ll end up searching
on “string*” instead of “*string*” like the script used to.

I have a feeling we’re going to be needing faster CPUs in the near
future… anyone want to buy the PII-300s we’re using for a
keepsake? }:8)

One other note… if you typically do a search on your email, you
might want to bookmark your summary page instead… it will come
up faster if you don’t go through the psearch script.

Sorry for the downtime and errors while I was debugging. Everything
seems to be working now.

:: 04-Jan-2000 08:21 (Tuesday) ::

For those who are wondering, I shut down statsbox web access temporarily
to build some additional indexes. We’re seeing a very heavy load right
now, and this should help. I’ll also modify the statproc routines to
build this index nightly in the future.

:: 31-Dec-1999 01:29 (Friday) ::

I’ve updated the team info in the master tables to reflect what it was
yesterday. If the csc logs ever show up from the master, we’ll be good
to go. If they’re not in by 1:45GMT, rc5 stats will end up running, in
which case we’ll run the CSC stats afterwards.

:: 30-Dec-1999 21:14 (Thursday) ::

For all of you who want to know the gory details of what happened to stats
last night, here they are.

About a week ago, Sybase skipped a bunch of identities in STATS_Participant,
the main participant info table. This tends to happen when the server is
shut down abnormally, and has happened several times in the past. Unfortunately,
we didn’t catch it when it happened this time, so there were several days of
data with bad participant IDs. This meant re-writing the script we use to
correct this problem.

We re-wrote the script a few days ago, but hadn’t run it yet. Yesterday, Bruce
came up with a change for the statsrun code that would allow us to eliminate
the identity field from STATS_Participant, thereby getting rid of this issue.
Since this would also require rebuilding STATS_Participant, it seemed a perfect
time to fix the identity problem.

With the new statsrun code in place and looking good, I was ready to run the
re-identity script. I almost made a backup copy of STATS_Participant, but
remembered that the re-identity script made a backup copy on it’s own. Of course,
this breaks an axiom of computers that goes something like ‘Too many backups is
almost enough.’

The script had a minor syntax error, and in the process of trying to debug it, I
ran the script several times. This had the unfortunate side-effect of sending all
traces of STATS_Participant to the great bit-bucket in the sky.

Not to worry, we make weekly backups of the database for this very reason. With a
boatload of help from Nugget, we got a copy of STATS_Participant from Dec 27 back
into the database. Though, it took us three tries to figure out how to get it back
in without having Sybase redo all the identities.

Once that was in, we fixed the identity problem (making loads of backups along the
way this time) and pondered how to recover the participant data and team info that
had been changed/added to STATS_Participant in the past few days. We decided that
re-running those days from logs would be the easiest.

We extracted the days in question from the master tables and used that info to reset
everyone’s team affiliation to what it was for the 12/28 statsrun (a process that
ended up taking a few hours). We fired off the statsrun and waited. We had to fine-
tune Bruce’s code a bit, but things seem to be going well now and the 12/29 RC5 data
is just about done.

The only thing left to do is re-assign blocks for the past few days to the correct
teams for the handful of people who changed their team membership during those days.
Because that hasn’t been done yet, some of the team stats might look a bit off.
After tonight’s statsrun, everything should be back to normal.

I’m glad we won’t have this problem with STATS_Participants again! }:8)

Thanks as always for your patience and CPU time.

:: 29-Dec-1999 23:54 (Wednesday) ::

Stats will be down for a while while I fix the participant ID overrun
that happened a few days ago. Anyone with a participant ID >= 500,000
will end up with a new ID as a result. This should only affect participants
who started Dec. 20th or later. All of your blocks and team affiliation will
be saved, the only change will be to your ID number.

I’ll post more info once everything is back online.

:: 18-Dec-1999 23:09 (Saturday) ::

Well, something compelled me to investigate whether all of the tables
in stats that contain block totals (there’s 3 of them) matched or not.
Unfortunately, I discovered that they don’t.

In addition to the master table, which contains 1 record for each day
that each participant has submitted blocks, there is a platform table,
which contains a row for each cpu/os/client version combination for each
day, and a dailies table, that contains info like how many total blocks
were completed on that day, how many participants, how many teams, etc.

Naturally, if you sum the blocks from all these tables, the totals should
match. Unfortunately, this isn’t the case for rc5-64.

Bruce and I have managed to correct many of these errors (including one
bad day for CSC). This means that our most astute of stats observers will
probably notice that our total blocks done will go down for both contests.
This does *not* mean that we’ve lost work, it only means that there were
some days when the statsrun goofed up and essentially double-counted the
blocks for that day. Also, from what we’ve seen, there are no cases of
errors in the master table, so personal stats shouldn’t be affected.

The bad news is that we haven’t fixed all the errors. It seems that there
was some kind of bug in the old stats code that ran on statsbox-I which
resulted in minor differences between the master and platform tables,
differences on the order of a few thousand blocks per day. These errors
will be much more difficult to track down and fix, and since they should
only affect cpu/os information, we might just leave them alone.

The good news is that, by and large, these errors disappeared when we
cut over to statsbox-II. There’s only 12 days since Feb. 1, 1999 that
have errors.

We’re beginning to work on what will amount to stats, version 3. We’ll
be adding some processes that will continuously scan through stats in
the background, looking for any anomalies like this, so hopefully we’ll
be able to quickly catch anything like this in the future.

Sorry for the confusion… keep cracking!

:: 15-Dec-1999 12:14 (Wednesday) ::

Statsbox is up to it’s old tricks again. }:8( Several of the pages
seem to be running off of Dec 13 data, instead of Dec 14 data.
Unfortunately, I have to be on a plane to Detroit in just over an
hour, so I couldn’t stay up late last night to look into it. With
(a lot of) luck, things will correct themselves with to-night’s run.
If not, I’ll have to look into it tomorrow.

:: 09-Dec-1999 08:36 (Thursday) ::

It seems that the gremlins aren’t dont messing with statsbox just
yet. For some reason, all the data for rc5 for Dec. 7th is gone.
I looked at the logs, and it did get imported, so I have no idea
where the data went.

I’m re-running stats for Dec. 7th. This will almost certainly
screw up all the ‘ranking deltas’ (those cute little up/down arrows
that let you know how much your ranking changed from yesterday),
so don’t be alarmed when things look a little weird. All should be
back to normal after the Dec. 9th statsrun.

:: 09-Dec-1999 03:53 (Thursday) ::

For those of you who are either hyper-observant, or addicted to
stats, you will notice that although the CSC main page says we
did 1.7% of the keyspace yesterday (which we did), and reports
that we’ve completed 26.9% of the keyspace overall (which we
have), that this math doesn’t make sense since yesterday it
reported that we had completed 26.4% of the keyspace overall
(which we hadn’t). This was because I forgot to delete a
record when the Dec 5 statsrun got botched.

This only affects the main CSC stats page and the CSC graphs.
No participant info was harmed in the making of this ‘whoops’.

All is now well again on that front.

Unfortunately, rc5-64/psearch.php3 is now broken. To make a long
story short, nugget had backed out a change made to this page
on the live site manually. I then fixed the file and put the
fixed version into CVS, and ran the normal command to update
the main site from CVS. CVS saw that the version of psearch on
the main site had been modified, so it created an ‘annotated
checkout’ of the file, with some nice commentary to let us know
what changed. Of course, PHP has not clue what all the extra
stuff in this file is supposed to do, so trying to pull up that
page results in a ‘The document contains no data’ error. Since
I don’t have root on the box, I can’t fix the garbled file either.
So, it’ll be broken until I can find someone rootly to fix it,
which hopefully won’t be long.

Sorry for the cowfusion and thanks for your patience!