:: 05-Sep-2003 14:24 GMT (Friday) ::

We just realized that participant emails aren’t being checked properly before
being added to the database; specifically, emails are being checked in a
case-sensitive manner.

I’m working on fixing the problem right now, but I’ve taken the stats-runs
off-line until I can get everything straightened out. Hopefully shouldn’t
affect tonight’s statsrun.

:: 03-Sep-2003 11:18 GMT (Wednesday) ::

I’m trying to fix the kernel shared memory parameters so we can run more
PostgreSQL processes. Stats should be back in 10 minutes or so.

:: 02-Sep-2003 17:02 GMT (Tuesday) ::

Good news; I’ve found and corrected the problem with the apache processes
consuming several hundred meg, which means not only can I bump the timeout
setting back up, but that you’re much less likely to run into database
connection errors (which are still masquerading as other errors).

:: 02-Sep-2003 13:58 GMT (Tuesday) ::

I’m doing some quick maintenance which means some pages won’t be working for
the time being. Should hopefully take only 10 minutes or so.

:: 02-Sep-2003 09:46 GMT (Tuesday) ::

Two issues people should be aware of:

– There is a problem with apache processes sometimes consuming several hundred
megs of memory. We don’t know what’s causing this yet, although it seem to
only happen during certain parts of the daily statsrun. Because of this, I’ve
dropped the connection timeout to 30 seconds. This could affect users who are
on dial-up.

– Database connection errors are not being properly trapped. When PHP can’t
connect to the database, you’ll get incorrect output, such as ‘team does not
exist’. Give it a few minutes and try again; the data is there.

:: 01-Sep-2003 18:31 GMT (Monday) ::

I forgot to mention that stats are working through a backlog right now; should
be caught up in a day or two.

:: 01-Sep-2003 18:26 GMT (Monday) ::

Stats are finally back again, now powered by PostgreSQL, though there’s still
work to be done.

All code that would modify the database is currently disabled, so you can’t
edit your info, join teams, etc.

There are still some other minor bugs. If you find any bugs, please enter
them into bugzilla, but PLEASE check to make sure someone else hasn’t already
entered it!

:: 31-Aug-2003 22:39 GMT (Sunday) ::

Paul’s done a lot of work today, and except for a few bugs I think we’re ready
to roll out mostly functional stats. I’d do it tonight but I want to make sure
he’s around in case of problems.

All of the lookup functionality exists; what is missing is participant editing.
We should be able to get that done fairly soon though.

:: 30-Aug-2003 18:04 GMT (Saturday) ::

The PostgreSQL copy of stats is up-to-date now, although it does need to
process August. The HTML side still needs to be setup, which paul will work on
in 10 hours or so (it’s midnight his time right now).

Unfortunately this means another day without stats, but I’m really
uncomfortable with trying to get data back into Sybase properly. There might
be some information that has been lost going into PostgreSQL, but it should
only be some changes that were made in a window of about 8 hours. The Sybase
database is currently 6 weeks out of date, so there’s much more room for
problems.

While paul is getting the HTML stuff setup there might be periods where you can
not get to any stats web pages at all. Don’t worry, this is just him working on
stuff.

I want to thank everyone for their patience; the end is in sight.

:: 30-Aug-2003 13:21 GMT (Saturday) ::

Quick update…

Sybase is up and running again, but with month-old data. Sybase’s BCP routine
doesn’t handle embedded delimiter characters very well, so I’m a bit uneasy
about trusting the data pulled out of the corrupted database, especially for
loading in a month of changes impacting 2600 participants.

Because of this, I’m concentrating on getting the data loaded into PostgreSQL,
since there should be very few changes (the data is PostgreSQL is only a few
hours older than when I shut down stats).

In any case, progress is being made and we’re much closer to working stats now
than we were yesterday.