:: 03-Apr-2002 15:42 GMT (Wednesday) ::

In less than an hour, rc5 will be up-to-date in the new stats processing
system. This means we can finally migrate to a single code base for everything,
which paul is working on. It also means improved data integrity for the future,
as the new stats system has extensive auditing to ensure that errors don’t go
un-detected. Unfortunately, in order to move the RC5 info into the new system
without re-processing every log that exists for RC5, we had to make some
assumptions about the data that is in the old system, which does contain some
inconsistencies. Correcting this will require re-processing all existing RC5
logs, which will happen at some point in the future.

Anyway, look for a new set of stats pages in the near future.

Moo!

:: 01-Apr-2002 15:09 GMT (Monday) ::

I think I may have actually figured out a way to do real-time stats,
something that some participants have been foaming at the mouth about for
a very, very long time. The technique will involve utilizing our network
itself for stats computation. The problem is that right now it looks like
it will require about 80% of our network to make it work. We’ll have to see
if that can be improved upon.

BTW, happy April 1st to everyone.

:: 24-Mar-2002 09:55 GMT (Sunday) ::

RC5 stats didn’t run last night; the process that pushes
logs from the master to blower needs to be tweaked to
deal with the UD NAT. I’ve just started the run for the
23rd manually. Expect RC5 runs to be delayed until one of
the master folks can fix the log push.

Also, not all proxies have been adjusted for the master’s
new location, so expect work to show up as missing until
all the proxies have been able to flush their work.

:: 22-Mar-2002 23:56 GMT (Friday) ::

With the move of the master to a new colo, stats will probably be
delayed for a bit until we can get all the details sorted out. We’ll
have them back online ASAP.

:: 20-Mar-2002 23:34 GMT (Wednesday) ::

The reason stats are so low for today is the master
was down most of the day. It will hopefully catch-up
with the backlog tomorrow. See bovine’s .plan for
more info.

:: 08-Mar-2002 02:49 GMT (Friday) ::

And there was much rejoicing.

Sybase 11.0.3 can’t optimize stored procedures very well, which was causing a
problem for the participant history page. Every time someone hit that page, it
would cause the database to read a substantial portion of the (now 52 million
row) Email_Contrib table. Not only would it take forever to get that page back,
but all of that disk IO was seriously impacting the server itself.

Anyway, I’ve replaced the stored proc with a raw query (so much for code
abstraction), and everything seems to be running OK again.

:: 07-Mar-2002 07:46 GMT (Thursday) ::

Blower is quite unhappy with the rc5 data in the new table format.
Specifically, web site access is generating a huge amount of disk IO. Based on
a quick check of page load times, the problem seems to be on the email side,
rather than the team’s side.

I’ve updated index statistics on all the major tables used by the new stats
system, but that didn’t help.

We’re going to have to try and figure out what queries are taking so long to
run from the website. Until we can do that, things are going to run a bit slow.
Hopefully we can get this cleaned up in the next week or so. In the meantime,
expect web access to be turned off at times to facilitate the statsruns.

:: 07-Mar-2002 04:14 GMT (Thursday) ::

Blower is still unhappy. I’m updating index statistics on the off-chance that
will fix the problem. More info as available.

:: 06-Mar-2002 08:10 GMT (Wednesday) ::

Blower’s been running terribly slow on the RC5 data for Mar 4, so I’ve shut
off the website for the next two hours. Hopefully it will be well caught up by
then.

:: 05-Mar-2002 23:34 GMT (Tuesday) ::

There will be a brief network outage at UD tonight. This will affect stats and
cvsweb.