:: 06-Oct-2000 07:22 (Friday) ::
With some of the bad things that have been happening with stats lately,
I thought it’d be nice to report something good for a change. :)
First, I’ve made a few changes to the old rc5 statsrun code recently that
have had a dramatic impact on the run time. Retroactive team joins used
to take an hour; it’s now under a minute. Email ranking for yesterday used
to take over an hour; it’s down to a few minutes. The overall run is back
down to a ‘reasonable’ amount of time (it was back in the 6-10 hour range,
depending on when it started running (the heavier activity during the US
day would slow it down)).
Here’s the real good news though: I’ve been running some rc5 logs through
the new stats system. Here’s a sample…
[10:46:56] Spawning daily.pl for rc5 [10:47:01] Beginning daily processing
routines [10:51:30] retire.sql 5 completed successfully (276 seconds)
[10:53:24] dy_appendday.sql 5 completed successfully (100 seconds)
[10:56:39] em_update.sql 5 completed successfully (183 seconds) [10:58:03]
em_rank.sql 5 completed successfully (85 seconds) [11:02:30] tm_update.sql
5 completed successfully (274 seconds) [11:02:45] tm_rank.sql 5 completed
successfully (14 seconds) [11:03:02] dy_dailyblocks.sql 5 completed
successfully (16 seconds) [11:03:56] Looking for new logs, last log
processed was 20001005-14 [11:04:09] ogr20001005-15.log.gz received: 35,857
bytes in 1.5 seconds (23.9 KB/s) [11:04:16] ogr20001005-15.log.gz
successfully decompressed (82.8% compression) [11:04:18]
ogr20001005-15.log.gz successfully filtered through ./logmod_ogr.pl.
[11:04:47] ./workdir/ogr20001005-15.log.filtered successfully BCP’d; 2,275
rows at 58.33 rows/second. [11:05:04] ogr20001005-15.log.gz successfully
processed. [11:05:31] audit.sql 5 completed successfully (141 seconds)
[11:05:38] clearday.sql 5 completed successfully (15 seconds) [11:11:46]
backup.sql 5 completed successfully (368 seconds) [11:11:51] Daily
processing for 20000902 has completed [11:12:02] daily.pl complete for
rc5
This means that the rc5 run took less than half an hour, even though a
OGR hourly run snuck in the middle. Compare that to the 3+ hours that the
statsrun takes with the old system! Granted, there are only 3 days worth
of rc5 data in the new system at this time, but due to the design of the
system, that should not have a large impact on the amount of time the run
takes.
We are still in the process of creating a version of the new php code for
rc5, but we should be ready to drop the old system entirely in the very
near future.