:: 19-Nov-1999 00:37 (Friday) ::

Just a few “known issues” with stats…

o the code that determines how many new participants and new teams
joined is horribly confused by the addition of the csc contest,
so don’t rely on any reported values there. Still need to figure
out how to work that out accurately.

o http://stats.distributed.net/csc/money.html describes the prize money
distribution for the csc challenge. Thanks mostly to the head-start
that dcti staff had on the general userbase, distributed.net has a
fragile lead on fsf for receiving the bulk of the prize money. Looking
at the numbers, it’s pretty clear that fsf will pull ahead tomorrow.

Just as a reminder, nonprofit voting doesn’t work like teams. If
you’d like to change your nonprofit selection for the csc contest,
all your blocks (past, present, and future) will count as votes for
whichever nonprofit you select. It sure would be nice to see some
competition in this area, instead of the landslide of fsf votes we
see for rc5-64. Perhaps the recent article on project gutenberg
as seen on slashdot.org would compel someone to vote for P.G. :)

:: 18-Nov-1999 22:17 (Thursday) ::

CSC stats are up, barely. There are still quite a number of minor “issues”
that need to be cleaned up, but for the most part we’re running just fine.

I apologize for taking so long, but a number of factors contributed to
my delinquency. Stacey and I moved into our new home this past weekend,
and slacker.com has lost its ISDN pipe and is on a 28.8 connection to
the net for the time being. (finger nugget@slacker.com for the gory
details, if you care…)

You’ve probably also noticed that last night’s rc5 stats run failed.
As soon as it rolls around to 00:00, I’m going to kick off the 18-Nov
CSC stats run (should take about two minutes to run) and then kick off the
17-Nov and 18-Nov RC5-64 stats. I’d guess we’ll be all straightened out
by 04:30 or so.

Thanks for your patience and support. And, *please*, if you haven’t
upgraded your clients to the latest CSC-enabled versions, now’s a good
time to do so. Think how fun it will be to tear through a quick contest
like CSC after watching RC5-64 lumber along for over two years.

:: 31-Oct-1999 18:28 (Sunday) ::

One of the benefits of insomnia is that from time to time you can get a
lot done in the middle of the night when you cannot sleep.

I closed out a lot of the tiny bugs in stats last night, and added a new
feature which I think will be quite popular. Major issues, in no
particular order…

o Your participant listmode is now honored much more reliably. Most
of the places on stats that didn’t previously honor a request to
“list me using my real name” now correctly do so.

o the “neighbor” list on the bottom of each psummary page now also shows
current rate as well as overall rate.

o Each participant can now select up to five other participants as
“friends” and those other participants will be listed/linked on the
participant’s summary page. You can see an example on my summary page at
http://stats.distributed.net/rc5-64/psummary.php3?id=1
You can edit your own list from your pedit participant editing page.
The interface is extremely crude at this stage, it’s just five boxes
which allow you to enter other people’s participant IDs. You’ll have
to look up your friends’ ID numbers manually for now.

:: 12-Oct-1999 12:24 (Tuesday) ::

distributed.net spent US$2,600 yesterday in order to upgrade our main
server (nodezero) and bring a second web server online. For an
organization that effectively has no income, this is always a scary
thing. Our cash reserves are down to just over US$10,000 which is
about as low as I’d want to see it get. Our ability to properly
maintain the server network and stats server are directly affected by
our ability to buy the hardware we need.

The most important way you can help distributed.net is to continue to
run the client and encourage your friends to do so as well. However,
if you’re inclined to help in a more direct manner, there are a number
of ways you can do so. First, we are a 501(c)(3) tax-exempt corporation
and donations are tax-deductible for you .us folks.

There’s also a way to donate money to distributed.net that’s free —
www.iGive.com

iGive has basically funded our group, having donated more than US$19,000
since they were founded in 1998. Basically you earn money for yourself
and/or distributed.net by viewing advertisements and purchasing things
through iGive vendors.

Our experiences with iGive have been quite rewarding. These guys are
building a very interesting business model and doing everything right.
These guys don’t spam, or send out annoying adverts. There’s no silly
banner bar you have to find space for on your desktop. You just visit
their site each day, at your convenience, and get paid to look at ads.
If you buy a product from one of their listed merchants, distributed.net
gets a percentage of the sale price at no cost to you. I’ve been able
to raise about $60 for distributed.net just by buying all my audio CDs
and software through cdnow.com and beyond.com, which are two of the
vendors.

Right now iGive is running a promotion where distributed.net gets US$10
for every person who joins by November 15th and makes an online purchase
by November 30th. If ten people sign up with iGive and buy a cheap CD
from cdnow.com, that’s $100 to distributed.net.

To join, just follow this url…
http://www.iGive.com/html/ssi.cfm?cid=1098&mid=3311

To keep an eye on distributed.net, you can see where our money has gone:
http://www.distributed.net/legal/ledger.html

As always, thanks for your continued support and enthusiasm.

:: 07-Oct-1999 15:37 (Thursday) ::

OK, It looks like x-empt was right… There’s definitely a problem in
stats that’s preventing new participants from being properly added to the
database. You’ve probably noticed that the past several days have shown
“(and 0 of them are brand-new)” on the main rc5-64 stats page.

The strange aspect to this is that not only has the code in question not
changed, but when I run the sql manually I’m unable to reproduce the
problem.

I just ran the sql code by hand on the 6-Oct daytables and it added
the new 6-Oct participants in without a hitch. Yet for some reason when
that exact same code ran last night during the update, it didn’t work.
Very odd.

So far, all I’ve done is just add a bunch of logging to the suspect code
to see if I can isolate the exact spot where it’s falling down. It
just doesn’t make any sense.

I’m going to kick off a re-ranking, which is necessary for the affected
emails to be searchable and appear in on the stats web site. One
side-effect of a re-ranking is that the ranking offsets (the green and red
numbers showing your change from the day before) will all be affected by
this.

I’ll post more information as I figure out what’s going on. There’s no
reason to assume that the code will start working tonight, I’m just hoping
to get a few clues on why it’s failing.

:: 04-Oct-1999 13:31 (Monday) ::

It looks as if statsbox decided to be a bit mischievous this weekend
while dbaker and I were both out of town. Certainly nothing critical,
but it was a bit annoying. A strange set of circumstances deep within
a “shouldn’t happen” loop in the stats and logcopying code resulted in
last night’s stats run only incorporating 23 of the 24 logfiles from
the keymaster. Any blocks flushed from 3-Oct-1999 23:00 to 23:59 will
not be reflected in today’s stats.

These blocks will be processed as part of tonight’s run (but still as
3-Oct blocks, not 4-Oct). Sorry for the inconvenience. dbaker and I
are going to spend some time today hardening the log processing code
to ensure that this sort of thing doesn’t happen again.

:: 01-Oct-1999 18:53 (Friday) ::

Well, (knock on wood) it sure seems like statsbox is stable once again.
I broke the cardinal rule in troubleshooting and changed a bunch of things
at once, so we many never know which change (or combination of changes)
ultimately fixed the problem. At this point, I’m just happy to have the box
running smoothly again in time for me to go out of town this weekend.

The wife and I will be in Nashville this weekend and it looks like statsbox
doesn’t need any more attention for the time being.

A fresh start on monday when I return… time to turn my attention to the
multitude of stats bugs as reported at http://www.distributed.net/bugs/
and then we can get back into feature additions.

It also looks like functional ogr stats will be needed quite soon, so
that’s on the docket as well. I’d like to, at the very least, get all
the _raw stuff implemented and get cpu/os stats back before we focus
on ogr implementation issues.

And to Kevin Bentz, thanks for the Aerosmith CD. It was the soundtrack for
the statsbox recovery after I ran out of RATM music. :)

:: 29-Sep-1999 22:25 (Wednesday) ::

I massaged the stats database today. Made a few changes which may or
may not correct our recent difficulties.

First, I gave us another 2GB of space in the stats database. We were
getting a bit low (not uncomfortably so, but still…) on space in
the database and I figured that now was as good a time as any to
make more room. This puts us at 4GB reserved for data, 1GB reserved
for transaction logs.

Second, I completely re-built the main rc5_64_master table. The old
table allowed for nulls in team field, and apparently it’s a bit more
efficient if nulls aren’t allowed in fields, so the new copy of the
table doesn’t allow nulls in any field. There’s no functional difference
for us, since “team 0” is used to denote no team, and it’s impossible to
have a null id, date, or block count.

A necessary aspect of rebuilding rc5_64_master was re-creating all the
indexes as well.

We won’t really know if this helped until tonight’s run (in an hour or
so) — so cross your fingers.

with apache down, everything’s running faster than it was, so I’m hopeful.

:: 28-Sep-1999 20:39 (Tuesday) ::

Note to self: SQL written while listening to Rage Against the Machine will
need to be cleaned up at a later date.

I’ve got a couple working theories on the stats slowdown, and some test
code to run some benchmarks. Unfortunately I noticed that the monday-
morning statsdb backup didn’t happen monday as it should have, so I’m
in a holding pattern as the tables bcp out to a safe place. As soon
as this is wrapped up I’ll get to run some time tests to see if today’s
changes fix the problem.

The “I searched for my stats and got nugget’s” problem is fixed. Most
of the php3 pages have defaults where they show id #1 or team #1 if no
valid data is provided to them. I made a change to psearch.php3 last
night that jumps straight to psummary.php3 if the search only yields
one email address. It seems that psearch was confusing a lot of people
who thought didn’t realize that they could click on the email address
and get to a full summary page. I screwed up when I made the change,
but didn’t notice when I tested it because (drum roll) I tested it with
my email address. In reality, it wasn’t passing the id properly to
psummary, but I thought it was because the default was me. oops. :)

It’s fixed now, and should be working just fine.

I’m pretty sure we’re on the right track, that the recent slowdown in
processing is triggered/exacerbated by the fact that as of last week,
normal stats viewing now touches the “big” rc5_64_master table. Until
we added phistory, no stats pages actually hit the main table.
As explained in decibel’s .plan, he disabled the phistory pages to see
if that would help. As you know, it didnt’, but that may very well be
because we missed a spot. The “today this participant did foo% of their
maximum” line also hits the rc5_64_master table, and we overlooked that.
So, you guys have been hitting the rc5_64_master table this whole time,
even with phistory disabled.

Assuming this is indeed the cause of the problem, I made changes to the
php3 code today which may eliminate the conflict. If the changes do not
help (and tonight’s run also takes way too long), the next step will be
to disable phistory and the “today compared to best day” psummary line
during the update. (not all the time, just when the update runs).

If that doesn’t help, well… we’ll burn that bridge when we come to it.

:: 27-Sep-1999 20:09 (Monday) ::

Finally have a chance to do some forensics on statsbox, I’m starting off with
the normal “dbcc checkdb” and “update statistics” just to make sure
everything is tidy at the lowest levels. I also see that we’re getting close
to capacity (only about 150mb free on the main statsdb device) so I’m going
to slice off another half gig or so of space for the data tables.

I think we’re within a few months of needing another drive in there,
the main RC5_64_master table is nearing 20 million rows.

The goal today is to determine the cause of the daily update slowdowns that
started last week, although I’d settle for prodding at the box and having it
automagically start working properly. :)