:: 09-Oct-2003 18:06 GMT (Thursday) ::

They say that all good things must come to an end. (Don’t you hate messages
that start with “They say…”?)

I regret to announce that I will be resigning from DCTI after this weekend.
This was not an easy decision on my part, and I will miss working with the
staff and participants terribly. I’m truly amazed at how much I’ve learned
from working with the rest of the DCTI staff, as well as from hanging out in
#distributed.

In the end, it comes down to personal priorities. I’ve used this space more
than once to point out that we volunteers have personal lives. I finally
listened to my own advice and realized that any time I spend on DCTI is time
taken away from family, church, and work. There just isn’t enough time left
after those three to give DCTI the time it deserves.

After this weekend, my bwilson@distributed.net account will be inactive or
forwarded to someone else. (I leave that up to them to decide.) I will probably
continue to run the client and go back to being a stats junkie with many of the
rest of you. That’s what got me started in the first place, you know.

As always, I encourage any of you with ideas for projects, or with an
interest in helping out, or with coding/documenting skills to volunteer
and join up. The first step is to catch one of the current staff on
irc.distributed.net:#distributed and find out more.

It’s been a fun ride, and I thank you all for the ways you’ve helped us to
succeed as an organization. I wish you all the best.

As always, keep those CPUs crunching…

— Bruce Wilson bwilson@distributed.net

:: 16-Dec-2002 20:56 GMT (Monday) ::

While I haven’t been helping Decibel much with recovering
from the problems in stats, I think I have an acceptable
excuse…

http://www.toomuchblue.com/gallery/EmmaBirth

Moo!

:: 21-Dec-2001 00:38 (Friday) ::

Though we’ve had some outages recently because of the migration from tally
to blower, the current outage is unrelated. According to PetrDoubt, the
switch between tally and the outside world has frozen up and needs to be
rebooted. I’m sure Petr will take care of it as soon as he can. In the
mean time, please be patient. Also, you can be sure that this will not
affect any stats totals – all the work is queuing up and will be reported
correctly when we catch up.

Moo!

:: 23-Aug-2001 01:51 (Thursday) ::

Moo!

I’ve recently taken on answering some of the help@distributed.net mail,
specifically the topics our front-line help people have categorized as
requiring the attention of a stats dba. (For those of you still in the
queue, thank you for your patience.)

In the course of trying to answer some of the questions, I’ve made some
interesting discoveries:

. A lot of you depend on the phistory_raw page to produce additional
stats you can’t get any other way. (For those who don’t recognize the
page names, this is the Block Submission History page which shows each
participant’s lifetime history as a histogram.)

. Some of you use it so much, it’s putting a heavy workload on both
apache and sybase on tally.

. Some of you don’t really need the history for all time, just for a day,
or a couple of weeks.

. A certain @home subscriber in the northwest Chicago suburbs is
producing somewhere around 1/7 of all website traffic on tally, primarily
on the phistory_raw page.

I’d like to address all these problems real soon. As a first attempt,
I’ve cobbled together some variations on existing pages that I hope will
solve most of these problems. I’d like to ask you to test these out,
especially those of you who do a lot of scripting. They’re not linked
from the existing stats pages, but if they work as well as I hope, they
will be soon.

The new pages are:

http://stats.distributed.net/rc5-64/phistory30.php3?id=1
http://stats.distributed.net/rc5-64/phistory30_raw.php3?id=1

As the filename suggests, these pages return the same information but only
for the last 30 days. You can substitute your own id for the 1 at the
end. They’re only available for RC5-64 at this time. Depending on your
feedback, I might even make the number of days a parameter. Please submit
all comments and feedback about these pages to bugzilla,
http://www.distributed.net/bugs/

As for the participant in the Chicago area, we appreciate your enthusiasm
for stats. We’re not mad, and we’re not going to cut you off or anything,
but please PLEASE drop me an e-mail me directly (bwilson@distributed.net).
We’d like very much to find a solution that gives you what you need without
producing quite so much traffic for both of us. Hopefully, this quick
fix will serve as a reasonable alternative.

On a different note, we’ve recently “hired” two new guys to help with
stats. We’re planning an all-stats meeting this weekend (Aug 25, 2001)
to bring them up to speed, rebalance the workload, and set some priorities.
With that in mind, if you have any features you’d really like to see in
stats, I’d like to know about it ASAP. Items already on our dream list
include:

. OGR recycle/repair

. Cross-project stats

. Better inter-project links

. Site redesign

. my.distributed.net

. Sub-teams

. A decaying average score, so newer participants rise faster and
inactive participants fall faster

. Using id as the id instead of e-mail (retires become a thing of the
past!)

.

As always, thanks for all the cycles!

:: 21-Jun-2001 18:56 (Thursday) ::

Last night’s stats have been running for a very long time, so we’ve stopped
the stats server to let it catch up. I’m afraid it’s partly my fault…
I was investigating some possible causes for discrepancies in team credit,
and I ran a Very Large Query (VLQ) ™ at a bad time. As soon as stats
finish, I’ll turn the web back on.

The silver lining to this cloud is that I’ve found some work that should
be credited towards certain teams, and soon it will.

Sorry for the inconvenience.

:: 10-Jun-2001 21:12 (Sunday) ::

Hello again.

As long as we had tally off-line to expand the database, decibel and I
decided to perform some much needed maintenance. We rebuilt some of the
indexes on the tables we torture the most, and updated statistics on The
Big Table (that’s the detail table for RC5-64). For comparison, this one
table occupies almost 3GB, which is about 80% of the data in the stats
database, and nearly one fourth of the total disk on tally.

We had hoped this wouldn’t be necessary – we had hoped to be moved over
to the new server (blower) by now. Technology and time schedules prevented
us from completing the migration. The 2GB file size limit on Linux is
starting to be a real problem. All the stats folks are getting together
by phone soon to discuss the possibility of switching to Free BSD. This
option would give us more room to grow, but would not (easily) support
some of our current tools. I think we’re leaning that direction, but we
don’t want to backtrack that far just to find it’s not going to work.

Another maintenance task I performed was to purge many of the teams that
had been created but never used. This is mostly our fault – the
team-creation forms on the website make it easy for people to create a
team and then forget how to join it. Over the years, many people have
created a team several times in a row, never succeeding in joining. This
purge had never been performed in the past, to my knowledge, so it was
long overdue.

Rest assured, I have only purged those teams that never had any work
submitted, never had any members, and do not currently have any members.
I also preserved any empty team created in the last several months, just
in case the work is being saved up to submit in the near future. Even as
careful as I was, it turns out over half the teams in the table were
completely inactive. I saved an archive of the removed teams, but being
in a different SQL table, they will have less impact on statsrun. If by
some chance, you had created a team and I have removed it, please e-mail
me directly to have it replaced.

The next time we have occasion to modify the team-creation code, we’ll
add a date-created field to the teams table and establish a policy where
teams will be removed if no work is submitted in a certain time-frame.

That’s all the news that’s fit to print. Thanks again for all the cycles.

:: 09-Jun-2001 01:26 (Saturday) ::

Now we know href=”http://oldcgi.distributed.net/cgi/planarc.cgi?user=nugget&plan=2001-06-08.06:06″>something.

The stats database is running short on space. This may have contributed
to the unexpected outage, but more importantly, a Sybase database this
full and this heavily used bogs down terribly. We have the stats pages
turned off while we correct the problem. The stats run will be delayed
as a result (about 3 hours or so). Decibel will announce when it’s been
reactivated.

:: 13-Jul-2000 01:00 (Thursday) ::

Greetings, loyal cows!

If you’ve read bovine’s .plan
(http://n0cgi.distributed.net/cgi/planarc.cgi?user=bovine&plan=2000-07-06.03:43),
you know we’re experimenting with a larger minimum block size to see if
it will improve our network performance. Our tests so far are encouraging
(about a 30% reduction in traffic, I believe), and we’re discussing the
pros and cons of applying this change more widely, as well as some alternate
approaches.

In the mean time, I thought I’d point out that using larger blocks not
only reduces network traffic (yours and ours) but will boost your key
rate, especially on faster machines. The network and buffer management
components of the client take time away from the cores, just like every
other application on your computer. If you fetch/flush after every block,
you have 32 times less network overhead with a 2^33 as with a 2^28. It
takes just as long to submit a completed 2^33 as a 2^28, and it takes just
as long to connect and disconnect from the key servers, no matter how many
blocks are transferred in between. We recommend you fetch/flush only once
or twice a day. Your stats are based on the total work you submit, not
how often you submit it.

For example, a Mac G4 at 450MHz can do a 2^28 block in about 70 seconds.
If that machine was working only on 2^28 blocks, and updated after each
one, it would connect to our network over 1200 times a day. This example
goes beyond mere carelessness and borders on abuse.

Larger blocks also mean less data to transfer from keymaster to tally,
and less data for tally to, um, tally each day. Again, you still get the
same credit in stats whether you do 32 separate 2^28 blocks or 1 block of
2^33. The difference is in the bandwidth dcti uses, and the overhead on
your computer. The network bandwidth used by distributed.net (proxies,
keymaster, tally, website and ftp and e-mail lists like the .plan mailing)
is all donated – we’re guests. If we aren’t polite guests, we could be
asked to leave. Even if we don’t upset anyone, we’d like to make the most
of the bandwidth we have.

When we first started RC5-56, we decided on blocks of 2^28 keys because
the fastest personal computer money could buy was a Pentium 133. That
CPU could only do about 114,000 keys per second, which would finish a 2^28
block in about 40 minutes. Some older CPU’s may have a hard time completing
even a single 2^28 block in a 24 hour period. I’m sure it’s hard to feel
like any work is being done when it takes so long to see anything happen.

We certainly don’t want to alienate these participants by forcing them to
work on blocks so large it doesn’t finish for a month. Unfortunately,
that could eventually be the case if the people with faster machines don’t
voluntarily tune up their settings. As Moore’s Law
(http://www.tuxedo.org/~esr/jargon/html/entry/Moore’s-Law.html) continues
to push technology forward, it’s inevitable that we’ll need to raise the
ceiling beyond blocks of 2^33.

The other extreme, saving blocks up for weeks or months for a “mega-flush”
can also be harmful to the network. The sudden high volume of work
submitted all at once can saturate the proxies and the master. A big
enough mega-flush could theoretically take more than one day to be processed
into stats (please, don’t try!).

Though our biggest concern is the central network, these factors can also
improve performance between clients and personal proxies. I hope this
information will help you adjust your distributed.net clients in a way
that helps us all be more successful.

As always, thanks for all the cycles!

:: 07-Apr-2000 16:26 (Friday) ::

OK, I know I’m supposed to be getting OGR stats working, but I couldn’t
stand it any longer.

I’ve made some tweaks to the RC5-64 stats processing to eliminate some
unnecessary bottlenecks and resource usage. Things like eliminating
“distinct” in the same query with “group by”, since “group by” will return
distinct groups anyway. And like updating three fields at once instead of
three separate queries.

Also, I changed the ranking scripts so, in the case of a tie for rank,
the newest participant (largest id) will be listed first. This should
allow new members to see themselves in the ranking list sooner. Before,
they were sorted low-to-high ID.

Apart from the sorting, the only visible impact should be a slight (and
I do mean slight) increase in processing speed.

On a personal note, I wanted to apologize that the OGR stats are taking
so long. The rewrite is turning out to be more involved than I thought.
I’m trying to control the scope-creep, and also juggle my time with work
and now, buying a house. I have been making inroads, but I’ve still got
some work before we can start testing.

Thanks for your patience, and for all the cycles!

:: 13-Feb-2000 02:30 (Sunday) ::

Two quick updates on stats.

I’m working on enabling stats for OGR. It’s not as simple as it sounds,
because there’s no 100% we can measure ourselves against. There’s also
a lot more work units than any of our crypto contests to this point. I’m
out of town Sunday until Tuesday at a conference for my day-job. Don’t
know how much I’ll get done while I’m out. Just to let you know what to
expect, it’s very possible we won’t have OGR stats until OGR-24 is finished
– it’s a very short OGR ruler. You have my word, you’ll get them, even
if it’s after the fact.

I’ve been made aware that participant stats for Feb 10 are missing. I
can’t do much until Nugget returns after this weekend, but I’m confident
we’ll get them back. Don’t worry, none of your work has been lost.
Remember, the real RC5-64 work happens before stats sees anything.