:: 29-Sep-1999 22:25 (Wednesday) ::

I massaged the stats database today. Made a few changes which may or
may not correct our recent difficulties.

First, I gave us another 2GB of space in the stats database. We were
getting a bit low (not uncomfortably so, but still…) on space in
the database and I figured that now was as good a time as any to
make more room. This puts us at 4GB reserved for data, 1GB reserved
for transaction logs.

Second, I completely re-built the main rc5_64_master table. The old
table allowed for nulls in team field, and apparently it’s a bit more
efficient if nulls aren’t allowed in fields, so the new copy of the
table doesn’t allow nulls in any field. There’s no functional difference
for us, since “team 0” is used to denote no team, and it’s impossible to
have a null id, date, or block count.

A necessary aspect of rebuilding rc5_64_master was re-creating all the
indexes as well.

We won’t really know if this helped until tonight’s run (in an hour or
so) — so cross your fingers.

with apache down, everything’s running faster than it was, so I’m hopeful.

:: 29-Sep-1999 04:00 (Wednesday) ::

Well, our luck just seems to be against us as far as stats-box goes. As Nugget
mentioned in his .plan, we’ve tried a few more things, and they don’t seem to
be working. And right now, I can’t get to the Sybase online manual, so I can’t
really try anything new right now. Fear not though, we still have a few more
tricks up our sleeves (my current theory is that Sybase is very unhappy that
we’ve nearly filled a couple of the database devices… note that these aren’t
directly related to physical devices).

But, amidst all this stats-anguish, there is good news! Since many of you may
not have seen it on the stats main page today, here’s the relevant bit:

33,195,551 blocks were completed yesterday (0.048% of the keyspace)
at a sustained rate of 103,134,987 KKeys/sec!

Yes, you read that correct, and it’s not an error. We averaged 103Gk/s on
Monday. To help put this into perspective…

To put this in perspective, this rate is 217 times faster than our first day’s
rate of 475 *M*k/s. Even if you look to one month after the start of rc5-64,
when our rate had stabilized at about 8Gk/s, yesterday’s rate is still 13
times faster.

And, for those who are dying to know, at 103Gk/s it would take us 4.9 years
to polish off the remaining keyspace.

So, even though stats box isn’t too happy right now, our network is doing great!

:: 28-Sep-1999 20:39 (Tuesday) ::

Note to self: SQL written while listening to Rage Against the Machine will
need to be cleaned up at a later date.

I’ve got a couple working theories on the stats slowdown, and some test
code to run some benchmarks. Unfortunately I noticed that the monday-
morning statsdb backup didn’t happen monday as it should have, so I’m
in a holding pattern as the tables bcp out to a safe place. As soon
as this is wrapped up I’ll get to run some time tests to see if today’s
changes fix the problem.

The “I searched for my stats and got nugget’s” problem is fixed. Most
of the php3 pages have defaults where they show id #1 or team #1 if no
valid data is provided to them. I made a change to psearch.php3 last
night that jumps straight to psummary.php3 if the search only yields
one email address. It seems that psearch was confusing a lot of people
who thought didn’t realize that they could click on the email address
and get to a full summary page. I screwed up when I made the change,
but didn’t notice when I tested it because (drum roll) I tested it with
my email address. In reality, it wasn’t passing the id properly to
psummary, but I thought it was because the default was me. oops. :)

It’s fixed now, and should be working just fine.

I’m pretty sure we’re on the right track, that the recent slowdown in
processing is triggered/exacerbated by the fact that as of last week,
normal stats viewing now touches the “big” rc5_64_master table. Until
we added phistory, no stats pages actually hit the main table.
As explained in decibel’s .plan, he disabled the phistory pages to see
if that would help. As you know, it didnt’, but that may very well be
because we missed a spot. The “today this participant did foo% of their
maximum” line also hits the rc5_64_master table, and we overlooked that.
So, you guys have been hitting the rc5_64_master table this whole time,
even with phistory disabled.

Assuming this is indeed the cause of the problem, I made changes to the
php3 code today which may eliminate the conflict. If the changes do not
help (and tonight’s run also takes way too long), the next step will be
to disable phistory and the “today compared to best day” psummary line
during the update. (not all the time, just when the update runs).

If that doesn’t help, well… we’ll burn that bridge when we come to it.

:: 27-Sep-1999 20:09 (Monday) ::

Finally have a chance to do some forensics on statsbox, I’m starting off with
the normal “dbcc checkdb” and “update statistics” just to make sure
everything is tidy at the lowest levels. I also see that we’re getting close
to capacity (only about 150mb free on the main statsdb device) so I’m going
to slice off another half gig or so of space for the data tables.

I think we’re within a few months of needing another drive in there,
the main RC5_64_master table is nearing 20 million rows.

The goal today is to determine the cause of the daily update slowdowns that
started last week, although I’d settle for prodding at the box and having it
automagically start working properly. :)

:: 27-Sep-1999 15:51 (Monday) ::

Ok… I’ve been pretty busy lately, so I haven’t had much of a chance
to work on stats. Right now I am helping GregH out with some OGR
testing. So far, OGR is looking pretty nice. Here is a teaser from
my test full proxy:
ogr r=4998/5000, d=0/0, 0.0 Mkeys/sec, tot=89034

Yes we know that the rate screwed up.

I don’t know when the release is actually planned though, so don’t ask.

:: 24-Sep-1999 14:52 (Friday) ::

There is something quite unhealthy on statsbox at the moment which is
causing daily updates to take an extremely long time to complete. Last
night we disabled the phistory queries to see if that would alleviate the
problem, and it didn’t seem to help. I’m hoping to have time tomorrow to
really take a close look at the issue. In the meantime, we’re going to
temporarily start restricting access to stats during portions of the
update.

I’ll post more as I know what’s going on. It might turn out to be
something simple… no way to know yet.

:: 24-Sep-1999 01:40 (Friday) ::

Well, there’s good news and there’s bad news. The good news is that
I think we found out what was causing the major slowdown on
stats-box (yesterday’s stats-run took about 20 hours to complete,
obviously not a good thing }:8( ).

Working off of a NuggetHunch, I’ve disabled the participant history
functions. It’s too soon to tell for sure, but this seems to have
brought the currently running update back up to normal speed. Of
course, this means that you can’t check your history right now.

The good news is that if this is the problem, we can probably
cure it by adding memory to the box (256 meg ECC DIMMS, for those
of you in a generous mood };8P ). The current theory is that the
history query is killing our caching, since it has to sift through
all ~20 million records in the main rc5 table.

We should be able to tell in a few hours if this actually solved
the problem or not. Thanks for your patience!

:: 20-Sep-1999 05:57 (Monday) ::

Finally, some OGR news! An OGR test master is now operational. For the
latest news, see:

http://www.distributed.net/~gregh/ogr-todo.html

:: 19-Sep-1999 13:23 (Sunday) ::

As some of you have noticed, the classic “win32gui” clients have
recently been removed from the clients.html and select.cgi link pages.
The separate “GUI” client was originally deemed necessary because the
only alternative was a console-based version that had a number of
significant limitations:

– required significant user involvement for installation.

– no user interface for client configuration.

– could not be used in environments needing an unobtrusive, hidden
mode of operation that did not interrupt normal logouts

– inability to minimize to the system tray

– no extras, such as “moo sounds” or “log graphing” or pretty
percentage bars.

HOW IT WAS
———-
We have maintained separate releases of both the GUI and CLI Win32
clients for quite awhile now, with our client development focused
around the common CLI-based source code, and the GUI client involving
a fair amount of clever wrappers around it. Although this made
development of the separate versions easier than they would have been
otherwise, there was always a significant delay in the release of the
Win32GUI relative to the equivalent Win32CLI. Additionally, the GUI
client has always suffered from numerous obscure operational bugs
ranging from benign to severe issues.

Meanwhile, each newer release of the common code-base tried to make
incremental code infrastructure changes. This new infrastructure made
construction of other “GUI wrappers” with less effort than what was
necessary to produce the level of integration that the Win32GUI had.
However, because of these major adjustments in the organization of the
common code-base, the Win32GUI began to lag even further behind and
grew kludgier since much of the back-bending efforts that were
previously necessary no longer were. The Win32GUI portion of the code
base really just needed a complete rewrite.

Additionally, the Win32CLI releases had slowly been getting enhanced
with each successive release. The Win32CLI was no longer a console
app at all, and had actually become a GUI-based application itself. A
reasonably friendly textual configuration menu-set was now available
for all clients derived from the common code-base. The Win32CLI was
also able to cleanly start and stop in hidden environments, run as an
NT service, or minimize itself to the system tray. More importantly,
since the new “GUI-ish CLI” was directly compiled from the common
code-base, and the Win32CLI was traditionally one of our primary
development environments, the release delays and obscure GUI-specific
bugs were minimized.

WHAT IS HAPPENING NOW
———————
A new “self-installer” version of the Win32CLI will now be available
for download to make installations simplified. This installer is
nearly identical to the installer that the Win32GUI came packaged in.
Significant effort has been put into making “upgrading” from a prior
Win32GUI installation easy.

The Win32CLI will also now ship with a WinHelp HLP file to make
introductions easier for first time users. Additionally, the
distributed.net Log Visualizer is a new utility that is based heavily
upon code derived from the log grapher portion of the Win32GUI. Full
source code for the Log Visualizer will be available for download on
our source code page.

Although the last released version of the Win32GUI will continue to
operate for the foreseeable future, there will be no newer versions of it
released, and support questions regarding it will generally recommend
switching to the CLI. The Win32GUI will continue to be available on
our FTP distribution sites, although it will not be in the
“current-clients” directory.

We will continue to list and support GUI clients for the other
operating systems that we currently provide separate GUI clients for
(MacOS and RiscOS), however they too will similarly become unnecessary
once our plans (described below) are fully completed.

THE FUTURE
———-
It had always been part of our plans to try to slowly move away from
producing a separate GUI and CLI versions. This included the Win32GUI
and Win32CLI, the MacOS GUI and FBA clients, OS/2 GUI and CLI clients,
RiscOS GUI and CLI, and possible X11/Gnome/etc-based clients.
Instead, we planned to create “detached” front-end GUI applications
that could be run alongside the traditional CLI clients. This would
prevent the complexity of various OS-specific GUI programming
requirements from affect the design of the common client source pool.
Additionally, this detached mechanism might allow for remote
configuration and monitoring to occur over a local network, which has
always been a popularly requested feature for our clients.

So with all of this said, we will now be diverting our efforts from
attempting to maintain separate GUI-specific versions of our clients
and toward achieving our separated GUI frontend and CLI back-end
model. We are beginning development of an extensible communications
interface that will allow multi-platform GUI configuration utilities
to be constructed and arbitrarily configure and monitor remote or
local clients. The projected availablilty of these new clients cannot
be easily estimated at this time, though we will definitely make
future announcements when further information is known.

:: 17-Sep-1999 21:19 (Friday) ::

Since Silby’s temporary departure to concentrate on school,
I am helping out with help@distributed.net support. I had
been helping Silby over the past 6 months or so in this role,
but now that role has gotten larger. I am also helping out
with documentation.

—–BEGIN PGP SIGNATURE—–
Version: PGPfreeware 6.5.1 for non-commercial use

iQA/AwUBN+KuhtllZllpF5qpEQJPRQCdFsDKQytKaFXheokLSkotJd5Km/oAoLhm
2yr/i7RVW1A4TNfDTqpHC/Sg
=p5pb
—–END PGP SIGNATURE—–