:: 21-Apr-2004 15:27 GMT (Wednesday) ::

A word or two on the status of ogr-verification. As you know, We’re
having to restore from backups as of February. Unfortunately, when that
backup was taken, there was a problem and the ogr-ver data was not up to
date. The end result is that we have several months of logs to go
though to catch up.

I haven’t been able to do many test runs without something else running
on the box, but it looks like it’s going to be quite a bit faster than
the old box. It’s taking around 2-2.5 hours to process a day now, which
is already an improvement and stats are still catching up, so we’re
competing for I/O.

:: 18-Apr-2004 22:28 GMT (Sunday) ::

The restore is done and fritz is now churning through log files. I’m about to
turn apache back on. Something to keep in mind is that many user changes take
effect on the log date that the change was made. So if you retired an account
or joined a team on March 3rd, you won’t see the change take effect until the
daily stats run for March 3rd.

:: 18-Apr-2004 15:40 GMT (Sunday) ::

Ooops. Looks like we forgot to restore an incremental backup from just before
the crash. In order to restore that backup and have it take effect, all changes
participants have made since fritz went online well unfortunately be lost. We
will also have to re-run all stats for the past two months or so, which will
take a few days.

Sorry for the inconvenience.

:: 17-Apr-2004 00:03 GMT (Saturday) ::

Stats are back up, but fritz (the new statsbox) is currently having some issues
with sending mail. Because of this, the mail me your password button won’t be
of much use.

:: 16-Apr-2004 23:30 GMT (Friday) ::

Ok, the new statsbox should now be online and accessible at the usual
http://stats.distributed.net/ after the DNS changes finish
circulating. Although it is believed that everything is caught up and
working, there may be some minor issues still remaining so be patient.

:: 11-Apr-2004 14:51 GMT (Sunday) ::

The new statsbox is built out and I’ve got the backup of the statsdb loaded
and audited. Apache and PHP are up and running and I’ve got our admin
scripts loaded and going.

I’ve done some preliminary tuning of PostgreSQL and I’m not seeing any
disk-based sort_memory used during the normal stats processing process.

Some timing comparisons between blower and the new box:

Sample run from blower’s logs (45 minutes total):
00:49 (statsbox-iii/r72) Beginning daily processing routines
01:10 (statsbox-iii/r72) Daily processing for 20040121 has completed
01:10 (statsbox-iii/ogr) Beginning daily processing routines
01:34 (statsbox-iii/ogr) Daily processing for 20040121 has completed

On the new box (9 minutes total):
19:40 (statsbox-iv/r72) Beginning daily processing routines
19:46 (statsbox-iv/r72) Daily processing for 20040205 has completed
19:46 (statsbox-iv/ogr) Beginning daily processing routines
19:49 (statsbox-iv/ogr) Daily processing for 20040205 has completed

I’m seeing raw hourly log import times in the 20-30 second range. Overall
it’s taking the new box ~32 minutes to import an entire day’s worth of logs
for both projects combined.

This is giving us a total daily processing time of 41 minutes with log
saturation from the keymaster. At that rate, stats should be current
before too long.

Thanks again for your patience, everyone. We’re plugging away at getting
this box up and available again.

:: 07-Apr-2004 16:47 GMT (Wednesday) ::

The new statsbox has arrived from asacomputers.com — It’s plugged in and I’ve
got FreeBSD installed and running (albeit a bit bare at the moment). We’re
just going to chug through getting everything running and our data loaded into
the database over the next day or so. Assuming everything goes smoothly, we’ll
have stats back up in short order.

Front view, times two, open and closed:
http://www.slacker.com/photos/computers/open_and_closed

Top down, deconstructed:
http://www.slacker.com/photos/computers/inside

It’s a sweet little box (but loud!)

Mem: 15M Active, 375M Inact, 150M Wired, 32K Cache, 199M Buf, 3231M Free
Swap: 8192M Total, 8192M Free

Filesystem Size Used Avail Capacity Mounted on
/dev/twed0s1a 989M 25M 885M 3% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/twed0s1g 40G 1.4M 37G 0% /home
/dev/twed0s1e 3.9G 12K 3.6G 0% /tmp
/dev/twed0s1f 97G 893M 88G 1% /usr
/dev/twed1s1d 541G 4.0K 498G 0% /usr/local/raid10
/dev/twed0s1d 31G 862K 29G 0% /var

CPU: AMD Opteron(tm) Processor 244 (1793.33-MHz K8-class CPU)
Origin = “AuthenticAMD” Id = 0xf58 Stepping = 8
Features=0x78bfbff
AMD Features=0xe0500000
real memory = 4211015680 (4015 MB)
avail memory = 4006965248 (3821 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs

:: 02-Apr-2004 09:49 GMT (Friday) ::

Last night BovineOne brought the old, dead statsbox over to my house to see
what I could do with it. Moose has already exhausted all the recovery
options I’d know about (and quite a few more, I’m sure) with no luck.

I thought maybe I could at least get Knoppix booted up so I could get dnetc
going again, but had little luck. Knoppix panics trying to probe the SCSI
adapters.

At the very least I’ll be able to pull the database dump off the DLT,
although I think that it’s older than the copy I’m working with currently
from Decibel.

http://www.slacker.com/photos/computers/IMG_1792 for a pic of Slacker NOC,
if anyone’s curious.

As far as I know, we’re still looking at a Tuesday ship date on the new
server. I’ve asked if that date could be pushed up, but no word yet from
the vendor.

Lastly, I got bored yesterday and coded up an RSS feed of .plans. which is
available at http://n0cgi.distributed.net/cgi/rss-plans.cgi or on
LiveJournal at http://www.livejournal.com/userinfo.bml?user=dnetplans

Moo.

:: 01-Apr-2004 12:32 GMT (Thursday) ::

Hi everyone.

I’d like to introduce myself to everyone. My name is Thorsten Wolf and I’m new
to the d.net team.

Since we’re all waiting for the new statsbox, I’d like to make a public call to
everyone out there willing to help creating a new design for the stats pages.

We want to have it look similar to the d.net standard design, but new ideas
will be appreciated.

So send me your suggestions (please make a screenshot of your HTML instead of
sending me plain HTML).

Send them to: href=”mailto:t_wolf@distributed.net”>t_wolf@distributed.net

:: 01-Apr-2004 09:51 GMT (Thursday) ::

Since the delivery timeline on the Dual Opteron box we ordered keeps
slipping and we realize that it’s crucially important that we gets stats
back up and running in a reasonable timeframe, I’ve just cancelled the
order on the Opteron box. We all decided it would make more sense to buy a
machine locally so we wouldn’t have to wait for shipping delays.

This morning I headed over to CompUSA and picked up a nice eMachines
Minitower. Since we’d lose the 90 days of telephone support if we
installed FreeBSD on the box, we’re just going to stick with Windows ME
which was preinstalled on the machine.

As soon as I’ve had a chance to get Access installed we should be ready to
start serving up stats again. Thanks again, everyone, for your patience
through this unexpected downtime. I’m excited about getting something up
and running now instead of just waiting for that other box!

(By the way, does anyone know the proper modem initialization string for
the eMachines winmodem? I need to get the box online asap)