:: 11-Apr-2004 14:51 GMT (Sunday) ::

The new statsbox is built out and I’ve got the backup of the statsdb loaded
and audited. Apache and PHP are up and running and I’ve got our admin
scripts loaded and going.

I’ve done some preliminary tuning of PostgreSQL and I’m not seeing any
disk-based sort_memory used during the normal stats processing process.

Some timing comparisons between blower and the new box:

Sample run from blower’s logs (45 minutes total):
00:49 (statsbox-iii/r72) Beginning daily processing routines
01:10 (statsbox-iii/r72) Daily processing for 20040121 has completed
01:10 (statsbox-iii/ogr) Beginning daily processing routines
01:34 (statsbox-iii/ogr) Daily processing for 20040121 has completed

On the new box (9 minutes total):
19:40 (statsbox-iv/r72) Beginning daily processing routines
19:46 (statsbox-iv/r72) Daily processing for 20040205 has completed
19:46 (statsbox-iv/ogr) Beginning daily processing routines
19:49 (statsbox-iv/ogr) Daily processing for 20040205 has completed

I’m seeing raw hourly log import times in the 20-30 second range. Overall
it’s taking the new box ~32 minutes to import an entire day’s worth of logs
for both projects combined.

This is giving us a total daily processing time of 41 minutes with log
saturation from the keymaster. At that rate, stats should be current
before too long.

Thanks again for your patience, everyone. We’re plugging away at getting
this box up and available again.

:: 07-Apr-2004 16:47 GMT (Wednesday) ::

The new statsbox has arrived from asacomputers.com — It’s plugged in and I’ve
got FreeBSD installed and running (albeit a bit bare at the moment). We’re
just going to chug through getting everything running and our data loaded into
the database over the next day or so. Assuming everything goes smoothly, we’ll
have stats back up in short order.

Front view, times two, open and closed:
http://www.slacker.com/photos/computers/open_and_closed

Top down, deconstructed:
http://www.slacker.com/photos/computers/inside

It’s a sweet little box (but loud!)

Mem: 15M Active, 375M Inact, 150M Wired, 32K Cache, 199M Buf, 3231M Free
Swap: 8192M Total, 8192M Free

Filesystem Size Used Avail Capacity Mounted on
/dev/twed0s1a 989M 25M 885M 3% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/twed0s1g 40G 1.4M 37G 0% /home
/dev/twed0s1e 3.9G 12K 3.6G 0% /tmp
/dev/twed0s1f 97G 893M 88G 1% /usr
/dev/twed1s1d 541G 4.0K 498G 0% /usr/local/raid10
/dev/twed0s1d 31G 862K 29G 0% /var

CPU: AMD Opteron(tm) Processor 244 (1793.33-MHz K8-class CPU)
Origin = “AuthenticAMD” Id = 0xf58 Stepping = 8
Features=0x78bfbff
AMD Features=0xe0500000
real memory = 4211015680 (4015 MB)
avail memory = 4006965248 (3821 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs

:: 02-Apr-2004 09:49 GMT (Friday) ::

Last night BovineOne brought the old, dead statsbox over to my house to see
what I could do with it. Moose has already exhausted all the recovery
options I’d know about (and quite a few more, I’m sure) with no luck.

I thought maybe I could at least get Knoppix booted up so I could get dnetc
going again, but had little luck. Knoppix panics trying to probe the SCSI
adapters.

At the very least I’ll be able to pull the database dump off the DLT,
although I think that it’s older than the copy I’m working with currently
from Decibel.

http://www.slacker.com/photos/computers/IMG_1792 for a pic of Slacker NOC,
if anyone’s curious.

As far as I know, we’re still looking at a Tuesday ship date on the new
server. I’ve asked if that date could be pushed up, but no word yet from
the vendor.

Lastly, I got bored yesterday and coded up an RSS feed of .plans. which is
available at http://n0cgi.distributed.net/cgi/rss-plans.cgi or on
LiveJournal at http://www.livejournal.com/userinfo.bml?user=dnetplans

Moo.

:: 01-Apr-2004 12:32 GMT (Thursday) ::

Hi everyone.

I’d like to introduce myself to everyone. My name is Thorsten Wolf and I’m new
to the d.net team.

Since we’re all waiting for the new statsbox, I’d like to make a public call to
everyone out there willing to help creating a new design for the stats pages.

We want to have it look similar to the d.net standard design, but new ideas
will be appreciated.

So send me your suggestions (please make a screenshot of your HTML instead of
sending me plain HTML).

Send them to: href=”mailto:t_wolf@distributed.net”>t_wolf@distributed.net

:: 01-Apr-2004 09:51 GMT (Thursday) ::

Since the delivery timeline on the Dual Opteron box we ordered keeps
slipping and we realize that it’s crucially important that we gets stats
back up and running in a reasonable timeframe, I’ve just cancelled the
order on the Opteron box. We all decided it would make more sense to buy a
machine locally so we wouldn’t have to wait for shipping delays.

This morning I headed over to CompUSA and picked up a nice eMachines
Minitower. Since we’d lose the 90 days of telephone support if we
installed FreeBSD on the box, we’re just going to stick with Windows ME
which was preinstalled on the machine.

As soon as I’ve had a chance to get Access installed we should be ready to
start serving up stats again. Thanks again, everyone, for your patience
through this unexpected downtime. I’m excited about getting something up
and running now instead of just waiting for that other box!

(By the way, does anyone know the proper modem initialization string for
the eMachines winmodem? I need to get the box online asap)

:: 31-Mar-2004 23:53 GMT (Wednesday) ::

There was an unplanned power outage at the facility hosting the
keymaster, but fortunately most of our fullservers had large enough
buffers to last for the entire duration and avoid any interruptions.

:: 25-Mar-2004 23:06 GMT (Thursday) ::

Just to answer some questions… Apologies for the acronym overload…

The old statsbox (aka “blower” or “statsbox3”) was a quad xeon 450mhz
with 2GB RAM and Dell Perc RAID. It had five SCSI disks configured in a
2xRAID1 3xRAID5 configuration.

The new statsbox has been ordered (as yet unnamed). It will be:

Dual Opteron 1.8GHz
4GB RAM
8X200GB SATA (Hotswap) on 3Ware 8506 RAID Controller
3U Case
No keyboard :)

The current plan is to split the 8 drives as:
2xRAID1 + Hot Spare
4xRAID10 + Hot Spare

We’ll of course be staying with PostgreSQL and FreeBSD, although bumping
to FreeBSD 5.x for amd64 support.

I expect to get an ETA for delivery tomorrow (Friday 26-March UTC-6)

:: 24-Mar-2004 20:37 GMT (Wednesday) ::

Thanks in part to user donations (including one VERY generous donation)
we’re close to being able to order a new stats server. After some
internal debate on the best approach, the current plan is to pick up a
dual opteron box and load it with memory and drives. Traditionally,
statsbox has been i/o bound on disk and memory but not very demanding of
CPU. An Opteron solution sounds like a good target platform for what we
need.

I spent a lot of today borrowing a surrogate opteron box from Bovine to
validate that postgresql and freebsd 5.x are a viable platform. I’ve also
confirmed with Doug White and Vinod Kashyap that 3Ware support
in FreeBSD 5.x is stable and reliable.

We’re also eager to move to a smaller sized case — blower was in a
gigantic Dell 6400 series case which limited our options for alternative
colos if we ever decided to move servers around. I think we can stuff
everything we need into a 3U chassis.

I’ve got a price quote that seems agreeable and I hope to place the order
tomorrow. It’s unlikely this will get us back online before next week,
though.

I’ll post more as the ordering+building+deployment progresses…

Thanks again everyone for your patience and understanding.

:: 19-Mar-2004 18:51 GMT (Friday) ::

Blower suffered a drive failure today. The bad news is that it’s refusing to
rebuild the raid array. The worse news is that a non-critical table in the
database has been corrupted.

We’re in the process of ordering a replacement for blower. We don’t have an ETA
for it yet.

For right now, I have web access turned off completely. If possible I’ll be
turning web access back on, but disabling updates to data (ie: changing teams,
etc), because there’s no way to know when that data might go poof.

Hopefully we’ll be able to get replacement hardware soon. On the bright side,
we’re looking at a dual Opteron machine with RAID10 for the database and RAID1
for the database logs, so the new box should really scream.

:: 07-Jan-2004 16:14 GMT (Wednesday) ::

Sorry, having some stats issues. They’ll be back up soon.