:: 31-Mar-2004 23:53 GMT (Wednesday) ::
There was an unplanned power outage at the facility hosting the
keymaster, but fortunately most of our fullservers had large enough
buffers to last for the entire duration and avoid any interruptions.
staff blogs
distributed.net staff keep (relatively) up-to-date logs of their activities in .plan files. These were traditionally available via finger, but we've put them on the web for easier consumption.
2004-03-31
bovine [31-Mar-2004 @ 23:53]
2004-03-25
nugget [25-Mar-2004 @ 23:06]
:: 25-Mar-2004 23:06 GMT (Thursday) ::
Just to answer some questions… Apologies for the acronym overload…
The old statsbox (aka “blower” or “statsbox3”) was a quad xeon 450mhz
with 2GB RAM and Dell Perc RAID. It had five SCSI disks configured in a
2xRAID1 3xRAID5 configuration.
The new statsbox has been ordered (as yet unnamed). It will be:
Dual Opteron 1.8GHz
4GB RAM
8X200GB SATA (Hotswap) on 3Ware 8506 RAID Controller
3U Case
No keyboard :)
The current plan is to split the 8 drives as:
2xRAID1 + Hot Spare
4xRAID10 + Hot Spare
We’ll of course be staying with PostgreSQL and FreeBSD, although bumping
to FreeBSD 5.x for amd64 support.
I expect to get an ETA for delivery tomorrow (Friday 26-March UTC-6)
2004-03-24
nugget [24-Mar-2004 @ 20:37]
:: 24-Mar-2004 20:37 GMT (Wednesday) ::
Thanks in part to user donations (including one VERY generous donation)
we’re close to being able to order a new stats server. After some
internal debate on the best approach, the current plan is to pick up a
dual opteron box and load it with memory and drives. Traditionally,
statsbox has been i/o bound on disk and memory but not very demanding of
CPU. An Opteron solution sounds like a good target platform for what we
need.
I spent a lot of today borrowing a surrogate opteron box from Bovine to
validate that postgresql and freebsd 5.x are a viable platform. I’ve also
confirmed with Doug White and Vinod Kashyap that 3Ware support
in FreeBSD 5.x is stable and reliable.
We’re also eager to move to a smaller sized case — blower was in a
gigantic Dell 6400 series case which limited our options for alternative
colos if we ever decided to move servers around. I think we can stuff
everything we need into a 3U chassis.
I’ve got a price quote that seems agreeable and I hope to place the order
tomorrow. It’s unlikely this will get us back online before next week,
though.
I’ll post more as the ordering+building+deployment progresses…
Thanks again everyone for your patience and understanding.
2004-03-19
decibel [19-Mar-2004 @ 18:51]
:: 19-Mar-2004 18:51 GMT (Friday) ::
Blower suffered a drive failure today. The bad news is that it’s refusing to
rebuild the raid array. The worse news is that a non-critical table in the
database has been corrupted.
We’re in the process of ordering a replacement for blower. We don’t have an ETA
for it yet.
For right now, I have web access turned off completely. If possible I’ll be
turning web access back on, but disabling updates to data (ie: changing teams,
etc), because there’s no way to know when that data might go poof.
Hopefully we’ll be able to get replacement hardware soon. On the bright side,
we’re looking at a dual Opteron machine with RAID10 for the database and RAID1
for the database logs, so the new box should really scream.