staff blogs

distributed.net staff keep (relatively) up-to-date logs of their activities in .plan files. These were traditionally available via finger, but we've put them on the web for easier consumption.

2016-11-18

Backend server outages

Filed under: keyservers,stats — bovine @ 20:27 +00:00

There was a brief unplanned outage of the keymaster for about 8 hours earlier today, but the network is in the process of catching up right now and should be back to normal soon.

There will also be a planned outage of the stats server for this weekend due to building maintenance needing to service the air conditioning. Once service is restored on Monday, the stats server will again catch up.

We do not expect there to have been any significant loss of data or stats credit as a result of these events.

Thanks for your patience!

2015-07-22

Keymaster temporarily back online

Filed under: keyservers,stats — bovine @ 02:00 +00:00

Well, after nearly 12 days we’ve managed to get the old keymaster hardware back online. The problem ended up being some failing RAM, though the symptoms weren’t immediately obvious as that. Although we are currently still operating on the old hardware, we are planning to transition to newer hardware tomorrow. The newer hardware was shipped last week during the outage and is still in the process of being racked and powered up.

We do not expect tomorrow’s transition off the old hardware to be externally visible since our proxy network will have caught up and buffered enough work for the brief outage. Stats will have a multi-day gap during the time we were offline, but we expect that the statistics totals for 2015-07-22 will reflect a significantly higher-than-normal amount of work due to all of the backlogged data that had been buffered.

Additional measures are being made to ensure that an outage of this duration does not happen again.

Thanks again for your patience!

2015-07-10

Keymaster interruption again

Filed under: keyservers,stats — bovine @ 04:28 +00:00

We are experiencing another hardware issue with our central keymaster, and we do not yet have an estimate for it to be brought back online. However, our proxy network will continue to buffer completed work until the keymaster can be brought back online. Stats will also be delayed until that occurs. More information will be posted as it becomes available. Thanks for your patience!

2015-05-17

Keymaster interruption

Filed under: keyservers,stats — bovine @ 02:37 +00:00

We experienced some hardware issues with our keymaster server today, possibly due to a mechanical issue. It had a few hours of downtime on 2015-05-16 causing work to become backlogged and buffered on our keyserver network, which means that the stats for that day will be significantly lower than usual, however tomorrow’s stats should include all of that backlogged work. Newly submitted work is continuing to be processed without delay at this point.

Although the keymaster is back online for the moment, we are continuing to do remote hardware diagnostics. It sounds likely that we may need to replace a system case fan however. Thanks for your patience and support!

2013-11-16

November update

Filed under: clients,keyservers,project status — Mike Reed @ 01:38 +00:00

Dear friends,

We have completed over 96% of the work and are now close to wrapping up OGR-27. Some of you may have noticed some difficulty getting OGR packets earlier today, due to our need to begin recycling of the remaining stubs. As I write this, there are about 20 million left that need working on. It is likely that you may experience difficulty occasionally in the coming weeks when hoping to collect OGR stubs to process. This is due to the master key server being unable to speak and chew at the same time!

If you have a long memory, you may remember a note from me about needing to complete one more verification pass on a small number of stubs, about 2.2% of the total, that had been processed by buggy client versions. Now is the time! Some of you are already working on these stubs. One of our core contributors “Stream” has calculated that it will take us about 3 weeks at our current average (400 Gnode/sec) to complete the verification of the earlier stub spaces.

Today we also mark the 4000th day that we have been working on the RC5-72 project. We passed 3% completion only a few weeks ago and continue to make steady progress.

As ever, you can catch up with us in #distributed on IRC (irc.distributed.net).

Moo!

2012-06-07

World IPv6 day again!

Filed under: clients,keyservers — bovine @ 01:51 +00:00

As you might remember, one year ago we participated in World IPv6 day by permanently turning on IPv6 access for our main website (www.distributed.net). Today, we made the next step towards embracing IPv6 by adding AAAA records to our proxy keyserver DNS addresses and by releasing our first set of IPv6-capable dnetc clients on our pre-release download page, as version v2.9111.520. There should be no additional client configuration changes needed for the client to utilize IPv6 once your OS has been properly configured.

At the moment, we only have one of our keyservers currently configured to handle IPv6 connections and our pre-release page only has IPv6 clients for Solaris/SunOS, however we will be adding more over the next few days. Updated personal proxies that support IPv6 will also be made available in the coming weeks. We welcome bug reports on bugs.distributed.net if you encounter any connectivity problems with our new releases.

Happy IPv6

2010-11-17

rc5-72 workunit distribution restored

Filed under: keyservers — bovine @ 21:03 +00:00

There was a brief interruption in the distribution of RC5-72 workunits earlier today, however the problem has been fixed and clients should be again receiving workunits. OGR-27 was completely unaffected during this time. Additionally, RC5-72 results continued to be accepted without interruption.

The problem was caused by our migration to some new webserver hardware (unrelated to the new stats servers), which disrupted some of our alert monitoring systems.

Keep on crunching!

2010-01-24

bovine [24-Jan-2010 @ 06:17]

Filed under: keyservers — bovine @ 06:17 +00:00

:: 24-Jan-2010 06:17 GMT (Sunday) ::

We’re having some connectivity issues with our keymaster server, so
our proxy network is currently holding all submitted results until
connectivity is restored. As a result, stats did not run last night.
We hope to get things back in operational order soon. Thanks!

2009-11-29

bovine [29-Nov-2009 @ 06:15]

Filed under: keyservers — bovine @ 06:15 +00:00

:: 29-Nov-2009 06:15 GMT (Sunday) ::

Our keymaster is now back to normal operations and OGR-NG is sending
and receiving work again. We don’t think very many results, if any,
were lost as a result of the outage. The cause of the power failure
is still being investigated. Keep on crunching!

2009-11-28

bovine [28-Nov-2009 @ 19:48]

Filed under: keyservers — bovine @ 19:48 +00:00

:: 28-Nov-2009 19:48 GMT (Saturday) ::

There was a UPS or related power failure at our keymaster that caused
some data inconsistency in our OGR-NG database. As a result, the
project was automatically suspended until manual validation of the
database can be completed.

We hope to be able to bring OGR-NG back online within a day if
everything goes well. Unfortunately, some OGR-NG results that were
returned during this inconsistent period may have been lost.
Meanwhile, RC5-72 is continuing to run and is unaffected. Thanks for
your patience and support!