distributed.net staff keep (relatively) up-to-date logs of their activities in .plan files. These were traditionally available via finger, but we've put them on the web for easier consumption.
Well, after nearly 12 days we’ve managed to get the old keymaster hardware back online. The problem ended up being some failing RAM, though the symptoms weren’t immediately obvious as that. Although we are currently still operating on the old hardware, we are planning to transition to newer hardware tomorrow. The newer hardware was shipped last week during the outage and is still in the process of being racked and powered up.
We do not expect tomorrow’s transition off the old hardware to be externally visible since our proxy network will have caught up and buffered enough work for the brief outage. Stats will have a multi-day gap during the time we were offline, but we expect that the statistics totals for 2015-07-22 will reflect a significantly higher-than-normal amount of work due to all of the backlogged data that had been buffered.
Additional measures are being made to ensure that an outage of this duration does not happen again.
Thanks again for your patience!
We are experiencing another hardware issue with our central keymaster, and we do not yet have an estimate for it to be brought back online. However, our proxy network will continue to buffer completed work until the keymaster can be brought back online. Stats will also be delayed until that occurs. More information will be posted as it becomes available. Thanks for your patience!
We experienced some hardware issues with our keymaster server today, possibly due to a mechanical issue. It had a few hours of downtime on 2015-05-16 causing work to become backlogged and buffered on our keyserver network, which means that the stats for that day will be significantly lower than usual, however tomorrow’s stats should include all of that backlogged work. Newly submitted work is continuing to be processed without delay at this point.
Although the keymaster is back online for the moment, we are continuing to do remote hardware diagnostics. It sounds likely that we may need to replace a system case fan however. Thanks for your patience and support!
We have completed over 96% of the work and are now close to wrapping up OGR-27. Some of you may have noticed some difficulty getting OGR packets earlier today, due to our need to begin recycling of the remaining stubs. As I write this, there are about 20 million left that need working on. It is likely that you may experience difficulty occasionally in the coming weeks when hoping to collect OGR stubs to process. This is due to the master key server being unable to speak and chew at the same time!
If you have a long memory, you may remember a note from me about needing to complete one more verification pass on a small number of stubs, about 2.2% of the total, that had been processed by buggy client versions. Now is the time! Some of you are already working on these stubs. One of our core contributors “Stream” has calculated that it will take us about 3 weeks at our current average (400 Gnode/sec) to complete the verification of the earlier stub spaces.
Today we also mark the 4000th day that we have been working on the RC5-72 project. We passed 3% completion only a few weeks ago and continue to make steady progress.
As ever, you can catch up with us in #distributed on IRC (irc.distributed.net).
As you might remember, one year ago we participated in World IPv6 day by permanently turning on IPv6 access for our main website (www.distributed.net). Today, we made the next step towards embracing IPv6 by adding AAAA records to our proxy keyserver DNS addresses and by releasing our first set of IPv6-capable dnetc clients on our pre-release download page, as version v2.9111.520. There should be no additional client configuration changes needed for the client to utilize IPv6 once your OS has been properly configured.
At the moment, we only have one of our keyservers currently configured to handle IPv6 connections and our pre-release page only has IPv6 clients for Solaris/SunOS, however we will be adding more over the next few days. Updated personal proxies that support IPv6 will also be made available in the coming weeks. We welcome bug reports on bugs.distributed.net if you encounter any connectivity problems with our new releases.
There was a brief interruption in the distribution of RC5-72 workunits earlier today, however the problem has been fixed and clients should be again receiving workunits. OGR-27 was completely unaffected during this time. Additionally, RC5-72 results continued to be accepted without interruption.
The problem was caused by our migration to some new webserver hardware (unrelated to the new stats servers), which disrupted some of our alert monitoring systems.
Keep on crunching!
:: 24-Jan-2010 06:17 GMT (Sunday) ::
We’re having some connectivity issues with our keymaster server, so
our proxy network is currently holding all submitted results until
connectivity is restored. As a result, stats did not run last night.
We hope to get things back in operational order soon. Thanks!
:: 29-Nov-2009 06:15 GMT (Sunday) ::
Our keymaster is now back to normal operations and OGR-NG is sending
and receiving work again. We don’t think very many results, if any,
were lost as a result of the outage. The cause of the power failure
is still being investigated. Keep on crunching!
:: 28-Nov-2009 19:48 GMT (Saturday) ::
There was a UPS or related power failure at our keymaster that caused
some data inconsistency in our OGR-NG database. As a result, the
project was automatically suspended until manual validation of the
database can be completed.
We hope to be able to bring OGR-NG back online within a day if
everything goes well. Unfortunately, some OGR-NG results that were
returned during this inconsistent period may have been lost.
Meanwhile, RC5-72 is continuing to run and is unaffected. Thanks for
your patience and support!
:: 12-Mar-2009 22:12 GMT (Thursday) ::
There was an unplanned network outage earlier today that caused the
unavailability of our website and DNS services. No work should have
been lost, due to the buffering nature of our network. We will be
working on improving the redundancy of our DNS over the next couple of
days to avoid that part of the problem in the future.
Additionally, we are aware that the historical stats for OGR-26 are
currently not displaying accurate data. During the enablement of
OGR-27 stats, some data archive tables were inadvertently truncated.
We will be working on re-loading those historical OGR-26 stats from
archive logs over the next week.
In other news, if you happen to be attending the South By Southwest
festival in Austin, Texas next week you should feel free to stop by
the Nuclear Taco Night event on Monday March 16 from 6-9 PM. A few of
the distributed.net staff will be working behind the skillets to serve
these extremely spicy tacos to anyone masochistic enough for them: