Submitted by defilipj on Wed, 04/15/2015 - 07:35

Updated at 7:22am

The power shutdown had exceeded its scheduled window. The new estimate is power will be restored at 8am. The outage is limited to the emergency generator circuit, and most building power is up.

At this point all Cosine services are up, however temperature is rising in the datacenter, and if it does not hold at a reasonable level, we may need to start shutting down non-critical servers.

**************

The power was terminated at 5:10am.

Most Cosine Services remained live.

Networking in ALS was down from 5:10am to approximately 5:45AM, this was because the networking equipment was unexpectedly on the emergency generator circuit. The affected equipment was moved to house power and a downtime will be scheduled to move it back to the generator circuit at a later date.

The cluster was shut down for approximately 30 minutes, 20 minutes longer than expected.