The datacenter has contracted with an engineering firm to determine where the weak points are in their power distribution systems and what they can do to improve the power system.
Staring tomorrow evening we will begin moving servers and routers that provide load sharing and redundant routing into another cage. This work will take several days to complete and service outages are expected to be very brief. This work will be done between 20:00 and 00:00 EDT (00:00 - 04:00) UTC tomorrow evening and between 16:00 and 20:00 (20:00 - 00:00 UTC) Saturday and Sunday with the same schedule the following weekend.
Update on the power failure during hurricane Katrina. Lightning struck one of the generators supplying power to the first floor taking out the fuel pump that feeds fuel from the large ground level storage tanks to the generator day tank. The generator continued to operate untill its day tank was empty. To help prevent failures of this nature in the future, additional monitoring and pumps are being installed.
15:10 EDT (19:10 UTC) Power has been restored and services will be back online shortly.
2005-09-21: 15:50 UTC   Power Failure
The cause of the failure was an overheated 208V 3-phase circuit breaker in the distribution panel that supplies our racks. The breaker failure was complete and it had to be replaced.
Our second cage in the same datacenter is powered from a different 208V distribution panel. The redundant servers, routing, and switching, will be moved into that cage starting later this week.
We have lost all power to one of our cages at the datacenter. We will be moving enough servers to our second cage to bring services back online.
2005-09-09: 13:30 UTC   Scheduled Maintenance Saturday, Sep 10