|
2005-01-28: 16:00 UTC
Authentication problems
At 1037 EST, 1537 UTC, the authentication server for the mail
storage servers stopped responding to authentication requests.
Unable to determine the reason for failure we restarted the
server at 1055 EST, 1555 UTC, and it is now functioning properly.
We will continue to investigate the cause of this failure.
Update on the Jan 26, 2005 power outage.
Batteries for the failed UPS system arrived by truck from Tampa and were installed late yesterday afternoon. By 1830 EST, 2330 UTC, we were back on UPS power.
Update on the Jan 26, 2005 power outage.
Batteries for the failed UPS system arrived by truck from Tampa and were installed late yesterday afternoon. By 1830 EST, 2330 UTC, we were back on UPS power.
2005-01-26: 20:10 UTC Data center power failure
Power Outage Update: January 27 0330 EST, 0830 UTC
At 1505 EST, 2005 UTC, a UPS in the data center failed and our systems were without power for approximately 20 minutes. The engineers at the data center attempted to bring the UPS back online without success and utility power was switched to the circuits fed by the failed UPS. When power was restored one router failed to boot and its HSRP backup failed to announce our routes contributing to the problem.
All servers suffered some file system damage requiring another 10 to 90 minutes to repair before the servers could be brought online. Approximately 80 minutes after the power failure the incoming MX servers, the outbound SMTP servers, the MX relay servers, and two mail storage servers were back in service.
The last mail storage server was back online just before 1800 EST, 2300 UTC. The web sites were brought up shortly after all mailbox servers were up and functioning. The Account Manager followed shortly after that.
The UPS failure was not an inverter failure as reported earlier. When the Powerware engineers arrived on site diagnostics showed that the UPS was healthy but the batteries were not up to the task. Throughout today the facility was experiencing short but serve utility power fluctuations causing all of the UPS systems to switch in and out of battery operation and the batteries were not being fully recharged when returning to utility power. This frequent discharge and inadequate recharge caused the batteries in this UPS to fail resulting in this outage.
The data center is expediting delivery of new batteries for this UPS and the other UPS systems in the data center. The new batteries may arrive as early as tomorrow afternoon, Friday.
We sincerely regret the inconvenience to our customers caused by this unfortunate outage.
At 1505 EST, 2005 UTC, a UPS in the data center failed and our systems were without power for approximately 20 minutes. The engineers at the data center attempted to bring the UPS back online without success and utility power was switched to the circuits fed by the failed UPS. When power was restored one router failed to boot and its HSRP backup failed to announce our routes contributing to the problem.
All servers suffered some file system damage requiring another 10 to 90 minutes to repair before the servers could be brought online. Approximately 80 minutes after the power failure the incoming MX servers, the outbound SMTP servers, the MX relay servers, and two mail storage servers were back in service.
The last mail storage server was back online just before 1800 EST, 2300 UTC. The web sites were brought up shortly after all mailbox servers were up and functioning. The Account Manager followed shortly after that.
The UPS failure was not an inverter failure as reported earlier. When the Powerware engineers arrived on site diagnostics showed that the UPS was healthy but the batteries were not up to the task. Throughout today the facility was experiencing short but serve utility power fluctuations causing all of the UPS systems to switch in and out of battery operation and the batteries were not being fully recharged when returning to utility power. This frequent discharge and inadequate recharge caused the batteries in this UPS to fail resulting in this outage.
The data center is expediting delivery of new batteries for this UPS and the other UPS systems in the data center. The new batteries may arrive as early as tomorrow afternoon, Friday.
We sincerely regret the inconvenience to our customers caused by this unfortunate outage.
