2005-01-28: 16:00 UTC
At 1037 EST, 1537 UTC, the authentication server for the mail
storage servers stopped responding to authentication requests.
Unable to determine the reason for failure we restarted the
server at 1055 EST, 1555 UTC, and it is now functioning properly.
We will continue to investigate the cause of this failure.
Update on the Jan 26, 2005 power outage.
Batteries for the failed UPS system arrived by truck from Tampa
and were installed late yesterday afternoon. By 1830 EST, 2330
UTC, we were back on UPS power.
2005-01-26: 20:10 UTC
Data center power failure
Power Outage Update: January 27 0330 EST, 0830 UTC
At 1505 EST, 2005 UTC, a UPS in the data center failed and our
systems were without power for approximately 20 minutes. The
engineers at the data center attempted to bring the UPS back
online without success and utility power was switched to the
circuits fed by the failed UPS. When power was restored one
router failed to boot and its HSRP backup failed to announce our
routes contributing to the problem.
All servers suffered some file system damage requiring another
10 to 90 minutes to repair before the servers could be brought
online. Approximately 80 minutes after the power failure the
incoming MX servers, the outbound SMTP servers, the MX relay
servers, and two mail storage servers were back in service.
The last mail storage server was back online just before 1800
EST, 2300 UTC. The web sites were brought up shortly after all
mailbox servers were up and functioning. The Account Manager
followed shortly after that.
The UPS failure was not an inverter failure as reported earlier.
When the Powerware engineers arrived on site diagnostics showed
that the UPS was healthy but the batteries were not up to the
task. Throughout today the facility was experiencing short but
serve utility power fluctuations causing all of the UPS systems
to switch in and out of battery operation and the batteries were
not being fully recharged when returning to utility power. This
frequent discharge and inadequate recharge caused the batteries
in this UPS to fail resulting in this outage.
The data center is expediting delivery of new batteries for this
UPS and the other UPS systems in the data center. The new batteries
may arrive as early as tomorrow afternoon, Friday.
We sincerely regret the inconvenience to our customers caused by
this unfortunate outage.