Update: Power was restored around 12:10 EST (1610 UTC) and IMAP access via the webmail clients was restored shortly after that. Full access to the IMAP and POP servers followed and finally SMTP access was restored. We are still working to bring up the backup raid systems.

The cause of the failure was an overheated 208V 3-phase circuit breaker in the distribution panel that supplies our racks. The breaker failure was complete and it had to be replaced.

Our second cage in the same datacenter is powered from a different 208V distribution panel. The redundant servers, routing, and switching, will be moved into that cage starting later this week.

We have lost all power to one of our cages at the datacenter. We will be moving enough servers to our second cage to bring services back online.

Between 0300 and 0400 EDT, 0700 and 0800 UTC, our colocation provider will be upgrading their core routers to Cisco layer 3 switching technology. There will be a few short periods during this time when some customers may not be able to reach our servers.

The datacenter where our racks are located lost power to the first floor around 2230 EST (0330 UTC) taking out most carriers in the center. Power was restored to the first floor around 2300 EST (0400 UTC).

An update will be posted when we know more about what caused the power outage and what steps will be taken to prevent future occurances.

Between 0100 and 0130 EDT, 0500 and 0530 UTC, our colocation provider will be testing failover to backup circuits. There will be a few periods of routing instability during this period. We do apologize for the inconvience this will cause for some customers.

Between 0100 and 0200 EDT, 0500 and 0600 UTC, our colocation provider will be upgrading line cards in one of the border routers. We do apologize for the inconvience this will cause for some customers.

Between 0030 and 0045 EDT, 0430 and 0445 UTC, our colocation provider will be upgrading line cards in one of the border routers. There will be two periods of brief routing instability while routing is moved from and back to the router being upgraded.

Between 1200 and 1230 EDT, 1600 and 1630 UTC, we will be updating the software on the backend IMAP servers in preparation for supporting near real-time replication of IMAP events to the backup IMAP servers.

This update requires restarting the IMAP servers. IMAP and POP3 access will not be available during the update. This update is expected take between 5 and 15 minutes.

Accounts created after 1500 EDT (1900 UTC) today will use an alternate IMAP namespace. In this alternate namespace IMAP clients will see the INBOX folder on the same level as other top level folders. This change will better support IMAP clients that make invalid assumptions about the IMAP folder hierarchy.

The namespace can be changed per mailbox in the Manager.

Effective May 21, 2005, mail that is forwarded to AOL accounts that scores as spam will be silently discarded. This change is necessary to insure that legitimate non-spam mail can continue to be forwarded to AOL accounts by our customers.

Before you push that AOL 'Report Spam' button, please check the message headers to see if the spam was sent directly to your AOL account or forwarded from your domain. Some threshold of spam reports from our IP addresses will result in AOL blocking all mail from those IP addresses.

The routing issue caused 50% to 80% packet loss for about 20 minutes for some customers. One of the routers causing this has been taken offline and its backup has taken over active routing duty. The problem router will be restarted later tonight EST.

We are currently experiencing an internal routing problem at the data center. We hope to have this issue resolved ASAP.

We are testing new versions of the Squirrelmail and IMP webmail clients at our beta test site. Both clients have many new features and configuration options. Please try these new clients and contact support if you have any questions or comments about these new clients.

At 1037 EST, 1537 UTC, the authentication server for the mail storage servers stopped responding to authentication requests. Unable to determine the reason for failure we restarted the server at 1055 EST, 1555 UTC, and it is now functioning properly. We will continue to investigate the cause of this failure.

Update on the Jan 26, 2005 power outage.

Batteries for the failed UPS system arrived by truck from Tampa and were installed late yesterday afternoon. By 1830 EST, 2330 UTC, we were back on UPS power.