Effective today at 12:00 EST (17:00 UTC), all forwarded mail scoring 6.0 and higher is being discarded.

Today and over the next few weeks we will be introducing some changes that will improve the deliverability of forwarded mail. There are two problems that these changes will solve or at least minimize.

We are seeing more systems that are rejecting mail that violates the sender's published SPF policy. Mail from a domain with published a SPF policy that we forward on behalf of our customer may violate the sender's SPF policy because the sender has no way to know that the recipient is forwarding mail to another system. The recipient that is forwarding the mail has no way to know in advance that the sender does not wish mail from the their domain to be forwarded.

In most all cases the rejected mail is spam with a forged sender address but we have seen a small percentage of legitimate mail rejected. It is this small percentage that we would like to see accepted by the system that the mail is forwarded to.

The SPF forwarding problem is well known and the debate about wether SPF breaks forwarding or forwarding is inherently broken still rages on.

To ensure deliveriablity of mail that is forwarded, SRS will be enabled for all forwarded mail. Currently SRS is enabled for mail forwarded to AOL and a few dozen other systems.

Another problem is that more and more systems are rejecting spam that is forwarded to them. When a system that we are forwarding mail to rejects a message, a Delivery Status Notification must be delivered to the sender. If the rejected message is truly spam, and it almost always is, the sender is most likely forged and the DSN is sent to the innocent forgery victim adding to the bad and growing backscatter problem.

Another issue with forwarding spam is that many systems pay attention to what IP addresses are sending them spam in the same way that this system does. The recipient system does not know or care that the spam is forwarded on behalf of a customer. To the receiving system the message is just spam. Too much spam from an IP address and the IP address is blocked. Blame the spammers, not the mail system operators.

To reduce the backscatter that this system generates and to reduce the possibility of our forwarding servers being blocked and preventing mail from being forwarded, messages with a spam score of 6.0 and higher will no longer be forwarded. This change has been in affect for mail to AOL and its subsidiaries since May 2005 with no customer complaints.

Starting today messages scoring 6.0 and over will not be forwarded to adelphia.net, yausi.com, yahoo.com, hotmail.com and cox.net. SRS envelope sender re-writing is also being applied to mail forwarded to those domains.

Along with this change is a real-time report showing forwarded messages that have been discarded.

To: somebody@aol.com
recipient@domain-hosted-here
1 18.2 chancei@ambienttech.com
1 28.5 charlenehipes@asamoacpa.com
1 20.3 daiangelag@0451.com
1 15.5 fpeters@globalfriends.us
1 12.5 gar@directwest.com
1 18.2 gunnersoe@rednet.com
1 12.5 kph@gfj.bdstnetd.com
1 23.9 meeny.beatrix9hm@gmail.com

Select Account -> Real-Time Reports -> Forwarded Discards to view the reports for your account.

The errors customers are seeing for box polls configured to fetch mail from Fastmail are due to a server failure at Fastmail and not due to problems with this system.

Update Oct 31, 2005: Telephone service was restored this past weekend.

We had dialtone and working lines yesterday but this morning nothing. Problem is most likely due to batteries failing in one of the BellSouth fiber huts. We have no estimate to when telephone service will be restored.

Late this morning a problem with the IMAP and POP multiplexors caused some connections to timeout. Webmail and SMTP services were not affected. The exact time and duration of this problem is lost in the blur of the day resulting from dealing with hurricane Wilma last night and this morning. The duration was most likely less than 90 minutes.

Real-time reporting of messages rejected by the MX servers, messages infected with viruses that were discarded, and messages delivered to mailboxes was added to the Account Manager last week in an expanded beta test. Real-time reporting is available for hosted email accounts and for will be for MX relay customers later this month.

The last known bug was fixed today and with that fix the delivery reports now include messages accepted for forwarding to external accounts as well as messages that will be delivered to mailboxes. Fixing this bug required deleting all delivery data prior to 1400 EDT, 1800 UTC, today.

Because of the huge number of unknown address rejections due to backscatter from addresses forged by spammers and viruses, the reject reports include only a count of those messages.

Update: All services were back online by 15:55 EDT (19:55 UTC). The outage was caused when the building electricians accidently tripped the main panel breaker while removing the panel cover for further investigation into why the breaker failed yesterday.

The datacenter has contracted with an engineering firm to determine where the weak points are in their power distribution systems and what they can do to improve the power system.

Staring tomorrow evening we will begin moving servers and routers that provide load sharing and redundant routing into another cage. This work will take several days to complete and service outages are expected to be very brief. This work will be done between 20:00 and 00:00 EDT (00:00 - 04:00) UTC tomorrow evening and between 16:00 and 20:00 (20:00 - 00:00 UTC) Saturday and Sunday with the same schedule the following weekend.

Update on the power failure during hurricane Katrina. Lightning struck one of the generators supplying power to the first floor taking out the fuel pump that feeds fuel from the large ground level storage tanks to the generator day tank. The generator continued to operate untill its day tank was empty. To help prevent failures of this nature in the future, additional monitoring and pumps are being installed.

15:10 EDT (19:10 UTC) Power has been restored and services will be back online shortly.

Update: Power was restored around 12:10 EST (1610 UTC) and IMAP access via the webmail clients was restored shortly after that. Full access to the IMAP and POP servers followed and finally SMTP access was restored. We are still working to bring up the backup raid systems.

The cause of the failure was an overheated 208V 3-phase circuit breaker in the distribution panel that supplies our racks. The breaker failure was complete and it had to be replaced.

Our second cage in the same datacenter is powered from a different 208V distribution panel. The redundant servers, routing, and switching, will be moved into that cage starting later this week.

We have lost all power to one of our cages at the datacenter. We will be moving enough servers to our second cage to bring services back online.