2007-10-12: 23:30 UTC     Level3 routing issues

Earlier this morning a customer reported that they could not connect to one of our webmail servers. They provided a traceroute showing a routing loop where Level3 transits to one of our connectivity providers, Host.net. The problem appeared to be a link load sharing issue, an adjacent IP address in the same /24 network was routed correctly.

Load sharing with equal cost routes is based on the source IP address, the destination IP address, and possibly the source and destination ports. That and the fact that the bulk of our traffic does not transit Host,net, limited the number of customers affected by this issue. SMTP, IMAP, and POP3 access was not affected.

We have seen a number of problems with Level3 routing over the last few months. Last month we dropped our BGP announcements to Level3 to keep as much inbound traffic off of that network as possible. Some outbound traffic will still transit the Level3 network to Level3 customers.

Unfortunately our web services were not in the IP address space that we announce to the Internet with BGP. We have renumbered the webmail client servers and the Manager server into the address space that we announce via BGP. The beta server will be renumbered later this evening.

Page delivered in 0.025956 seconds, 39 files included