One of our hosting providers along with most of the world, experienced a brief outage due to a known router memory limit issue. The following support notice explains the situation well.


08/12/14 Update
08/12/14 6:00PM (EDT)
 At this time, our customers appear to have regained connectivity from all locations. If you see any additional issues, please do not hesitate to contact us via telephone, email or livechat. We have outlined a comprehensive recap of events at the following URL: http://www.liquidweb.com/status/0812014networkevent.html 

 08/12/14 5:00PM (EDT)
 Last night the global routing table exceeded 512,000 entries. This caused problems for many popular router models, including the Cisco routers that Liquid Web uses. The default memory allocation for the BGP table size is not large enough to hold the global table after surpassing the 512,000 entry mark. This caused many routers across the globe to experience issues, including the routers at Liquid Web. Unfortunately at this time we do not have an exact diagnosis of what has definitively caused this issue. We are still investigating and troubleshooting all possible solutions. 

 What are we currently doing to fix the issue?
 We believe that allocating more memory to our core routers to handle the additional BGP routes will potentially remedy this situation on our network. We have begun the process to upgrade the memory allocation on our core routers. As a part of this memory upgrade, a reboot of some core routers is required and we are in the process of completing this. While performing these reboots we encountered a problem with a line card in Core 5 and we are currently working to repair this. We believe that this issue should be alleviated after successfully allocating more memory to the core routers and performing the reboot, however, there still may be problems with other providers outside of our network. 

 08/07/14
 After increasing the capacity to all areas of the network in US Central Zone B, we have completed re-enabling backups in this area. For additional details pertaining to the updates in US Central Zone B, please check our Status page, located at the following URL: http://www.liquidweb.com/status.html
Network Outage on Global Routers

You May Also Like

Leave a Reply