Monday, August 5, 2019
08/05/2019 06:57 - We are having connectivity issues in Dallas at the moment. Our engineers is working to resolve the issue.
08/05/2019 07:49 - Our engineers are still trying to resolve the issue. Your patience is appreciated.
08/05/2019 08:02 - We have found the root cause of the issue and 90% of the servers are back online now.
08/05/2019 09:53 - All the servers are back online. The issue was related to a bug in our networking gear. We have applied a fix for this and we'll upgrade our core networking gear to have full redundancy to avoid issues like these in the following month. A full RFO will be posted here.
------------------------------
Reason for Outage:
Out of a sudden we have started to notice some of the IP addresses disappearing from the ARP table of our core router and we have started investigating the issue.
All the network configuration was correct and it's been running with the same configuration for the past 4 years without issues.
We have then identified that once a random MAC ID gets online, the ARP table was getting filled up in a few seconds which was causing the unstability.
We have narrowed down to which MAC ID was causing the issue and then we have replaced it.
We have notified the manufacturer of the routing gear about the issue and we are awaiting an explanation.
We are sorry for the inconvenience caused by this.