Service Status

Resolved

Seattle network issues

We are aware of the Seattle network issues. We are working with the datacenter staff to resolve these as soon as possible.

Update 04:36 AM This has been resolved. The datacenter confirmed a bad PDU caused this. We are sorry for the downtime.

Resolved

Rebooting dalvz7highram12

We are currently rebooting dalvz7highram12 node due to instability issues.

26 Jan 2020 11:15: Reboot has been completed.

Resolved

NJ network problem

Wednesday, 08 January 2020, 04:22 - We're investigating a network outage affecting certain IP ranges our New Jersey servers. We'll update this announcement as we have more information.

Wednesday, 08 January 2020, 05:18 - The issue has been resolved.

Resolved

seapure1 connectivity

 Tuesday, 07 January 2020, 11:46 - We're investigating connectivity issues to our seapure1 node now. 

 Tuesday, 07 January 2020, 12:20 - Server connectivity has been restored.  

Resolved

dalvz7highram6

Saturday, 21 December 2019, 07:17 - we're investigating an issue with our dalvz7highram6 server now. 

Saturday, 21 December 2019, 07:39 - issue resolved and servers are back online now. 

Resolved

seavz7highram1 accessibility issues

Tuesday, 03 December 2019, 00:03 - Our internal monitoring has alerted us about an outage of our seavz7highram1 node. Our engineers are investigating.

Tuesday, 03 December 2019, 00:25 - Our engineers have resolved the problem.

Resolved

seavz7highram1 connectivity issues

Sunday, 01 December 2019, 06:02 - we've been alerted via our internal monitoring, about an outage of our seavz7highram1 node. Our engineers are investigating.
Sunday, 01 December 2019, 06:10 - our engineers have detected and resolved the problem.

Resolved

Network Issue in Dallas

Monday, 28 October 2019, 15:53 - we're aware of the Dallas connectivity issues and are investigating them now. 
Monday, 28 October 2019, 19:18 - DFW is now completely back online. Official RFO will be posted to this announcement as it becomes available.

Tuesday, 29 October 2019, 14:04 - RFO is as follows:

Hello,

OFFICIAL RFO - 10/28/2019
Summary of Incident:

———————————————

Yesterday, Monday October 28th 2019, at approximately 4:23pm portions of customers in our TPA1, TPA2 and DAL1 data centers experienced a loss of network that lasted anywhere from a few minutes to a few hours depending on your server(s) location. The cause of the issue has been identifed and is as follows:

At roughly 4:23pm one of our Network Engineers applied a policy update to our DAL1 edge routers. This policy update was incomplete which led to the full internet routing table being propogated throughout the aggreagation layer of DAL1. This mistake was further exacerbated when that full routing table was automatically injected into the Hivelocity DDoS protection network resulting in the full routing table being distributed to other Hivelocity facilities, i.e. TPA1 and TPA2. The full internet routing table injection led to multiple network devices having their resources exhausted which ultimately led to the network disuption. Once our Network Engineers identified the cause of the issue we began reloading each of the affected network devices to correct the problem. Ultimately, yesterday's network event was a result of human error.

Service Impact Times:

———————————————
October 28th, 4:23pm - 6:44pm EST

Remediation Plans:

———————————————

We have implemented new router policies that will prevent full route tables being similarly propogated should human error ever occur again. Additionally, we have implemented new review protocols to minimize the likelihood of any human error occurring.

For years most of our customers have experienced 100% uptime due to our redundancies and nearly 2 decades of experience. We take our responsbility to you very seriously and no one hates it more than us when we fall short of our goals. We are deeply sorry for the inconvenience and any negative impact this disruption had on your operation.

Resolved

seavz7highram1 connectivity errors

Monday, 28 October 2019, 14:20 - we're investigating an accessibility issue on our seavz7highram1 node now. 

Monday, 28 October 2019, 15:30 - we have resolved the NIC HW issue and the VPS's are back up.

Resolved

LA server connectivity issues

Monday, 14 October 2019, 13:00 - We're investigating issues with LA server connectivity now and will update this announcement as we have more information. 

Monday, 14 October 2019, 13:12 - All LA connectivity has been restored. 

Resolved

ukhighram2 is down

Thursday, 10 October 2019, 18:43 - Our monitoring has alerted us that this node is down. Preliminary analysis shows that this is due to a networking event. Our team has alerted the datacenter and is awaiting updates.
Thursday, 10 October 2019, 21:10 - Service has now been restored. We await RFO from the DC.

Resolved

lastorage1 is offline

Tuesday, 01 October 2019, 15:21 - We're aware that this server is currently offline. We've reached out to the datacenter and are currently assessing the situation. Further updates will be tagged to this announcement as they become available. Thank you for your patience.

Wednesday, 02 October 2019, 23:12 - From what it looks like some of the harddisks in the RAID60 array have silently spitted errors and the RAID card (HP P420) wasn't aware of this and therefore the filesystem has crashed. We have brought the node back online and we are currently running fsck.

Friday, 04 October 2019, 18:10 - We're still working on this. We'll update you as soon as further details become available.

Sunday, 06 October 2019, 04:56 - fsck process is still ongoing.

Tuesday, 08 October 2019, 11:55 - fsck remains ongoing at this time, and is expected to take a bit of time given the size of the disk array. We'll update this anouncement as we have more information. 

Thursday, 10 October 2019, 09:06 - fsck process continues. 

Monday, 14 October 2019, 02:06 - fsck process has been finished. Some of the VPS's are online. Some are not booting up due to corrupt files. We are working on this.

Monday, 23 October 2019, 14:18 - We were only able to recover around half of the VPS's from this node. We have built a new node and we are syncing the data to it right now. A ticket will follow for your situation.

Resolved

Dallas network connectivity issues

08/05/2019 06:57 - We are having connectivity issues in Dallas at the moment. Our engineers is working to resolve the issue.

08/05/2019 07:49 - Our engineers are still trying to resolve the issue. Your patience is appreciated.

08/05/2019 08:02 - We have found the root cause of the issue and 90% of the servers are back online now.

08/05/2019 09:53 - All the servers are back online. The issue was related to a bug in our networking gear. We have applied a fix for this and we'll upgrade our core networking gear to have full redundancy to avoid issues like these in the following month. A full RFO will be posted here.

------------------------------

Reason for Outage:

Out of a sudden we have started to notice some of the IP addresses disappearing from the ARP table of our core router and we have started investigating the issue.

All the network configuration was correct and it's been running with the same configuration for the past 4 years without issues.

We have then identified that once a random MAC ID gets online, the ARP table was getting filled up in a few seconds which was causing the unstability.

We have narrowed down to which MAC ID was causing the issue and then we have replaced it.

We have notified the manufacturer of the routing gear about the issue and we are awaiting an explanation.

We are sorry for the inconvenience caused by this.

Resolved

Partial network connectivity issues in Dallas

We are experiencing partial network connectivity issues in Dallas. This is related to one of our uplink providers.

Your VPS's are not offline completely but they may be having some issues for some parts of the world.

Our engineers are working on the issue at the moment.

Update 17 July 2019, 2:55 PM: The issue has been narrowed down by our DC network engineers.

Update 17 July 2019, 3:02 PM: Everything is working as it should for the past 12 minutes as of now. We continue to monitor the situation.

Resolved

Issue with two servers in UK

Tuesday, 09 July 2019, 08:49 - We're investigating an issue with our ukhighram1 and ukvz7highram1 host nodes. All other UK services are currently online. We'll provide updates here as we have them. 

Tuesday, 09 July 2019, 10:01 - This issue is still under investigation. An ETA is not available beyond as soon as possible. 

Tuesday, 09 July 2019, 11:00 - Our datacenter hasn't provided us with enough information about the source of the outage. They are still working on to get them back online.

Tuesday, 09 July 2019, 11:29 - Our datacenter has notified us that the power distribution unit in the rack which is connected to these servers is having issues and that they are working on to replace it.

Tuesday, 09 July 2019, 12:25 - The PDU has been replaced and all servers are back online now. If you're still seeing a connectivity issue please don't hesitate to ticket in to inquire and we'll investigate. Thank you. 

Resolved

Storage server connectivity with IP range 63.142.248.xxx

Friday, 28 June 2019, 12:15 - we're aware of a connectivity issues for storage servers in IP range 63.142.248.xxx and are working this issue now. 

Friday, 28 June 2019, 12:39 - this is fixed. If you're still seeing an issue with 63.142.248.xxx please do tricket in. 

Resolved

Dalhighram16 - connectivity issues

Wednesday, 26 June 2019, 02:37 - We are experiencing connectivity issues and are currently investigating the problem on dalhighram16. Updates to follow.
Wednesday, 26 June 2019, 02:53 - We have identified the problem and applied a fix. Connectivity is restored in full.

Resolved

Network Downtime in Dallas

Friday, 21 June 2019, 15:19 - We're aware of Dallas connectivity issues and are investigating now. 

Friday, 21 June 2019, 15:41 - This is a networking issue. Our engineers is working on it at the moment.

Friday, 21 June 2019, 15:50 - All the data is safe. We are working on getting our spare router up and working.

Friday, 21 June 2019, 15:59 - Our network is back online. Our engineers is doing their checks before confirming this is stable now.

RFO:

We have experienced a BGP flap on Thursday, 20 June 2019, 08:59 and our engineers have investigated the situation and found out that our main router has rebooted itself for an unknown reason.

We have been monitoring our router since and we haven't seen any irregularities until Friday, 21 June 2019, 15:19.

On Friday, 21 June 2019, 15:19, our router became inaccessable again.

We have dispatched datacenter engineers to physically connect to the router and unfortunately they weren't able to.

We have then immediately attempted to put our spare core router online to replace the faulty one.

Finally on Friday, 21 June 2019, 15:59, the spare router was in place and up and running and our BGP session was reestablished.

For this not to happen again in the future, we are moving forward with our our core redundancy setup plans to have full network redundancy in advance of our scheduled date to complete this.

We sincerely apologize for the inconvenience this situation has caused you and we appreciate your understanding very much.

Resolved

Connectivity issues in Dallas Location

 Thursday, 20 June 2019, 08:59 - we're aware of Dallas connectivity issues and are investigating now. 

Thursday, 20 June 2019, 09:09 - issue resolved and everything back online. Please do open a ticket if you're still seeing connectivity issues. Thank you. 

Resolved

Varied Dallas network connectivity issues

Friday, 31 May 2019, 07:45 - Some customers are reporting connectivity issues to our Dallas location. If this is affecting you, please run a traceroute from yourself to one of your Dallas services and provide it in a ticket. Thank you.

Friday, 31 May 2019, 08:11 - Issue is solved, waiting for confirmation of stability from datacenter.

Friday, 31 May 2019, 09:40 - Issue confirmed solved, RCA from datacenter: We received this from he.net. One of the routers in Ashburn unexpectedly reloaded, resulting in some routing irregularities while paths converged.