Service Status

Resolved

Dallas Network Issue

Wednesday, 28 November 2018, 03:31 - We're aware of the issue in Dallas, and are attempting to follow up with our connectivity provider. Further updates will be posted to this issue as we receive them.
Wednesday, 28 November 2018, 04:21 - Everything should be operating normally already, please get in touch with us in case you're not able to access your service.

Resolved

Network wide port blocks

Due to recent reflection exploits involving ports 111 (portmap/NFS), 161 (snmpd), and 389 (ldap), we've implemented a network-wide filter at our network edge to avoid your services running on these ports from being exploited and used to attack other computers on the internet.

We understand that this may be disruptive to your usage, and we can provide guidance on how to adjust these services to use different ports so that you may continue to enjoy uninterrupted connectivity.

These services are not installed on our VPS by default, so if you do not recognize any of the above services or ports, you're likely not affected by this action and this message is informational only.

Resolved

New Jersey Maintenance

Thursday, 04 October 2018, 15:58 - We've just been briefed that our provider in New Jersey has scheduled a 30 minute maintenance window that may have service impacts for our clients on the nj1 node. See notice below:

Event Type: Network Upgrade
Start Time: 10/9/2018 12AM EST
End Time: 10/9/2018 12:30AM EST
Location(s): Piscataway, NJ
Device(s): c8-10-b4-1.pnj1
Client Affecting: Yes

Event Summary: Important maintenance will be performed on one or more devices that may affect network connectivity to your services.

 

Resolved

Intermittent connectivity affecting all Dallas nodes

Thursday, 27 September 2018, 18:24 - We're aware of the issue, and are attempting to follow up with our connectivity provider. Further updates will be posted to this issue as we receive them.
Thursday, 27 September 2018, 18:33 - Our provider has confirmed that this was a misconfigured network-wide-filter applied on our upstream port(s). The rogue change has now been rolled back, services are back online.

We apologize for any inconvenience caused.

Resolved

NJ Maintenance

Our New Jersey Datacenter is currently performing a scheduled maintenance

Event Type: Network Upgrade
Start Time: 2018-09-24 - 03:00:00 AM EST
End Time: 2018-09-24 - 05:00:00 AM EST
Location(s): NYC Metro Datacenter 
Device(s): Core/Edge Network Devices
Client Affecting: Yes

Event Summary: Important maintenance will be performed on one or more devices that may cause intermittent network connectivity issues to your services. We do not anticipate extended outages for this network maintenance.

We will continue to post updates as we receive them.

Resolved

ukhighram3 offline

Thursday, 23 August 2018, 06:21 - We have just been made aware of an event concerning this node. Further investigation is ongoing, more details will be posted to this issue as they become available.
Thursday, 23 August 2018, 06:38 - Our datacenter has confirmed that they're experiencing a network event that's affecting this server. An explicit ETA has not been communicated, but we remain hopeful about service restoration soon.
Thursday, 23 August 2018, 08:40 - This remains in progress. We will update this as we have more information. 
Thursday, 23 August 2018, 10:01 - Datacenter has confirmed that this is still in progress. 
Thursday, 23 August 2018, 11:22 - An update from the datacenter has specified that a PDU in the rack where this server is located is malfunctioning, and they are working on moving the server to a temporary rack to get it back online. 
Thursday, 23 August 2018, 13:49 - We continue to wait for the server to be moved to a different rack to be put back online. 
Thursday, 23 August 2018, 15:15 - We have noted that the server is back online, and we're waiting for confirmation of stability. 
Thursday, 23 August 2018, 15:45 - Confirmed that server will remain online for the foreseeable future. We will be providing one month of credit for the downtime. All VPS are online and running, so if your server is still offline or inaccessible, please reach out to us and we'll investigate further. 

Resolved

dalstorage4 disk array rebuild

Tuesday, 07 August 2018, 00:08 - Earlier today, our dalstorage4 node suffered a disk failure and it is currently operating in a degraded state. There unfortunately isn't enough I/O to go around, so I/O bound programs are taking a very long time to respond (due to the I/O wait). We're investigating, further updates will be posted to this issue. Customers not on the dalstorage4 node are not affected by this issue.

Wednesday, 08 August 2018, 09:03 - This situation does remain in progress. Your VPS should be online and accessible though, since the I/O contention seems to come in short infrequent waves. If you're having an issue with access beyond being slower than usual, please inquire and we can investigate. 

Thursday, 09 August 2018, 08:23 - This continues to be in progress. Reboots via the client area/VM control system have been disabled as rebooting won't magically fix this situation and will only contribute to more I/O contention. If you need your VPS rebooted and are truly unable to figure out how otherwise, please ticket in and we'll review your request. 

Friday, 10 August 2018, 11:46 - We're going to shut down the server to replace the drives and let it rebuild without VM contention; this will take less time than if we left VMs running, resulting less net time back to normal. This shouldn't have much of an impact on actual usage given the speeds of the server as it stands. The current estimate is 24 hours for array rebuild.

Saturday, 11 August 2018, 06:25 - All the processes successed and we have booted up the VPS's. Everything should be healthy at this point.

Resolved

Dallas connectivity

Tuesday, 24 July 2018, 10:19 - the Dallas connectivity issue is acknowledged and being investigated now.

Resolved

nj1 downtime

11 July 2018, 10:45 - Planned network maintenance for this server has begun

11 July 2018, 11:25 - Planned network maintenance is complete and the server is back online

Resolved

Connectivity Problem in Dallas

Wednesday, 30 May 2018, 12:21 - we are aware of the Dallas connectivity issue. Datacenter can be followed here: https://twitter.com/InceroStatus - we'll update this when we hear back. 

Wednesday, 30 May 2018, 12:29 - connectivity restored

Resolved

nj1 connectivity

Sunday, 22 May 2018, 18:00 - we're investigating an outage of our NJ1 server. We'll update this announcement as we have more information.

Sunday, 22 May 2018, 18:40 - server is back online without issue noted.

Resolved

FW intermittent network issues

 Monday, 21 May 2018, 07:49 - We're aware of the intermittent network issues plaguing our Dallas (DFW) POP. This seems to be due to heavy packet loss on our upstream's end. We're collaborating with our upstream to mitigate this, thank you for your patience.

Resolved

Kernel panic on dalhighram10

We have experienced a kernel panic on dalhighram10 and we are currently looking into it.

Update: All the VPS's booted back up again.

Resolved

Dallas network errors

Wednesday, 28 February 2018, 10:49 - We're investigating a Dallas connectivity issue. We'll update this announcement as we have more information. This is a network issue. We are experiencing high package loss. Our datacenter is working on the issue.

Wednesday, 28 February 2018, 11:41 - Root cause has been determined: outbound network attacks from insecure memcached installations. More info here: https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ We're looking at how to adjust them now. 

Wednesday, 28 February 2018, 12:28 - This has been mitigated, root cause was a new memcached amplification attack. If you're still seeing connectivity issues, or truly need internet-accessible memcached access for some reason, please ticket in. 

Resolved

nj1 network connectivity

Sunday, 25 February 2018, 13:48 - we're investigating an outage of our NJ1 server. We'll update this announcement as we have more information. 

Sunday, 25 February 2018, 14:52 - the server is back online, and we're awaiting confirmation that the network is stable

Resolved

dalstorage1 unreachable

Tuesday, 20 February 2018, 02:55 - We've been made aware of an event on this node and are currently investigating.
Tuesday, 20 February 2018, 05:16 - Node is back up, this was due to a storage event. VMs are now booting back up. Please expect to be contacted individually about remedies.

Resolved

dalhighram25 unreachable

The node is currently unreachable from the panel and the VMs are working extremely slowly. We're already looking into the issue but it has already taken more time than expected. Please excuse us for the inconvenience.

Update: The situation is under control now, please open a ticket if you're still having issues.

Resolved

Dallas network problems

Tuesday, 06 February 2018 11:34 - We're receiving reports of an unexpected partial routing failure somewhere, or similar network issue. Please don't stop/start/reboot your VPS, it won't fix the issue. When opening a ticket to inform us that your VPS is offline, could you please send a TEXT traceroute between yourself and your service with us? This will assist us with resolving the issue. 

Tuesday, 06 February 2018 11:59 -  The issue should have been rectified now. We'll update the announcement with a RFO.

Friday, 02 March 2018, 10:46 - Formal RFO is available at the following link: https://d.pr/f/RN2AjO

Resolved

ukhighram2 issues

ukhighram2 is currently facing issues. We are working on the problem at the moment.

Update: The server is back online now. We are investigating the reason for the lock up.

Resolved

Storage VPS Line Meltdown and Spectre Maintenance

We are going to reboot storage VPS nodes to patch them against Meltdown and Spectre vulnerabilities.

We are going to update this announcement once we have updates.

09:56 AM: All the nodes have been rebooted and all the VPS's are back online.