Service Status

Resolved

nj1 downtime

11 July 2018, 10:45 - Planned network maintenance for this server has begun

11 July 2018, 11:25 - Planned network maintenance is complete and the server is back online

Resolved

Connectivity Problem in Dallas

Wednesday, 30 May 2018, 12:21 - we are aware of the Dallas connectivity issue. Datacenter can be followed here: https://twitter.com/InceroStatus - we'll update this when we hear back. 

Wednesday, 30 May 2018, 12:29 - connectivity restored

Resolved

nj1 connectivity

Sunday, 22 May 2018, 18:00 - we're investigating an outage of our NJ1 server. We'll update this announcement as we have more information.

Sunday, 22 May 2018, 18:40 - server is back online without issue noted.

Resolved

FW intermittent network issues

 Monday, 21 May 2018, 07:49 - We're aware of the intermittent network issues plaguing our Dallas (DFW) POP. This seems to be due to heavy packet loss on our upstream's end. We're collaborating with our upstream to mitigate this, thank you for your patience.

Resolved

Kernel panic on dalhighram10

We have experienced a kernel panic on dalhighram10 and we are currently looking into it.

Update: All the VPS's booted back up again.

Resolved

Dallas network errors

Wednesday, 28 February 2018, 10:49 - We're investigating a Dallas connectivity issue. We'll update this announcement as we have more information. This is a network issue. We are experiencing high package loss. Our datacenter is working on the issue.

Wednesday, 28 February 2018, 11:41 - Root cause has been determined: outbound network attacks from insecure memcached installations. More info here: https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ We're looking at how to adjust them now. 

Wednesday, 28 February 2018, 12:28 - This has been mitigated, root cause was a new memcached amplification attack. If you're still seeing connectivity issues, or truly need internet-accessible memcached access for some reason, please ticket in. 

Resolved

nj1 network connectivity

Sunday, 25 February 2018, 13:48 - we're investigating an outage of our NJ1 server. We'll update this announcement as we have more information. 

Sunday, 25 February 2018, 14:52 - the server is back online, and we're awaiting confirmation that the network is stable

Resolved

dalstorage1 unreachable

Tuesday, 20 February 2018, 02:55 - We've been made aware of an event on this node and are currently investigating.
Tuesday, 20 February 2018, 05:16 - Node is back up, this was due to a storage event. VMs are now booting back up. Please expect to be contacted individually about remedies.

Resolved

dalhighram25 unreachable

The node is currently unreachable from the panel and the VMs are working extremely slowly. We're already looking into the issue but it has already taken more time than expected. Please excuse us for the inconvenience.

Update: The situation is under control now, please open a ticket if you're still having issues.

Resolved

Dallas network problems

Tuesday, 06 February 2018 11:34 - We're receiving reports of an unexpected partial routing failure somewhere, or similar network issue. Please don't stop/start/reboot your VPS, it won't fix the issue. When opening a ticket to inform us that your VPS is offline, could you please send a TEXT traceroute between yourself and your service with us? This will assist us with resolving the issue. 

Tuesday, 06 February 2018 11:59 -  The issue should have been rectified now. We'll update the announcement with a RFO.

Friday, 02 March 2018, 10:46 - Formal RFO is available at the following link: https://d.pr/f/RN2AjO

Resolved

ukhighram2 issues

ukhighram2 is currently facing issues. We are working on the problem at the moment.

Update: The server is back online now. We are investigating the reason for the lock up.

Resolved

Storage VPS Line Meltdown and Spectre Maintenance

We are going to reboot storage VPS nodes to patch them against Meltdown and Spectre vulnerabilities.

We are going to update this announcement once we have updates.

09:56 AM: All the nodes have been rebooted and all the VPS's are back online.

Resolved

Reboot for Meltdown / Spectre patching

Today, Saturday January 6th, 2018, we will be rebooting host nodes serving all VPSDime Linux VPS to execute a new kernel patched against the recently discovered Meltdown and Spectre exploits.

No doubt that in recent days you have noted news that two vulnerabilites have been discovered in all CPUs, ranging back to the development of speculative execution on modern processors over 20 years ago. Although both attacks are based on the same general principle, Meltdown allows malicious programs to gain access to higher-privileged parts of a computer's memory, while Spectre steals data from the memory of other applications running on a machine. Nearly every modern computer in use is vulnerable at some level.

Our virtualization vendor for the Linux VPS product has recently produced a patch to mitigate these two issues, and this patch has been generally accepted by the computing community at large as being an effective method to ensure that the exploit is ineffective. At this time there is no evidence that the exploit has been applied to gain access to other systems, either on our host nodes or elsewhere.

Due to the grievous nature of these attacks, we are updating all host nodes and rebooting them immediately to execute the new kernel. Note that all vendors of kernel live patching are either taking a very long time to implement a live patch (CloudLinux's Kernelcare) or have directly stated that a live patch won't be available (Canonical's Livepatch). Therefore, a reboot of the host node is mandatory in this situation.

We apologize for the inconveience that you will experience due to this, however, please note that we are committed to the utmost stability and security of your virtual machines.

There are some key points to note:

- We'll use this page to make further announcements regarding this reboot action. So please follow up with this page for the updates.

- All VPS will be gracefully shut down, similar to how the "shutdown -h now" or "halt" command works inside your VPS' operating system.

- This update and reboot action will make absolutely no changes to your VPS whatsoever. No changes will be made to your configuration, software, applications, data, or anything else regarding your VPS directly. All VPS will be checked that they are running after the reboot, however, if an application or software you've installed is not responding, please log in to your VPS and troubleshoot your software and make sure your services are running before opening a ticket.

- Barring any unexpected issues, we expect no more than 30 minutes per host node to reboot. This is the time for VPS to shut down and host node to complete a reboot; your VPS may take a bit longer to boot after the host node comes back online. Your patience is appreciated.

Please note that we are unable to provide a certain time frame for your VPS to be shut down because we are going to apply the security patch on one host node at a time. Again, we apologize for the inconvenience of a reboot.

If you have any questions or concerns regarding this, please click here and open a ticket with us.


============

We're working through the reboots now.

If you view your service and see "This node is locked", the node has not yet been rebooted.

If you view your service and see the usual start/stop/reboot buttons and traffic/memory/load graphs, the reboot action has been completed for that node and any services on that node will remain online.

============

All the nodes in Dallas, TX but 2 have been rebooted on Saturday January 6th, 2018.

============

Sunday January 7th, 2018:

We are continuing to reboot the nodes in our other locations today.

Resolved

ukhighram1 issues

ukhighram1 is currently facing issues. We are working on the problem at the moment.

Update: The server is back online now. We are investigating the reason for the lock up.

Resolved

dalhighram9 availability issues

Monday, 01 January 2018, 04:40 CDT - We are looking into a problem on this node that is affecting availability. Please stay tuned for further updates.
Monday, 01 January 2018, 04:55 CDT - Node has had to be taken down for an emergency reboot, it is currently booting back up.
Monday, 01 January 2018, 04:59 CDT - Node is online. VPS's are booting up at the moment. Your VPS should be online in the next 5 minutes.

Resolved

seapure3 emergency reboot

Friday, 29 December 2017, 02:01 - seapure3 has kernel panicked, and had to go through an emergency reboot.

Friday, 29 December 2017, 02:09 - The server is back online. VPS's are now booting up. All VPS's should be online in the next 10 minutes.

Resolved

UK Packet loss

Tuesday, 14 November 2017, 02:30 - It has come to our attention that some of our UK nodes (UKHighram { 1 | 2 } in particular) are currently experiencing rather high levels of packet loss. This appears to be being caused by an issue on our upstream provider's side. They have been ticketed accordingly, and we're awaiting response.
Tuesday, 14 November 2017, 04:41 - This is now fixed, root cause analysis and a formal RFO from our providers is still pending.

Resolved

VPS control system issues

09/30/2017 13:35 - we are aware of an issue with the communication of our client area against our VM control system. Your VPS is likely online; please check with ping/SSH before opening a ticket that your VPS is offline. Please open a ticket if you need any start or rebuild actions performed on your VM until this is fixed.

Resolved

dalpure11 responsiveness

Sunday, 13 August 2017, 10:41 - We are investigating a responsiveness issue with our dalpure11 server. We will post more information here as we have it. 

Sunday, 13 August 2017, 11:26 - We have discovered a failing RAM module in this server and are working to replace it now. We will post more information here as we have it. 

Sunday, 13 August 2017, 11:45 - We have replaced all the memory sticks to avoid any future problems and the server has been booted into the OS. VPS's are starting at the moment. Your VPS should be online momentarily if not already. Please note there may be some slowness for the first 30 mins.

Resolved

Scheduled maintenance for ukpure11

We have been notified by our datacenter that they have to physically move ukpure11 server to another rack to improve their infrastructure.

The date and time booked for this maintenance is Sunday 07/30/2017 10:00 AM GMT+1.

They anticipate a 20 minute downtime due to this maintenance.

We are sorry the inconvenience this maintenance that may cause you and we thank you for your understanding.

If you have any questions or concerns regarding this, please click here and open a ticket with us.

--------------------------------

Update 03:59 AM CST: We are shutting down the VPS's right now.

Update 04:04 AM CST: The server is powered off and the datacenter technicians is now working on to move it to the new rack.

Update 04:14 AM CST: The server is moved and it's powered back on and the VPS's are now starting up. The load may be high at this time and your VPS could be slow. This should be resolved in the next few minutes.

Update 04:25 AM CST: Everything is back to normal.