Service Status

Resolved

Kernel panic on dalhighram10

We have experienced a kernel panic on dalhighram10 and we are currently looking into it.

Update: All the VPS's booted back up again.

Resolved

Dallas network errors

Wednesday, 28 February 2018, 10:49 - We're investigating a Dallas connectivity issue. We'll update this announcement as we have more information. This is a network issue. We are experiencing high package loss. Our datacenter is working on the issue.

Wednesday, 28 February 2018, 11:41 - Root cause has been determined: outbound network attacks from insecure memcached installations. More info here: https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/ We're looking at how to adjust them now. 

Wednesday, 28 February 2018, 12:28 - This has been mitigated, root cause was a new memcached amplification attack. If you're still seeing connectivity issues, or truly need internet-accessible memcached access for some reason, please ticket in. 

Resolved

nj1 network connectivity

Sunday, 25 February 2018, 13:48 - we're investigating an outage of our NJ1 server. We'll update this announcement as we have more information. 

Sunday, 25 February 2018, 14:52 - the server is back online, and we're awaiting confirmation that the network is stable

Resolved

dalstorage1 unreachable

Tuesday, 20 February 2018, 02:55 - We've been made aware of an event on this node and are currently investigating.
Tuesday, 20 February 2018, 05:16 - Node is back up, this was due to a storage event. VMs are now booting back up. Please expect to be contacted individually about remedies.

Resolved

dalhighram25 unreachable

The node is currently unreachable from the panel and the VMs are working extremely slowly. We're already looking into the issue but it has already taken more time than expected. Please excuse us for the inconvenience.

Update: The situation is under control now, please open a ticket if you're still having issues.

Resolved

Dallas network problems

Tuesday, 06 February 2018 11:34 - We're receiving reports of an unexpected partial routing failure somewhere, or similar network issue. Please don't stop/start/reboot your VPS, it won't fix the issue. When opening a ticket to inform us that your VPS is offline, could you please send a TEXT traceroute between yourself and your service with us? This will assist us with resolving the issue. 

Tuesday, 06 February 2018 11:59 -  The issue should have been rectified now. We'll update the announcement with a RFO.

Friday, 02 March 2018, 10:46 - Formal RFO is available at the following link: https://d.pr/f/RN2AjO

Resolved

ukhighram2 issues

ukhighram2 is currently facing issues. We are working on the problem at the moment.

Update: The server is back online now. We are investigating the reason for the lock up.

Resolved

Storage VPS Line Meltdown and Spectre Maintenance

We are going to reboot storage VPS nodes to patch them against Meltdown and Spectre vulnerabilities.

We are going to update this announcement once we have updates.

09:56 AM: All the nodes have been rebooted and all the VPS's are back online.

Resolved

Reboot for Meltdown / Spectre patching

Today, Saturday January 6th, 2018, we will be rebooting host nodes serving all VPSDime Linux VPS to execute a new kernel patched against the recently discovered Meltdown and Spectre exploits.

No doubt that in recent days you have noted news that two vulnerabilites have been discovered in all CPUs, ranging back to the development of speculative execution on modern processors over 20 years ago. Although both attacks are based on the same general principle, Meltdown allows malicious programs to gain access to higher-privileged parts of a computer's memory, while Spectre steals data from the memory of other applications running on a machine. Nearly every modern computer in use is vulnerable at some level.

Our virtualization vendor for the Linux VPS product has recently produced a patch to mitigate these two issues, and this patch has been generally accepted by the computing community at large as being an effective method to ensure that the exploit is ineffective. At this time there is no evidence that the exploit has been applied to gain access to other systems, either on our host nodes or elsewhere.

Due to the grievous nature of these attacks, we are updating all host nodes and rebooting them immediately to execute the new kernel. Note that all vendors of kernel live patching are either taking a very long time to implement a live patch (CloudLinux's Kernelcare) or have directly stated that a live patch won't be available (Canonical's Livepatch). Therefore, a reboot of the host node is mandatory in this situation.

We apologize for the inconveience that you will experience due to this, however, please note that we are committed to the utmost stability and security of your virtual machines.

There are some key points to note:

- We'll use this page to make further announcements regarding this reboot action. So please follow up with this page for the updates.

- All VPS will be gracefully shut down, similar to how the "shutdown -h now" or "halt" command works inside your VPS' operating system.

- This update and reboot action will make absolutely no changes to your VPS whatsoever. No changes will be made to your configuration, software, applications, data, or anything else regarding your VPS directly. All VPS will be checked that they are running after the reboot, however, if an application or software you've installed is not responding, please log in to your VPS and troubleshoot your software and make sure your services are running before opening a ticket.

- Barring any unexpected issues, we expect no more than 30 minutes per host node to reboot. This is the time for VPS to shut down and host node to complete a reboot; your VPS may take a bit longer to boot after the host node comes back online. Your patience is appreciated.

Please note that we are unable to provide a certain time frame for your VPS to be shut down because we are going to apply the security patch on one host node at a time. Again, we apologize for the inconvenience of a reboot.

If you have any questions or concerns regarding this, please click here and open a ticket with us.


============

We're working through the reboots now.

If you view your service and see "This node is locked", the node has not yet been rebooted.

If you view your service and see the usual start/stop/reboot buttons and traffic/memory/load graphs, the reboot action has been completed for that node and any services on that node will remain online.

============

All the nodes in Dallas, TX but 2 have been rebooted on Saturday January 6th, 2018.

============

Sunday January 7th, 2018:

We are continuing to reboot the nodes in our other locations today.

Resolved

ukhighram1 issues

ukhighram1 is currently facing issues. We are working on the problem at the moment.

Update: The server is back online now. We are investigating the reason for the lock up.

Resolved

dalhighram9 availability issues

Monday, 01 January 2018, 04:40 CDT - We are looking into a problem on this node that is affecting availability. Please stay tuned for further updates.
Monday, 01 January 2018, 04:55 CDT - Node has had to be taken down for an emergency reboot, it is currently booting back up.
Monday, 01 January 2018, 04:59 CDT - Node is online. VPS's are booting up at the moment. Your VPS should be online in the next 5 minutes.

Resolved

seapure3 emergency reboot

Friday, 29 December 2017, 02:01 - seapure3 has kernel panicked, and had to go through an emergency reboot.

Friday, 29 December 2017, 02:09 - The server is back online. VPS's are now booting up. All VPS's should be online in the next 10 minutes.

Resolved

UK Packet loss

Tuesday, 14 November 2017, 02:30 - It has come to our attention that some of our UK nodes (UKHighram { 1 | 2 } in particular) are currently experiencing rather high levels of packet loss. This appears to be being caused by an issue on our upstream provider's side. They have been ticketed accordingly, and we're awaiting response.
Tuesday, 14 November 2017, 04:41 - This is now fixed, root cause analysis and a formal RFO from our providers is still pending.

Resolved

VPS control system issues

09/30/2017 13:35 - we are aware of an issue with the communication of our client area against our VM control system. Your VPS is likely online; please check with ping/SSH before opening a ticket that your VPS is offline. Please open a ticket if you need any start or rebuild actions performed on your VM until this is fixed.

Resolved

dalpure11 responsiveness

Sunday, 13 August 2017, 10:41 - We are investigating a responsiveness issue with our dalpure11 server. We will post more information here as we have it. 

Sunday, 13 August 2017, 11:26 - We have discovered a failing RAM module in this server and are working to replace it now. We will post more information here as we have it. 

Sunday, 13 August 2017, 11:45 - We have replaced all the memory sticks to avoid any future problems and the server has been booted into the OS. VPS's are starting at the moment. Your VPS should be online momentarily if not already. Please note there may be some slowness for the first 30 mins.

Resolved

Scheduled maintenance for ukpure11

We have been notified by our datacenter that they have to physically move ukpure11 server to another rack to improve their infrastructure.

The date and time booked for this maintenance is Sunday 07/30/2017 10:00 AM GMT+1.

They anticipate a 20 minute downtime due to this maintenance.

We are sorry the inconvenience this maintenance that may cause you and we thank you for your understanding.

If you have any questions or concerns regarding this, please click here and open a ticket with us.

--------------------------------

Update 03:59 AM CST: We are shutting down the VPS's right now.

Update 04:04 AM CST: The server is powered off and the datacenter technicians is now working on to move it to the new rack.

Update 04:14 AM CST: The server is moved and it's powered back on and the VPS's are now starting up. The load may be high at this time and your VPS could be slow. This should be resolved in the next few minutes.

Update 04:25 AM CST: Everything is back to normal.

Resolved

dalstorage4 outage

5/30/2017 17:34 PM - Node dalstorage4 lost connectivity. Upon checking the serial console we found a kernel panic occured. 
5/30/2017 17:59 PM - All VPS are back onine.

Resolved

dalhighram17 degraded performance

We are aware of a performance degradation on dalhighram17. Our engineers are investigating the issue.

Resolved

lapure5 emergency reboot

05/03/2017 06:23 AM CST - We are going to reboot lapure5 to fix the high load average issue on it which was caused by a bug in KernelCare. We utilize KernelCare to do rebootless kernel upgrades but unfortunately a bug in the software is causing lock ups during the patching process. 
We will update this announcement once all VPS's are back online again.

05/03/2017 07:07 AM CST - All the VPS's are booted up now and we have confirmed the issue has been resolved.

Resolved

Dallas network connectivity (RFO Added)

04/26/2017 10:24 CST - We're aware of the network connectivity issue in Dallas and are working on the situation now. We'll update this announcement as we have more information.

04/26/2017 10:30 CST - We have pointed and resolved the issue. It will take around 5 mins before your VPS to come online if it's not already.

04/26/2017 10:54 CST - Everything should be back online now. A full Reason For Outage (RFO) report will be sent to ones who are affected shortly. We are sorry for the inconvenience.

Reason for Outage

This morning at around 10 AM CST we upgraded our Dallas core network switch and added an additional 10 Gbit capacity.

During the switchover to our new core switch, some VPS took longer than others to re-ARP into the VLAN. We had expected this to be transparent to all the customers based on the dry runs and tests we have processed during the previous weeks, but had a hard time getting the VLAN ARPed quickly enough.

Unfortunately, due to the slow re-ARP'ing, some of the IPs had around 5-25 minutes of downtime.

With this upgrade we now have full diverse path fibers configured to failover to avoid any interruptions due to physical factors.

We now also announce IPs under our own ASN, and can connect directly to other peering partners or carriers.

We can assure you that there won't be any more upgrades like these in the future as with this new setup, we have the ability to handle the new upgrades completely transparent.

We apologize for the inconvience that this unexpected network outage caused, however, we're committed to doing everything we can to ensure a consistent and reliable service at a great price.

If you have any questions or comments, please don't hesitate to inquire.