Service Status

Outage in Dallas

Our datacenter is reporting an issue with one of their switches: https://twitter.com/InceroStatus/status/648695085387747328?lang=en

This has killed connectivity to the nodeXssd and dalyearly servers, along with daldb. This announcement will be updated as we know more.

=====

This is now resolved, and all services should now be back online: https://twitter.com/InceroStatus/status/648706618335473664

Downage in Dallas. We are working on the situation.

ukpure2 reboot

seapure1 is unresponsive

seapure1 is rebooted

ukpure1 is rebooted

Network issues with LA servers

dalpure3 reboot

Major LA Packetloss

node3 outage

dalwest2 high packetloss

vlan outage in Dallas

rebooting dalpure7

dalstorage3 kernel panic

We have faced yet another kernel panic on dalstorage3. The server is rebooting.

We are going to send a replacement for this server and migrate the data.

Update: The server is offline. We are working on a solution.

Update: The server was keep rebooting itself. We are thinking that this was due to half applied patches by Kernel care. The last kernel has been reinstalled to avoid this issue. We are hoping this would resolve the problems on this node.

Update: Unfortunately, this looks like an hardware issue. We are going to have to replace the mainboard and the memory of this server, since it can't stay online.

Update: We're still working through the situation. We're working to boot the system so that we can disable the VM filesystem, and then remove one of the hardware components that seems to be causing issues.

Update: We have replaced the motherboard, CPUs and the memory on the server. After doing fsck 3 times which took around 20 hours alone, the server has booted and is currently booting the VPS's.

Network Issue affecting Dallas

dalstorage3 is offline

dalstorage3 is rebooted, back online

15:44 :investigating.
Node appears to have kernel panicked

15:48: node back online, VMs are booting

dalstorage4 failure, replacement

08:09: Our dalstorage4 storage server is being rebooted for a filesystem check. This announcement will be updated when complete.

08:55: This didn't go quite as planned. We are working the situation and will provide more information when we can.

09:17: All clients on this server have been emailled the following:

===

Hello,

We are writing to you because you have active service on our "dalstorage4" server.

The reason for the message is to inform you of a disk controller or disk error with this server. This morning, the file system began throwing egregious errors. In an effort to salvage the situation, we rebooted the server to run fsck on the VM partition, only to discover that the disk array is no longer readable, and that fsck is throwing an error that every block is unreadable. Unfortunately, at this juncture it appears that your data is lost.

At this point, we've shipped a replacement server, which will be online tomorrow afternoon, relative to the United States.

We're offering two options:

1. We can recreate your VM tomorrow on the new server, and provide you with a full two months of credit for the issue.

2. We can provide a full refund back to your payment method.

Please note that we're unable to recover your data, and please do not open tickets regarding that. This has been a hard decision that we did not take lightly, and every effort has already been made to fully recover your data.

Please open a ticket with your choice, and if you have any other questions, please don't hesitate to task.

Thank you for your understanding,
-VPSDime Team

===

Planned storage VPS server move in progress

dalpure6 is down