Service Status

Resolved

dalstorage3 is rebooted, back online

15:44 :investigating.
Node appears to have kernel panicked

 

15:48: node back online, VMs are booting

Resolved

dalstorage4 failure, replacement

08:09: Our dalstorage4 storage server is being rebooted for a filesystem check. This announcement will be updated when complete.

08:55: This didn't go quite as planned. We are working the situation and will provide more information when we can.

09:17: All clients on this server have been emailled the following:

===

Hello,

We are writing to you because you have active service on our "dalstorage4" server.

The reason for the message is to inform you of a disk controller or disk error with this server. This morning, the file system began throwing egregious errors. In an effort to salvage the situation, we rebooted the server to run fsck on the VM partition, only to discover that the disk array is no longer readable, and that fsck is throwing an error that every block is unreadable. Unfortunately, at this juncture it appears that your data is lost.

At this point, we've shipped a replacement server, which will be online tomorrow afternoon, relative to the United States.

We're offering two options:

1. We can recreate your VM tomorrow on the new server, and provide you with a full two months of credit for the issue.

2. We can provide a full refund back to your payment method.

Please note that we're unable to recover your data, and please do not open tickets regarding that. This has been a hard decision that we did not take lightly, and every effort has already been made to fully recover your data.

Please open a ticket with your choice, and if you have any other questions, please don't hesitate to task.

Thank you for your understanding,
-VPSDime Team

===

Resolved

Planned storage VPS server move in progress

Further to the email we sent out regarding the server move for dalstorage2 and dalstorage3, this is currently ongoing. We'll update this announcement when complete.

Update: dalstorage3 is shut down and is being physically moved at the moment.

Update: dalstorage2 is also shut down and is being physically moved.

Update: Both servers are moved. VPS's are booting on dalstorage2. dalstorage3 has all VPS's booted.

Update: All VPS's are online.

Resolved

dalpure6 is down

We are aware that dalpure6 is down. We are working on it.

Update: It was a kernel panic. We have rebooted the server and your VPS should come back online soon.

Update: Now, the node is having a network issue. We are working on it.

Update: This issue is resolved. It was caused by multiple 10G DDoS attacks to multiple IPs that our filtering couldn't handle.

Resolved

node3 is being rebooted

node3 has again faced IO lock up issues and is being rebooted. Your VPS should come back online in the following 30 minutes.

Update: 09:27 AM: All VPS's are online.

Resolved

dalpure1 is under DDoS attack

dalpure1 is under DDoS attack. We are working on it.

Update: 07:34: This is mitigated.

Resolved

dalstorage3 downtime

dalstorage3 is offline and we are investigating.

Update 05:06: It was a kernel panic. We have power cycled the server and it's doing a FSCK now. VPS's should be back online in around two hours or so.

Update 07:40: Automated FSCK has failed and a manual one is forced. We are performing that now.

Update 10:22: FSCK is completed with success. VPS's are booting.

Update 11:59: All VPS's are online except 2 which are calculating quota.

Resolved

node3 is rebooted

node3 has been rebooted due to IO lock up. We are going to migrate this server to a pure SSD server soon to avoid these issues.

Resolved

dalstorage2 is rebooting

We are rebooting this server (dalstorage2) due to soft lock up. Please be patient until your VPS boots back.

Update 06:59 AM: All VPS's are booted. Please open a ticket if you are having troubles.

Resolved

lapure2 is having hardware issues

lapure2 is having network hardware issues. We are investigating.

Update 7:36 AM: No updates, yet.

Update 8:35 AM: Turns out the onboard network port was faulty. We are using the other port and network is now stable. The server is currently booting. Your VPS's should be online in the next 15 minutes.

Update 8:39 AM: We are still investigating the packet loss issue.

Update 9:17 AM: The issue is resolved.

Here is the RFO:

The issue started with a major packet loss on this node. We couldn't even get into the shell to check.

First, we have suspected a DDoS attack and alerted our datacenter. They told us that there was no attack.

Then we've gotten them to replace our cables as well as their cables to the top of the rack switch. This has not solved the problem.

Then we've gotten them to swap the network cable to the other ethernet port. Unfortunately, this has not solved the problem, either.

After eliminating the hardware issues, we have started to troubleshoot the issue (very slowly, due to the high load caused by the attack!) on the IPMI console further. And then we have realized that there was indeed an attack on one of the IPs which our DC failed to recognize.

We have nullrouted the target IP manually and the packet loss issue is gone.

Your VPS is currently up and running.

We are terribly sorry for the downtime this has caused you and we want to make it up to you. Please open a ticket to redeem $5 credit on your account for the trouble.

Resolved

node1ssd kernel panic

node1ssd had a kernel panic and we are rebooting it now. Your VPS should come back online in around 15 minutes.

Update 02:22 AM: All VPS's are back up.

Resolved

Rebooting nj1

We are rebooting this node due to a unstability issue.

Resolved

Rebooting Colocrossing Los Angeles Server

We are doing an emergency reboot of this server. Please be patient.

Update 06:34 AM: This has been completed. All VPS's are online.

Update 11/30/2014 8:08 PM: The issue has come back and we are booting it again unfortunately.

Update 11/30/2014 8:31 PM: The node is back online.

Resolved

dalstorage3 Problem

We are currently working on to get the server back online.

Update 11:32: This server has been brought back.

Resolved

Network problem in Los Angeles

There is an ongoing network issue on one of the nodes in LA. If you are on it, your VPS is down at the moment. We are working on it to resolve ASAP.

Update 18:21: This is resolved. Reason for outage will follow...

Resolved

Dallas issues

We are having an issue with one server. If you are on that one, your VPS is down at the moment. Please be patient while we are looking into the issue.

Update: 14:10: This is resolved. It was a DDoS attack.

Resolved

UK Maintenance

ukpure1 is under an emergency maintenance at the moment. ETA to come back online is 20 minutes. We are sorry for the trouble.

Update 17:53: Unfortunately, this is taking very long. We are still waiting an update from the datacenter technicians.

Update 17:57: The server is back up.

Resolved

New Jersey Network Issues

We are experiencing netwoırk problems in our New Jersey location.

Update 13:46: Issue has been resolved.

Resolved

node3 Los Angeles Network Issues

One of our Los Angeles nodes is with ColoCrossing and they are having network issues. They are currently working on it.

Update: 20:25: The network is back online.

The cause of the issue was: "The cause for the incident was due to a bug in the firmware our switches utilize. It had a bit of an impact on us because we just recently completed a significant network wide upgrade of all cabinet level switches. Subsequently all of our cabinets run the same type of switch now, so when an issue occurs it can happen on more than 1 cabinet at once."

Resolved

node3ssd ddos

A large DDoS attack halted the NIC on node3ssd before we had any chance to take action. This is resolved and VMs are in the process of booting up.