Overall Service Status
Up
All services are operating normally at this time.
All Regions
| Data Center | Current | Last Day | Last Month | Last Year |
|---|---|---|---|---|
|
Up | 100% | 100% | 100% |
|
|
Up | 100% | 100% | 100% |
|
|
Up | 100% | 100% | 100% |
|
|
Up | 100% | 100% | 100% |
|
Up | 100% | 100% | 100% |
|
|
Up | 100% | 100% | 100% |
|
Up | 100% | 100% | 100% |
Past Incidents
Network issues with LA servers
Our upstream in LA is currently encountering some network issues. We are looking into it.
dalpure3 reboot
We are emergency rebooting dalpure3. It should come back in 10 minutes.
Update: All VPS's are back online.
Major LA Packetloss
Investigating High Packetloss in our LA location. At this time it appears to be DC wide.
This appears to be resolved now
node3 outage
We're currently investigating an outage of our node3 server in LA.
Update: This is resolved.
dalwest2 high packetloss
We are investigating high packetloss on node dalwest2.
Packetloss should be resolved. Cause was incoming DDOS
vlan outage in Dallas
2015-02-11 2039 CST: We are acknowleding a vlan outage in Dallas at the moment. We're engaging the datacenter to look into the issue. If your server is "down" and this announcement does not say [SOLVED], yes, your server is on this vlan.
2015-02-11 2055 CST: From the datacenter, this encompasses more than just us. Their twitter feed: https://twitter.com/InceroStatus
2015-02-11 2102 CST: Datacenter fixed issue, network back online.
rebooting dalpure7
We are rebooting dalpure7 at the moment. Your VPS should come back online in 10 minutes.
We are frequently having lock ups on this node and we are planning to migrate VPS's to a new node soon.
Update: All VPS's are back online.
dalstorage3 kernel panic
We have faced yet another kernel panic on dalstorage3. The server is rebooting.
We are going to send a replacement for this server and migrate the data.
Update: The server is offline. We are working on a solution.
Update: The server was keep rebooting itself. We are thinking that this was due to half applied patches by Kernel care. The last kernel has been reinstalled to avoid this issue. We are hoping this would resolve the problems on this node.
Update: Unfortunately, this looks like an hardware issue. We are going to have to replace the mainboard and the memory of this server, since it can't stay online.
Update: We're still working through the situation. We're working to boot the system so that we can disable the VM filesystem, and then remove one of the hardware components that seems to be causing issues.
Update: We have replaced the motherboard, CPUs and the memory on the server. After doing fsck 3 times which took around 20 hours alone, the server has booted and is currently booting the VPS's.
Network Issue affecting Dallas
We are facing a network issue that's affecting dalstorage2 dalstorage3 dalstorage4 dalpure7 and dalpure8.
This will be fixed in a few minutes.
Update: 21:18: We are back up.
dalstorage3 is offline
dalstorage3 is offline and we are working on it.
Update: The server has kernel paniced. We have rebooted it and it's currently running fsck.
Update: The serer has booted, it's booting VPS's. Your VPS should come online in the following 20 minutes of this post (3:56 AM CST)
dalstorage3 is rebooted, back online
15:44 :investigating.
Node appears to have kernel panicked
15:48: node back online, VMs are booting
dalstorage4 failure, replacement
08:09: Our dalstorage4 storage server is being rebooted for a filesystem check. This announcement will be updated when complete.
08:55: This didn't go quite as planned. We are working the situation and will provide more information when we can.
09:17: All clients on this server have been emailled the following:
===
Hello,
We are writing to you because you have active service on our "dalstorage4" server.
The reason for the message is to inform you of a disk controller or disk error with this server. This morning, the file system began throwing egregious errors. In an effort to salvage the situation, we rebooted the server to run fsck on the VM partition, only to discover that the disk array is no longer readable, and that fsck is throwing an error that every block is unreadable. Unfortunately, at this juncture it appears that your data is lost.
At this point, we've shipped a replacement server, which will be online tomorrow afternoon, relative to the United States.
We're offering two options:
1. We can recreate your VM tomorrow on the new server, and provide you with a full two months of credit for the issue.
2. We can provide a full refund back to your payment method.
Please note that we're unable to recover your data, and please do not open tickets regarding that. This has been a hard decision that we did not take lightly, and every effort has already been made to fully recover your data.
Please open a ticket with your choice, and if you have any other questions, please don't hesitate to task.
Thank you for your understanding,
-VPSDime Team
===
Planned storage VPS server move in progress
Further to the email we sent out regarding the server move for dalstorage2 and dalstorage3, this is currently ongoing. We'll update this announcement when complete.
Update: dalstorage3 is shut down and is being physically moved at the moment.
Update: dalstorage2 is also shut down and is being physically moved.
Update: Both servers are moved. VPS's are booting on dalstorage2. dalstorage3 has all VPS's booted.
Update: All VPS's are online.
dalpure6 is down
We are aware that dalpure6 is down. We are working on it.
Update: It was a kernel panic. We have rebooted the server and your VPS should come back online soon.
Update: Now, the node is having a network issue. We are working on it.
Update: This issue is resolved. It was caused by multiple 10G DDoS attacks to multiple IPs that our filtering couldn't handle.
node3 is being rebooted
node3 has again faced IO lock up issues and is being rebooted. Your VPS should come back online in the following 30 minutes.
Update: 09:27 AM: All VPS's are online.
dalpure1 is under DDoS attack
dalpure1 is under DDoS attack. We are working on it.
Update: 07:34: This is mitigated.
dalstorage3 downtime
dalstorage3 is offline and we are investigating.
Update 05:06: It was a kernel panic. We have power cycled the server and it's doing a FSCK now. VPS's should be back online in around two hours or so.
Update 07:40: Automated FSCK has failed and a manual one is forced. We are performing that now.
Update 10:22: FSCK is completed with success. VPS's are booting.
Update 11:59: All VPS's are online except 2 which are calculating quota.
node3 is rebooted
node3 has been rebooted due to IO lock up. We are going to migrate this server to a pure SSD server soon to avoid these issues.
dalstorage2 is rebooting
We are rebooting this server (dalstorage2) due to soft lock up. Please be patient until your VPS boots back.
Update 06:59 AM: All VPS's are booted. Please open a ticket if you are having troubles.
lapure2 is having hardware issues
lapure2 is having network hardware issues. We are investigating.
Update 7:36 AM: No updates, yet.
Update 8:35 AM: Turns out the onboard network port was faulty. We are using the other port and network is now stable. The server is currently booting. Your VPS's should be online in the next 15 minutes.
Update 8:39 AM: We are still investigating the packet loss issue.
Update 9:17 AM: The issue is resolved.
Here is the RFO:
The issue started with a major packet loss on this node. We couldn't even get into the shell to check.
First, we have suspected a DDoS attack and alerted our datacenter. They told us that there was no attack.
Then we've gotten them to replace our cables as well as their cables to the top of the rack switch. This has not solved the problem.
Then we've gotten them to swap the network cable to the other ethernet port. Unfortunately, this has not solved the problem, either.
After eliminating the hardware issues, we have started to troubleshoot the issue (very slowly, due to the high load caused by the attack!) on the IPMI console further. And then we have realized that there was indeed an attack on one of the IPs which our DC failed to recognize.
We have nullrouted the target IP manually and the packet loss issue is gone.
Your VPS is currently up and running.
We are terribly sorry for the downtime this has caused you and we want to make it up to you. Please open a ticket to redeem $5 credit on your account for the trouble.