Service Status

Resolved

dalpure4 Outage

We are aware that dalpure4 node is in Dallas is down. We are currently working on resolving the issue.

Update 01:40: We had to hard reset the node and currently it's booting up the VPS's. Your should be online in the following 15-20 minutes. Please open a ticket otherwise.

Resolved

Outage on dalwest4

10/27/2015 13:59 - We are investigating an outage of our dalwest4 server. We will update this announcement as we have more information.

10/27/2015 14:08 - We have determined the server is not showing a network link. We're looking further at the intermediate network hardware.

10/27/2015 15:02 - 10gbit NIC and fiber have been replaced, and the server is now back online.

10/28/2015 01:57 - We are facing the same hardware issue with the server again. We are working on getting this fixed ASAP.

10/28/2015 02:03 - We have got the server back online by downgrading it to 1G connection for now. We are going to schedule a complete server chassis swap in the US morning time.

Resolved

lapure1 rebooted

Node lapure1 requires a reboot, vms will be back up shortly

Resolved

Network outage in our Los Angeles location

We are aware of an network outage in our Los Angeles, USA location. We are currently looking into it.

Resolved

Outage in Dallas

Our datacenter is reporting an issue with one of their switches: https://twitter.com/InceroStatus/status/648695085387747328?lang=en

This has killed connectivity to the nodeXssd and dalyearly servers, along with daldb. This announcement will be updated as we know more.

=====

 

This is now resolved, and all services should now be back online: https://twitter.com/InceroStatus/status/648706618335473664

Resolved

Downage in Dallas. We are working on the situation.

Several nodes in Dallas are currently down. Our technicians are working to bring them back as soon as possible.

Resolved

ukpure2 reboot

ukpure 2 requires an emergency reboot and will be going down.

Update: server rebooted and appears to be back online

Resolved

seapure1 is unresponsive

seapure1 is currently unresponsive and we are working on it.

Update: This is resolved. However, some of the VPS's got auto suspended due to extreme load on the server. These are now unsuspended.

Resolved

seapure1 is rebooted

Due to a kernel panic, we have rebooted seapure1 and your VPS should come back online in the following minutes.

Resolved

ukpure1 is rebooted

Due to a lock up, we have rebooted ukpure1 node. Your VPS should come back online in the following minutes.

Resolved

Network issues with LA servers

Our upstream in LA is currently encountering some network issues. We are looking into it.

Resolved

dalpure3 reboot

We are emergency rebooting dalpure3. It should come back in 10 minutes.

Update: All VPS's are back online.

Resolved

Major LA Packetloss

Investigating High Packetloss in our LA location. At this time it appears to be DC wide.

This appears to be resolved now

Resolved

node3 outage

We're currently investigating an outage of our node3 server in LA.

Update: This is resolved.

Resolved

dalwest2 high packetloss

We are investigating high packetloss on node dalwest2.

Packetloss should be resolved. Cause was incoming DDOS

Resolved

vlan outage in Dallas

2015-02-11 2039 CST: We are acknowleding a vlan outage in Dallas at the moment. We're engaging the datacenter to look into the issue. If your server is "down" and this announcement does not say [SOLVED], yes, your server is on this vlan.

2015-02-11 2055 CST: From the datacenter, this encompasses more than just us. Their twitter feed: https://twitter.com/InceroStatus

2015-02-11 2102 CST: Datacenter fixed issue, network back online. 

Resolved

rebooting dalpure7

We are rebooting dalpure7 at the moment. Your VPS should come back online in 10 minutes.

We are frequently having lock ups on this node and we are planning to migrate VPS's to a new node soon.

Update: All VPS's are back online.

Resolved

dalstorage3 kernel panic

We have faced yet another kernel panic on dalstorage3. The server is rebooting.

We are going to send a replacement for this server and migrate the data.

Update: The server is offline. We are working on a solution.

Update: The server was keep rebooting itself. We are thinking that this was due to half applied patches by Kernel care. The last kernel has been reinstalled to avoid this issue. We are hoping this would resolve the problems on this node.

Update: Unfortunately, this looks like an hardware issue. We are going to have to replace the mainboard and the memory of this server, since it can't stay online.

Update: We're still working through the situation. We're working to boot the system so that we can disable the VM filesystem, and then remove one of the hardware components that seems to be causing issues.

Update: We have replaced the motherboard, CPUs and the memory on the server. After doing fsck 3 times which took around 20 hours alone, the server has booted and is currently booting the VPS's.

Resolved

Network Issue affecting Dallas

We are facing a network issue that's affecting dalstorage2 dalstorage3 dalstorage4 dalpure7 and dalpure8.

This will be fixed in a few minutes.

Update: 21:18: We are back up.

Resolved

dalstorage3 is offline

dalstorage3 is offline and we are working on it.

Update: The server has kernel paniced. We have rebooted it and it's currently running fsck.

Update: The serer has booted, it's booting VPS's. Your VPS should come online in the following 20 minutes of this post (3:56 AM CST)