Tuesday, August 7, 2018
Tuesday, 07 August 2018, 00:08 - Earlier today, our dalstorage4 node suffered a disk failure and it is currently operating in a degraded state. There unfortunately isn't enough I/O to go around, so I/O bound programs are taking a very long time to respond (due to the I/O wait). We're investigating, further updates will be posted to this issue. Customers not on the dalstorage4 node are not affected by this issue.
Wednesday, 08 August 2018, 09:03 - This situation does remain in progress. Your VPS should be online and accessible though, since the I/O contention seems to come in short infrequent waves. If you're having an issue with access beyond being slower than usual, please inquire and we can investigate.
Thursday, 09 August 2018, 08:23 - This continues to be in progress. Reboots via the client area/VM control system have been disabled as rebooting won't magically fix this situation and will only contribute to more I/O contention. If you need your VPS rebooted and are truly unable to figure out how otherwise, please ticket in and we'll review your request.
Friday, 10 August 2018, 11:46 - We're going to shut down the server to replace the drives and let it rebuild without VM contention; this will take less time than if we left VMs running, resulting less net time back to normal. This shouldn't have much of an impact on actual usage given the speeds of the server as it stands. The current estimate is 24 hours for array rebuild.
Saturday, 11 August 2018, 06:25 - All the processes successed and we have booted up the VPS's. Everything should be healthy at this point.