NetHosted Community
https://www.nethosted.co.uk/community/

[15/02/11] Venus unavailable
https://www.nethosted.co.uk/community/viewtopic.php?f=9&t=3269
Page 1 of 1

Author:  NetHosted - Darryl [ Tue Feb 15, 2011 5:49 pm ]
Post subject:  [15/02/11] Venus unavailable

Hi,

Venus is currently being restarted, due to a possible raid issue this is taking longer than usual, we are working to get this back up and running ASAP.

Regards,

Darryl

Author:  NetHosted - Darryl [ Tue Feb 15, 2011 6:02 pm ]
Post subject: 

Venus is currently running a file system integrity check and should be back up once this finishes.

Regards,

Darryl

Author:  NetHosted - Darryl [ Tue Feb 15, 2011 6:31 pm ]
Post subject: 

The file integrity check is not running as expected unfortunately, we are working on alternative methods to get the server back up and running.

Regards,

Darryl

Author:  NetHosted - Darryl [ Tue Feb 15, 2011 7:25 pm ]
Post subject: 

The integrity check is currently running in rescue mode and appears to be progressing correctly, we are unsure as to how long this will take to complete at present but wanted to provide an update.

Regards,

Darryl

Author:  NetHosted - Darryl [ Tue Feb 15, 2011 8:32 pm ]
Post subject: 

Venus is back up and running, there may be some unexpected side-effects of the file system repairs, we are verifying many things manually now. If you notice any issues that did not exist prior to this please submit a support ticket and we will investigate.

Regards,

Darryl

Author:  NetHosted - Darryl [ Tue Feb 15, 2011 8:51 pm ]
Post subject: 

We will be posting an incident report in this thread once we have had time to analyse the cause of this, currently the priority is to ensure the server is running correctly and stable, once this has been fully established we will be investigating the cause of this, thanks for your understanding.

Regards,

Darryl

Author:  NetHosted - Andrew [ Wed Feb 16, 2011 4:43 pm ]
Post subject: 

Hi,

Here is the full incident report of last evening's events:

I was personally working in the datacentre when this happened. Upon attaching a console to Venus I could see numerous file system errors and took the decision to power off the server. Upon starting up again the RAID card had problems detecting the drives in place. Once this issue was fixed the operating system did not successfully complete it's standard file system integrity checks so I mounted the disks from a live CD and manually forced a "deep" file system inspection. This is what caused the bulk of the downtime that was seen as this process takes a long time to complete.

After this deep inspection the system was bootable and only required minor fixes to bring back all services. Since then the server has been running as expected and we believe this issue to be fully resolved.

Apologies to anyone that was affected by this unscheduled downtime.

Thanks,

Andrew

Page 1 of 1 All times are UTC + 1 hour
Powered by phpBB® Forum Software © phpBB Group
https://www.phpbb.com/