[26/11/09] Datacentre power issues

NetHosted Staff Joined: 22 Mar 2004 Posts: 7017

Hi,

We experienced an outage today due to a power cut in the Stratford area which affected our services hosted at our Stratford datacentre. We believe that they had an issue with starting the power generator and whilst the UPS power supplies worked they were not sufficient to cover the outage without the backup generator.

We apologise for the inconvenience caused by this and will post a full breakdown of the causes and solutions that will be implemented to ensure this does not happen in future when we have discussed this with the datacentre.

Regards,

Andrew

NetHosted Staff Joined: 22 Mar 2004 Posts: 7017

Hi,

We have been made aware that the current power situation at the datacentre is temporary and there will be an additional downtime for the nethosted.co.uk, Neptune, Uranus, Saturn and Jupiter servers to follow.

This downtime will be 5-10 minutes while power is restored through the correct, redundant channels.

The exact time of this future outage is not yet known but we will post an update as soon as we have one.

Thanks,

Andrew

NetHosted Staff Joined: 24 Jun 2005 Posts: 1013

Hi,

We have subsequently been informed that no restart will be necessary.

Regards,

Darryl

NetHosted Staff Joined: 22 Mar 2004 Posts: 7017

Hi,

We have now been sent a more complete analysis of what happened on the 26th.

The cause of this fault was not the generators failing to start as we were told during the incident. In fact the UPS's and generators all worked without issue up until the point when the mains power was restored to the datacentre. When the generators detected this they shut down and tried to move back to the mains supply but there was a problem with the switching gear designed to facilitate this move. It failed to draw the power it required from either it's battery or the mains source to allow the facility to return to the mains power. The generators could not be turned back on because this switch was now stuck in the off position and required power (it could not receive) to operate. This problem took some time to diagnose but once identified the switch was hooked up to additional power and was in effect "jump started" to allow it to perform it's function of returning the mains power to the facility.

This type of failure has been prevented from occurring in the future by installing a protected power supply to the switch and better backup batteries. Additional steps have also been taken to add further redundancy. This work was completed on the night of the incident and the system has been tested and proved to be working with the new setup.

Future changes to the testing policies in place to try and detect and test for edge cases such as this before they become problems are being discussed and planned for implementation as soon as possible.

The datacentre was at pains to mention that there have been multiple mains power failures in the last 5 years all of which were dealt with by their systems flawlessly and without any loss of power.

We at NetHosted have made some changes to our systems too to prevent the complete blackout of our site and email that was triggered by the datecentre downtime. Should an incident such as this occur again our email systems should seamlessly move over to a backup system and our website should have minimal downtime (5 minutes or so) while DNS changes occur. These measures will prevent the complete loss of contact via nethosted.co.uk that occurred during this incident.

Apologies once again for this unexpected downtime and any inconvenience caused by it.

Thanks,

Andrew

Client Support	Community	Server Status	Contact Us	Client Login
Email Hosting	Website Hosting	Reseller Hosting	VPS Hosting	Dedicated Servers