Guernsey States IT outages caused by old kit and lack of maintenance

The review said how quickly services will be moved out of the old or legacy data rooms into new centres should be reassessed

A failure of back-up systems and a reliance on out of date kit were behind a number of IT outages suffered by the States of Guernsey, a review has found.

The failure of air conditioning and power units caused four data outages between November and January making internal systems and the States website unavailable.

PwC's review found concerns about the data centres dating back years.

It recommended the States makes urgent changes to avoid further outages.

"The review is quite clear, there are failings we must acknowledge and actions that were missed that could have prevented these outages," said Deputy Peter Ferbrache, President of the Policy and Resources Committee.

"The review further highlights the need for a better, more resilient IT infrastructure and that we must get there faster.

"The consequences of these outages could have been much more severe this does not detract from the significant challenges which were experienced across a number of government services.

"Now we must and are acting quickly to ensure resilience improves and we reduce the risk of another incident."

When asked he said he could not put a figure on how much speeding up the IT transformation programme would cost.

Issues identified

• Delays in moving older systems to new data centres

• Lack of clarity on who was responsible for old data room equipment

• Lack of maintenance

• No continuity plan in place

• Actions were not taken after two "near miss events" at data centres at the hospital and Edward T Wheadon House

• Automated move of platforms between locations did not work as expected

• Air conditioning was not resilient

• Uninterruptible Power Supplies did not operate as expected

• The generator that was in place was not in service

• Monitoring and alerting of the equipment in the data rooms was not effective

The PwC report found ownership of the data centre infrastructure and responsibility for maintenance was unclear - it was not covered by the 10-year contract signed with IT provider Agilisys in 2019.

An example of this was one of the two air conditioning units at the Sir Charles Frossard House date centre failed in June 2022 and was not fixed when the second failed in November.

Maintenance contracts for the Uninterruptible Power Supplies and generator had expired in 2020 and 2021 respectively, and data storage support had only been in place for the hardware since 2020.

Among its 30 recommendations PwC suggested a review of the contract to identify any other undefined areas, and assigning responsibility for air conditioning and back-up power.

Andrew Mindenhall, Agilisys chief executive said: "We know outages like these are not what islanders expect and we're committed to working with the States on delivering its plan to improve resilience and mitigate the risks."

Analysis by BBC Guernsey political reporter John Fernandez

The IT outage wasn't just about a website going down - it meant people's benefits not being paid, children unable to learn and civil servants left unable to do their job.

This report by independent experts exposes failure upon failure.

Sources told me in November that "heads should roll" for this outage but there is an argument the people responsible for the delineation of responsibility that led to it have already gone or are no longer in government.

The States is now staring at a big bill to bring things up to standard when it comes to IT - at a time when government finances are straining at the seams.

One ray of light from this report is that it recommends a review of the contract with Agilisys to make sure there are no other oversights.

Follow BBC Guernsey on Twitter and Facebook. Send your story ideas to [e-mail].