Ulster Bank IT problems: What went wrong

It took until 10 July to solve the Ulster Bank problem while the Nat West situation had been largely fixed by 25 June

The problems at the RBS Group were centred on a botched IT upgrade - Ulster Bank customers were worst hit.

Every night bank IT systems process the day's business, bringing all account balances and transaction information up to date.

The different processes, which include counter transactions and making changes to standing orders, are referred to as "batch jobs".

Each batch job performs a specific task within the bank's systems and needs to be run in the correct logical sequence, relative to the others.

A "batch scheduler" is the software that executes the sequence.

On Sunday 17 June 2012, the software shared by Ulster Bank and the Nat West bank was upgraded.

The following day problems began to emerge - long response times and failures were occurring.

By the Tuesday, the fateful decision was made to reverse the upgrade.

But the upgrade had reformatted the data in the batch jobs and the older software now could not read the data.

Red alert

This meant transactions could not be processed and a "major red" alert was initiated by RBS.

IT staff focussed on manually reloading the batch jobs.

By the morning of Wednesday 20 June 2012, the NatWest batch was largely completed and account balances were up to date for the coming day, albeit more than four hours later than usual.

However, for Ulster Bank it was a different story.

It started the business day on the Wednesday with a significant backlog to batch processing.

By the Thursday, things became much worse - one day's batch started processing before the other day's was finished.

The two batches interfered with each other causing additional problems and further backlogs.

This snowball effect meant that it took until 10 July to solve the Ulster Bank problem while the Nat West situation had been largely fixed by 25 June.

RBS has now changed IT processes so that each of its brands now has its own dedicated batch scheduler.

Within Ulster Bank, the Northern Ireland business and the Republic of Ireland business also have separate schedulers.

All this mean should mean if there is a problem in the future it can be fixed much more quickly.