RBS and NatWest: The Rollback That Froze 6.5 Million Accounts
On Sunday 17 June 2012, RBS Group upgraded the software that updates NatWest and Ulster Bank accounts overnight. The first run struggled. On Tuesday the team rolled the upgrade back — without knowing the new version and the old version were incompatible. Jobs simply failed to appear in the queues. At least 6.5 million UK customers were affected, most RBS and NatWest systems were disrupted until 26 June, most Ulster Bank systems until 10 July, and in November 2014 the FCA and PRA fined the banks £56 million combined. The regulators' own words for the actual cause: a compatibility issue triggered by the rollback itself.
The short answer
In June 2012, the Royal Bank of Scotland Group — RBS, NatWest and Ulster Bank — upgraded the batch scheduler software that updates customer accounts overnight, from Version 1 to Version 2A. The first full run on Monday 18 June ran hot and failed in places, but staff got that night's work finished by hand. On Tuesday 19 June the team backed out the upgrade — uninstalled the new version and went back to the old one — without knowing the two versions were incompatible. That rollback is what the regulators call the actual cause of the incident: a significant number of jobs failed to appear in the batch queues, and unprocessed jobs multiplied. Disruption to the majority of RBS and NatWest systems lasted until 26 June 2012, and to the majority of Ulster Bank systems until 10 July 2012. At least 6.5 million customers in the United Kingdom were affected. On 19 November 2014 the Financial Conduct Authority (FCA) fined the banks £42 million and the Prudential Regulation Authority (PRA) fined them £14 million for breaching Principle 3 — the duty to organise their affairs with adequate risk management systems and controls.
The company: what RBS Group did
The Royal Bank of Scotland Group was one of the United Kingdom's major banking groups, headquartered in Edinburgh and serving, in the PRA's words, over 30 million customers worldwide. The three banks in this case are Royal Bank of Scotland plc, National Westminster Bank plc (NatWest) and Ulster Bank Ltd. The PRA notice records that RBS and NatWest together had over 26 million UK customers, 4,000 cash machines (ATMs) and 2,120 bank branches, and Ulster Bank in Northern Ireland had 450 ATMs and 90 branches.
A bank's core job sounds simple — keep accurate balances and move money when customers ask — but the machinery behind it runs at night. The notices explain it plainly: banks generally update that day's transactions in the evening. A software tool called a batch scheduler coordinates the order in which the data behind those updates is processed. The data covers customer withdrawals and deposits, interbank clearing (the system banks use to settle payments between each other, so a payee receives the full value of a cheque or standing order), money-market transactions, payroll processing, and requests to change standing orders and addresses. Each unit of work is called a job. The scheduler places jobs into queues and makes sure each job runs in the correct sequence. That day's batch processing is complete only when all balances are final.
Why this matters: the batch scheduler is not a back-office convenience. If it stops, balances stop being true. Everything a customer sees the next morning — the balance on the screen, the cash machine, whether a payment arrived — depends on last night's batch having finished, in order, completely.
The problem: an old scheduler, a shared system, and a known risk
Three facts set the trap, and all three are in the regulators' notices.
First, the old software had run out of support. Technology Services — the group's centralised IT function, which provided IT services to all three banks — upgraded the batch scheduler that processed updates for NatWest and Ulster Bank accounts because, in the PRA's words, the software could no longer be sufficiently supported. Version 2A, the upgrade, contained a modification known as a patch (a small change to existing software). Upgrading unsupported core software is normally the responsible thing to do. That is not where this case goes wrong.
Second, the systems were tangled together. A separate batch scheduler processed updates to RBS's own accounts. It did not stay separate in practice: the RBS scheduler was also affected because of interdependencies with the NatWest and Ulster Bank scheduler. One upgrade, in other words, could reach all three banks. The PRA later listed exactly this as a missed defence: the banks had not reduced the effect of a scheduler failure by, for example, reducing the number of jobs each scheduler managed, reducing the interdependencies between the schedulers, or using separate schedulers for NatWest and Ulster Bank.
Third, the risk had already been written down. The regulators date the breach period from 1 August 2010 — not because anything broke that day, but because a group internal audit on mainframe batch processes that month had identified the risk of a batch scheduler failure. The failure that froze millions of accounts in 2012 was, on paper, a known risk for about two years beforehand. A risk written in an audit report does not fix itself. Somebody still has to reduce it, test for it, or contain it — and the notices find that, in the end, none of that happened adequately.
An analogy: imagine a city's water pumps run by one old control program that the maker no longer supports. Replacing it is sensible. But the same control room also quietly steers a second city's pumps, the manual for switching back to the old program has never been tried for real, and an inspection two years earlier already flagged that a pump-control failure would be serious. The upgrade is not the reckless part. Doing it without a tested way back is.
What this shows: the mechanism the notices describe. The scheduler's whole job is order and completeness — which is why jobs silently missing from its queues were so damaging.
What happened
All dates and times are as given in the PRA Final Notice of 19 November 2014, which adopts the facts and matters in the FCA Final Notice printed as its Appendix 2. Read the first three days closely: the upgrade wobbles, the rollback breaks it, and then the backlog starts feeding on itself.
What this shows: three weeks on one line. The decisive moment is in the middle — not the Sunday upgrade, but the Tuesday decision to reverse it.
Sunday 17 June 2012 — the upgrade. A team from Technology Services upgraded the batch scheduler software that processed updates to customers' accounts at NatWest and Ulster Bank, from Version 1 to Version 2A, because the old software could no longer be sufficiently supported. A separate scheduler served RBS accounts — but, as the notices note in the same breath, it was affected anyway through its interdependencies with the NatWest and Ulster Bank scheduler.
Monday 18 June, evening — the first full run. Technology Services ran the first full batch since the upgrade. The mainframe computer was using a higher than normal percentage of its total processing capacity. That slowed the system down and caused several batch terminal failures — failures after which the batch could not continue on its own — which meant customers' accounts were not properly updated. The team escalated to internal IT experts, who re-ran the failed batches by entering commands into the system manually. That worked: the complete batch did run that night. It is worth pausing here, because this is the moment the story could have ended as a bad evening. The new version was misbehaving, but the work got done.
Tuesday 19 June — the rollback. Technology Services backed out the software upgrade: they uninstalled Version 2A and returned to the previous version. What the team did not know — the PRA's words — is that the upgraded version of the software was not compatible with the version that had been in place prior to the upgrade. The consequence was quiet and therefore worse than a crash: a significant number of jobs failed to appear in the batch queues, and increasing numbers of unprocessed batch jobs began to multiply. Nothing announced itself as broken. Work simply never arrived to be done.
The recovery grind, 19–21 June. Technical support staff focused on manually re-loading jobs into the batch queues — the standard manual intervention when a number of batch jobs fail to run. By the morning of Wednesday 20 June, the NatWest batch for 19 June was largely completed. Ulster Bank was another story: its batches were unfinished, leaving a significant backlog at the start of the working day. By 21 June, Ulster Bank batch processing was more than one day behind. That delay created a second, nastier problem: the next day's batch processing started before the current day's batch processing was complete. With multiple days' files in the processing system and multiple days' jobs in the queues, the simultaneous batches interfered with each other, causing additional recovery problems and further backlogs. A one-day hole had become a self-feeding queue.
What this shows: the failure loop the notices describe for Ulster Bank. Once a batch day cannot finish, the next day does not wait — it lands on top, and the two interfere.
The long tail, to 10 July. Disruptions to the majority of RBS's and NatWest's systems lasted until 26 June 2012. Disruptions to the majority of Ulster Bank's systems continued until 10 July 2012. Disruptions to other systems — including BankTrade, a system for processing and documenting international bond trades, and IFS, an international currency system — affected all the banks and lasted into July 2012.
The damage
The people count. The IT incident affected at least 6.5 million customers in the United Kingdom, 92 percent of whom were retail customers. The PRA puts that in population terms: it was 10 percent of the UK population. Keep a second figure separate, because it measures something different: the banks took corrective action on 7 million accounts, including 1.9 million accounts at Ulster Bank in Northern Ireland and the Republic of Ireland. Customers affected (6.5 million, UK) and accounts corrected (7 million, including Ireland) are not the same number wearing two coats, and this page does not merge them.
What stopped working. The PRA lists the core functions the incident affected or put at risk: updating customer balances; giving customers access to their accounts through online banking; customers getting an accurate picture of their accounts through ATMs; processing payments on time; participating in clearing; applying appropriate fees and charges; transferring and receiving payments; and processing loans. In plain terms, for millions of people the bank could no longer reliably answer the most basic question in banking: how much money do I have?
Businesses and bystanders. Commercial customers lost Bankline, the internet banking service — they were unable to manage payments, verify cheques or make international cash transfers. Other commercial customers could not finalise their audited accounts or meet payroll commitments. And the harm spread past the banks' own customers: people who were not customers were unable to receive money owed to them by the banks' customers, which in turn prevented them from honouring their own financial commitments. A batch failure inside one banking group became missed obligations in businesses that had never heard of a batch scheduler.
The regulatory bill. On 19 November 2014, after a joint FCA and PRA investigation — the first joint enforcement action of its kind, following the new twin-peaks regulatory structure that began on 1 April 2013 — the banks were fined £42 million by the FCA and £14 million by the PRA, £56 million combined. Both figures already include a 30 percent discount for settling at an early stage: without it, the FCA penalty would have been £60 million and the PRA's £20 million. The breach was of Principle 3 of the regulators' Principles for Businesses, over the period 1 August 2010 to 10 July 2012. The PRA also records, fairly, that the breaches were neither deliberate nor reckless. This was not misconduct for gain. It was a systems-and-controls failure — which is precisely why it is a useful case for engineers.
How they recovered
Step one: finish Monday by hand. On the evening of 18 June, when the new scheduler's first full run failed in places, internal IT experts re-ran the failed batches by entering commands manually, and that night's batch completed. Manual heroics worked — once. They are also the first sign a system is being held together by its people rather than by its design.
Step two: re-load the missing jobs by hand. After the rollback scattered the queues, technical support staff focused on manually re-loading jobs into the batch queues. That ground the NatWest backlog down first: by the morning of 20 June, the NatWest batch for 19 June was largely completed. Ulster Bank, further behind and then caught in the two-days-at-once interference, took until 10 July for the majority of its systems.
Step three: correct the accounts. Recovery was not only getting the scheduler running again; it was repairing what the missed days had left wrong. The PRA records corrective action on 7 million accounts, including 1.9 million Ulster Bank accounts in Northern Ireland and the Republic of Ireland. A batch incident ends in account-by-account correction work, long after the queues are moving again.
Step four: submit to independent scrutiny. Following the incident, the Financial Services Authority (the regulator at the time) required the banks to appoint a Skilled Person — an independent expert — to assess the immediate causes, consequences and management of the incident. The PRA and FCA then ran their joint investigation, and the PRA notes that the banks have since put in place an IT resilience programme and undertaken significant software improvements. This page cites that programme only in the notices' own words; the notices do not itemise it, and nothing beyond their wording is claimed here.
The fix: technology and controls that came out of it
The notices are unusually useful as a fix list, because the regulators itemise the defences that were missing. Read them as engineering requirements, not legal language.
Test the way back, not just the way forward. The single most direct finding: the banks failed in adequately testing the consequences of backing out the batch scheduler software upgrade in a representative testing environment. A representative environment means a test system that behaves like the real one closely enough for the test to mean something. Every upgrade plan has a forward path. This case is the standing argument that the rollback path is part of the change — version compatibility between old and new, data written by the new version that the old one must still read, and a rehearsed decision about when rolling back is more dangerous than pushing forward. On 19 June, rolling back felt like the safe option. It was the incident.
Plan and control changes as a discipline. Technology Services, the PRA found, did not take reasonable steps to ensure changes were carried out in a carefully planned and consistent manner: it lacked adequate processes for identifying, analysing and resolving IT incidents, and adequate policies for testing software. At group level it also failed to check that its IT policies were consistent with each other, to ensure changes could be made in a controlled way, to keep accurate and complete records of the changes it was making, and to maintain a complete view of IT risk — particularly in operations. Change records sound like paperwork until an incident starts and nobody can reconstruct exactly what is running where.
Shrink the blast radius. The regulators name the containment options that were not taken: reduce the number of jobs each batch scheduler managed, reduce the interdependencies between the RBS scheduler and the NatWest and Ulster Bank scheduler, and use separate schedulers for NatWest and Ulster Bank. Three brands shared one failure because the architecture let them. Segmentation — separating systems so a fault in one cannot reach the others — is unglamorous work, and it is the difference between one bank having a bad week and three banks having one together.
Make the three lines of defence real. The group's risk model had three layers, and the notices find all three wanting. Technology Services Risk, the first line, had a culture of reacting to events and a team with insufficient experience and skills. Business Services Risk, the second line, had limited IT skills and did not sufficiently challenge the first line's view of IT risk. Group Internal Audit, the third line, had weaknesses in communication with the first two. A defence layer that cannot understand the system it oversees, or will not challenge it, is an org chart, not a control.
Aim at resilience, not only recovery. The deepest finding is about what the group was optimising for. Its IT risk appetite covered business continuity — recovering from an incident — but, the PRA found, it should have included a much greater focus on IT resilience: designing IT systems to withstand or minimise the risks of disruptive events in the first place. Recovery asks how fast you can fix it. Resilience asks why one scheduler, one upgrade and one rollback could freeze a tenth of the country's bank accounts at all. The IT resilience programme the PRA notes the banks later put in place is the organisational form of that shift.
What this means for you
Enterprise network engineers
This is your case, even though the broken box was a mainframe scheduler and not a switch. Your equivalents are the core upgrade, the firewall pair, the controller cluster — any change where the rollback plan is a sentence in a ticket rather than a tested procedure. Three habits fall straight out of the notices. Test the rollback in an environment that represents production, including what the new version has already written or changed. Map interdependencies before the change, not during the incident: RBS had its own scheduler and was hit anyway, because nobody had priced the link between the two. And treat a known, audited risk as work, not wallpaper — the 2010 audit named this failure mode two years early, and the finding alone changed nothing.
Hyperscaler engineers
You run this pattern at fleet scale: staged rollouts, automatic rollbacks, and shared control planes serving many tenants who believe they are independent. RBS is what a shared control plane does to that belief. The lessons that map are blast radius and rollback semantics. An automatic rollback is still a version change, with the same compatibility questions as the upgrade — data formats, queue contents, in-flight jobs written by the newer version. And a backlog that feeds itself, like Ulster Bank's two days of interfering batches, is your queue-and-retry storm in banking dress: when recovery work competes with live work in the same queues, recovery itself becomes the load. Design the drain path before you need it.
Try it yourself
The RBS failure spread because connected systems were assumed to be independent — the RBS scheduler was affected through interdependencies nobody had contained. You can practise seeing your own interdependencies with Packet Path's free Config to Topology Mapper: paste device configurations and it draws the network they actually describe, with mismatches flagged. Then check the details with the free Network Config Verifier, which compares configurations side by side and flags inconsistencies. The habit both tools build is the one missing in June 2012: know what is actually connected to what, and verify it — before the change window, not during the incident.
Sources
Primary source — Prudential Regulation Authority, Final Notice to Royal Bank of Scotland plc, National Westminster Bank plc and Ulster Bank Ltd (19 November 2014), imposing a financial penalty of £14,000,000 (£20,000,000 before the 30 percent Stage 1 settlement discount) for breaches of Principle 3 between 1 August 2010 and 10 July 2012. Facts taken from this notice include: the batch scheduler mechanism and the jobs it queues; the 17 June 2012 upgrade from Version 1 to Version 2A and its reason; the 18 June first full run, its high mainframe use, slowdowns and batch terminal failures, and the manual re-run that completed that night; at least 6.5 million UK customers affected (92 percent retail, 10 percent of the UK population); disruption to the majority of RBS and NatWest systems until 26 June 2012 and of Ulster Bank systems until 10 July 2012, with BankTrade and IFS disruption into July; the effects on core functions, Bankline customers, audited accounts and payroll, and non-customers; corrective action on 7 million accounts including 1.9 million Ulster Bank accounts in Northern Ireland and the Republic of Ireland; the finding that the breaches were neither deliberate nor reckless; the Skilled Person requirement; and the note that the banks have since put in place an IT resilience programme and significant software improvements. Full text (Bank of England): https://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/enforcement-notice/en201114.pdf
Primary source — Financial Conduct Authority, Final Notice to Royal Bank of Scotland plc, National Westminster Bank plc and Ulster Bank Ltd (19 November 2014), printed as Appendix 2 of the PRA notice above, which the PRA expressly adopts and relies upon for its facts and matters. The FCA imposed a financial penalty of £42,000,000 (£60,000,000 before the 30 percent Stage 1 settlement discount). Facts taken from the facts and matters common to both notices include: the 19 June 2012 back-out of the upgrade, the team's unawareness that the upgraded and previous versions were incompatible, and the resulting jobs failing to appear in the batch queues; the manual re-loading of jobs, the NatWest position by the morning of 20 June, Ulster Bank being more than a day behind by 21 June, and the interference between simultaneous days of files and jobs; the statement that the actual cause was the software compatibility issue between the upgraded and previous versions, triggered by the back-out; the 1 August 2010 internal audit that identified the risk of a batch scheduler failure; and the failings in change planning, incident processes, software testing (including not adequately testing the back-out in a representative environment), the Three Lines of Defence, and an IT risk appetite focused on business continuity rather than IT resilience.
No other sources are used for facts on this page. The notices never name the scheduler product, so this page does not name it either, and press figures that differ from the notices (for example, larger account counts in contemporary coverage) are not used: every figure above is the regulators' own.
- The rollback, not the upgrade alone, was the actual cause in the regulators' findings: the upgraded batch scheduler and the previous version were incompatible, and backing out on 19 June made a significant number of jobs fail to appear in the batch queues.
- The rollback had never been properly tested: the notices fault the banks for not adequately testing the consequences of backing out the upgrade in a representative environment.
- The risk was already on paper: the breach period starts 1 August 2010, the date of an internal audit on mainframe batch processes that had identified the risk of a batch scheduler failure — about two years before it happened.
- At least 6.5 million UK customers were affected (92% retail, 10% of the UK population); the FCA fined the banks £42 million and the PRA £14 million, £56 million combined after early-settlement discounts.