Google Cloud’s June outage moved bank cloud diligence from uptime to change safety

The week

On the afternoon of June 12, 2025, a lot of the internet stopped working properly at the same time. Apps would not load, logins failed and services that seemed to have nothing to do with each other went down together. The cause, it turned out, was a single routine update inside Google Cloud.

Google Cloud is one of the handful of giant “hyperscale” cloud providers. Instead of buying and running their own servers, companies rent computing power, storage and ready-made software services from these providers and pay as they go.

That day, Google Cloud suffered a multi-hour outage that disrupted many of its services and a wide range of customer applications. Google’s incident report traced it to a component called Service Control, which checks API requests against policies and quotas. An API, or application programming interface, is the standard way one piece of software asks another to do something. Service Control decides whether each request is allowed and whether the customer has hit their limit.

Code for a new quota policy check had been rolled out in late May, but the path that would run it had never been exercised. On June 12, a policy change containing blank fields was replicated globally within seconds. The new code hit a null pointer error, which occurs when software reaches for data that is not there and has no plan for what to do next. It crashed in every region.

Google’s report gave the timeline in unusual detail. The new check had been added to Service Control on May 29 and rolled out region by region. On June 12, at about 10:45 a.m. Pacific time, the policy change with blank fields was written to the regional database tables Service Control reads. Global replication carried it everywhere within seconds, and failures began within minutes.

Engineers began triage within two minutes and identified the cause within about ten. The fix was an emergency off switch, which Google’s engineers called a red button, built into the new code. It was ready in about 25 minutes and rolled out within about 40. Smaller regions recovered first.

The largest region, us-central1 in Iowa, took far longer. As Service Control restarted there, its tasks overwhelmed the Spanner database they depend on. Google said the service lacked randomized exponential backoff, a standard technique in which retrying systems wait progressively longer, at slightly random intervals, so they do not all retry at once. Recovery in that region took up to about two hours and forty minutes.

Google’s status dashboard ran from the first failures to a final all-clear in the early evening, Pacific time, about seven and a half hours in all. Most of that tail involved residual effects in individual products. Dozens of Google Cloud and Google Workspace products were listed as affected.

Customers also lost their view of the problem. Google said its first public incident report appeared about an hour after the crashes began, because Cloud Service Health, the system that posts those reports, was itself affected. Some customers whose monitoring ran on Google Cloud lost that signal too.

Google said the feature had not been protected by a feature flag, an on-off switch that lets engineers ship new code in a dormant state and turn it on gradually, or off instantly if something goes wrong.

The trouble did not stop at Google. Cloudflare, which said an outage at a third-party cloud provider had disrupted a storage service its products depend on, also suffered failures, spreading the effect to many more sites and applications. Cloudflare said its Workers KV service, a key-value store many of its products rely on, lost access to storage that partly ran on the third-party provider. About 90 percent of Workers KV requests failed for roughly two and a half hours. Cloudflare Access, its login gateway for corporate applications, failed closed, so identity-based logins failed. New device registrations for its WARP client also failed.

Cloudflare said it remained responsible for the dependencies it had chosen, and committed to reduce its reliance on storage it did not own. One company’s bad afternoon became many companies’ bad afternoon.

Cloud outages are usually discussed as a concentration problem: too many institutions depending on too few providers. That framing is accurate and well covered. The June 2025 Google Cloud outage supports a narrower and more practical argument for banks, credit unions and broker-dealers.

The incident began with no hardware fault, attack or disaster. It began with a routine change. Most third-party risk programs ask providers for uptime figures and audit reports, and very few ask how changes are released. We break that question into four parts: how new code is activated, how configuration reaches production, how far a failure can travel, and what evidence the bank gets afterward.

The argument rests on a distinction most executives never need to think about until a day like June 12.

Code, configuration and the global control plane

A large cloud provider is never finished. Thousands of engineers make changes every day: new features, security fixes, performance improvements. Those changes come in two broad kinds.

The first is code, the instructions that make software behave a certain way. The second is configuration, the settings and rules the code reads to decide what to do in a given case: which customer gets which limit, which requests are allowed, where traffic should go.

Engineers have built careful habits around releasing code. The standard practice is a staged rollout: new code goes first to a small slice of servers, then one region, then another, with monitoring at each step. A region is a cluster of data centers in one geographic area; providers have dozens around the world. If something breaks in the first region, the rollout stops before it reaches the rest.

A second habit is the feature flag. New code is shipped switched off, and turned on separately and gradually. That separates two moments: when the code arrives and when it starts doing anything.

Finally, there is a distinction between the data plane and the control plane. The data plane is the part that does the actual work, such as storing files or running applications. The control plane is the management layer that decides who can do what: identity checks, permissions, quotas and policy. Many data plane services are regional. Some control plane services are global, meaning one system serves every region. Service Control, at the heart of the June 12 outage, is one of those global components.

Staged rollout, feature flags, the line between configuration and code, and the split between regional and global services all mattered on June 12. The feature flag was the safeguard that was missing.

Fail open or fail closed

One more design choice explains why a policy checker could stop so much. When a check like Service Control cannot run, the system has to decide what to do with the request. It can fail closed, refusing everything until the check works again. Or it can fail open, letting requests through unchecked.

Neither answer is free. Failing closed protects security and quota rules at the cost of availability. Failing open keeps services running at the cost of briefly unenforced rules. On June 12 the checker crashed in a way that blocked API requests, and one of Google’s remediation commitments was to modularize Service Control so that a failing check would fail open.

Cloudflare Access made the opposite choice, deliberately. As a security gateway, it fails closed, so when its storage failed, logins failed. For a product whose job is to keep unauthorized users out, that is defensible. It still meant that firms using it to reach internal applications were locked out of them.

Banks make the same choice in their own systems, often without writing it down. Card networks have long offered stand-in processing, which approves or declines transactions on the issuer’s behalf within preset limits when the issuer cannot respond. A bank’s fraud scoring call, sanctions screening step or login risk engine has a timeout too. Someone should own the decision about what happens when that dependency is unavailable, and the business should agree to it in advance.

How new code is activated

Large providers release code continuously. The usual safeguard is staged rollout, region by region, with new features held behind flags until they are deliberately switched on.

In Google’s account, the new code had already been deployed everywhere. Its path simply had not run, because the policy that would trigger it did not yet exist. Staged deployment therefore offered no protection. The risk moved to the moment of activation, which happened globally and at once.

A building analogy fits. Suppose a building manager installs new smoke detectors on every floor, one floor at a time, checking each as they go. Everything looks fine, because the detectors stay silent. Weeks later, someone sends a test signal to the whole building at once, and it turns out the new detectors respond to it by cutting the power. The careful floor-by-floor installation never tested that response, because nobody sent the signal until it went everywhere.

A bank cannot inspect a hyperscaler’s pipeline, but it can ask specific questions in its due diligence questionnaire and annual review, covering:

• Whether new features in critical control-plane services are protected by flags.
• Whether activation is staged as carefully as deployment.
• Whether dormant code paths are tested with realistic inputs, including malformed ones.

Providers may answer at a general level. The value lies partly in the answers and partly in the record. If a later incident contradicts a written answer, the bank has a documented basis for escalation, and its board has evidence that management asked.

What audit reports do and do not show

Most banks rely heavily on independent audit reports for cloud diligence. A SOC 2 Type II report, the most common, tests whether a provider’s controls operated over a period. Its criteria include change management: changes are authorized, tested and approved before release.

That is useful, and it answers a different question from the one June 12 raised. A provider can pass a change management test while a dormant code path waits, untested against malformed input, to be activated everywhere at once. The audit checks that the process was followed. It does not test how far a single change can reach if the process misses something.

The gap is not a criticism of auditors. Those reports were designed to show that controls exist and operate, not to model failure propagation. Diligence teams need to supplement them with questions about design: blast radius, activation and rollback.

Reading a provider’s answer

The answers worth having are specific. A strong response describes how new behavior in critical services is switched on in stages, how long each stage bakes before the next, and what signals halt progress. It explains how emergency off switches are themselves tested, since Google’s switch existed but the code lacked proper error handling. It says whether dormant paths are exercised with malformed and empty inputs before activation.

Staged activation has a familiar shape. New behavior is switched on for a small share of traffic in one region first, often called a canary after the birds miners once carried underground. Error rates are compared with the rest of the fleet. Only when the canary looks healthy for a set period does the switch-on widen. A provider that can describe its canary stages and halt criteria for control-plane services is answering the question that mattered on June 12.

A weak response restates the uptime commitment or points to the audit report. That tells the bank the question has not reached the engineers.

How much a bank can ask depends on whom it is asking

Bargaining power varies. A hyperscaler serves millions of customers and will usually answer through standard documentation, audit reports and account teams. A large bank may get technical briefings. A credit union will usually get the same documents as everyone else.

A specialist provider, such as a core banking platform, a digital banking vendor or a fraud scoring service, is different. These firms often serve a few hundred financial institutions and can be pressed for release calendars, change notification windows and the right to defer certain upgrades. For many smaller institutions, those providers are the more important place to ask about change safety. They also run on hyperscale clouds themselves, so their answers should cover how they absorb a failure in their own provider.

How configuration reaches production

The trigger on June 12 was a policy record. Configuration and policy data often move faster, and through fewer gates, than software, because teams treat them as data. Yet a malformed entry can do as much damage as a bad build. Here it replicated worldwide within seconds.

The reason configuration moves fast is partly good intentions. When a customer needs a limit raised or a security rule updated, nobody wants to wait days for a full software release. So configuration often travels through a quicker, lighter process. That is convenient until a bad entry rides the same express lane.

Banks know this pattern from their own estates. Many internal outages start with a firewall rule, an expired certificate or a routing change. (A firewall rule decides which network traffic is allowed in or out; a certificate is a digital credential that proves a system’s identity and expires on a set date; a routing change alters the path network traffic takes.) Their change advisory boards, the committees that review and approve changes before they go live, often govern application releases more tightly than these updates.

Why configuration escapes the gates

The mechanism is ordinary. Policy and quota settings for a global service are usually stored in a database and replicated to every region, so each regional copy of the service can read them locally and quickly. Writing a new record is a data operation. It does not trigger a software build, a code review or a staged deployment.

The replication system does exactly what it was built to do. It moves a valid-looking record everywhere, fast, because that is the requirement for quota and security rules. Unless something checks the record’s shape and content before it is written, the only test is what happens when every region reads it.

Google’s commitments addressed this directly. It said it would audit systems that consume globally replicated data and propagate such changes incrementally, with time to validate before they spread.

The same pattern runs through a bank’s own software-as-a-service estate. A conditional access policy in an identity platform, for example, can lock every employee out of email and core applications if it is written wrongly. Identity platforms often offer a report-only mode in which a new policy is evaluated and logged but not enforced. Using it before enforcement is the bank-side equivalent of staged activation, and change rules can require it for policies that affect all users.

The same scrutiny belongs in provider oversight. Third-party risk teams can ask about four things:

• Whether configuration and policy changes are validated against a schema before release. A schema is a template that defines what a valid entry looks like, so blank or malformed fields are rejected before they go anywhere.
• Whether those changes are staged across regions.
• Whether they can be reversed within minutes.
• Whether global replication can be paused, and who holds that authority.

A provider with a crisp answer is telling the bank something about its engineering culture. So is a provider without one.

How far a failure can travel

The failure spread across regions because Service Control is a global component. Recovery in one large region was then slowed as tasks restarted together and overloaded a shared database. This is sometimes called a “thundering herd”: when everything comes back at the same moment, the rush itself causes a new jam.

The knock-on effect at Cloudflare showed a further layer: a provider’s outage becomes the outage of every provider built on it. Risk managers call these fourth parties: your vendor’s vendors. A bank may have no contract with a company and still depend on it completely.

Banks cannot redesign a provider’s architecture, but they can design their own with it in mind. The first step is a dependency map that separates regional services, such as compute and storage, from global control-plane services, such as identity, API management and policy enforcement. A bank running in two regions of one cloud may still share a single global dependency, so its multi-region design protects against less than its architecture diagram suggests.

The second step is a fourth-party view. If a bank’s payment processor, fraud scoring vendor and contact center platform all run on the same provider, it holds a concentration that sits in no single contract. Recording those links lets the business continuity team plan for a scenario in which several vendors fail together.

Two regions, one hidden dependency

Take a regional bank whose online banking runs in two regions of one cloud provider, and whose architecture diagram shows them as fully independent. A dependency mapping exercise finds that both regions rely on the same global identity and API management services. It also finds that the bank’s card fraud scoring vendor and its contact center software, bought separately and years apart, run on that same provider. On a day like June 12, the bank could plausibly lose online banking, fraud checks on card payments and its customer phone lines at the same time. No individual vendor review would have shown that. The map takes a few weeks to build and changes the bank’s continuity planning immediately.

Retry storms start at home too

The slow recovery in us-central1 had a bank-side counterpart. When a service fails, every client that depends on it tends to retry. Mobile apps refresh, batch jobs restart and middleware reconnects. If all of them retry immediately and repeatedly, the recovering service meets a wall of traffic far larger than normal.

The remedy is the one Google said Service Control lacked: exponential backoff with jitter, meaning each retry waits longer than the last, with a random element so clients spread out. Circuit breakers help too. A circuit breaker stops calling a failing dependency for a short period, then tests it with a small amount of traffic before resuming.

Banks can check their own applications for both. A mobile app that hammers a recovering login service, or an integration layer that resubmits every queued payment at once, can turn a provider’s recovery into a second outage of the bank’s own making.

Different institutions, different rulebooks

The analysis is the same across financial services, but the expectations differ by charter and regulator.

• Banks. The June 2023 interagency guidance from the Federal Reserve, FDIC and OCC applies to banking organizations they supervise. It scales expectations to the criticality of the activity, so a critical cloud service draws the most scrutiny.
• Credit unions. NCUA has long said it lacks the statutory authority banking agencies have to examine third-party service providers directly, and has asked Congress for it. Credit unions therefore carry more of the diligence themselves, often through shared service organizations and core processors whose own cloud dependencies they may not see.
• Broker-dealers. FINRA Rule 4370 requires business continuity plans, and FINRA’s 2021 regulatory notice on vendor management reminded firms that outsourcing a function does not outsource supervisory responsibility. The SEC’s Regulation SCI, with its specific systems change and testing requirements, applies to exchanges, clearing agencies and certain other market infrastructure, not to most broker-dealers.
• Firms with UK or EU operations. UK operational resilience rules required firms to be able to stay within impact tolerances for important business services by March 31, 2025. In the EU, DORA requires a register of all ICT third-party arrangements, which is the regulatory version of the fourth-party map described above.

Counsel and compliance should confirm which apply. The engineering questions are the same whichever rulebook asks them.

There is a cost trade-off. True independence from a provider’s global control plane usually means a second provider for the most critical services, with duplicated skills and tooling. Few banks will do that broadly. Many can justify it for a short list of services where a few hours of downtime would breach a regulatory or customer commitment.

What evidence the bank gets afterward

Google published a detailed incident report within days, naming the component, the missing flag and the remediation. That transparency is valuable, and banks should not leave it to goodwill. Contracts for critical services can require a written root cause analysis, an explanation of the underlying reason for the failure rather than just its symptoms, within a set period. They can also require notice of remediation commitments, and a later confirmation that those commitments were met.

Supervisors support this line of inquiry. The 2023 U.S. interagency guidance on third-party risk management expects banks to understand how critical providers manage operational risk. In the European Union, the Digital Operational Resilience Act, known as DORA, has applied since January 2025. It requires financial entities to manage ICT (information and communication technology) third-party risk and sets up direct oversight of critical ICT providers.

U.S. banks also have a notification rule that touches provider outages. Under the computer-security incident notification rule, in force since 2022, a bank must notify its primary federal regulator within 36 hours of determining that a qualifying incident has occurred. Bank service providers covered by the Bank Service Company Act have a duty too. They must notify affected bank customers as soon as possible when an incident has disrupted, or is likely to disrupt, covered services for four or more hours.

Whether a given cloud provider falls under that rule is a question for counsel. The operational point is simpler. A bank should know, before an outage, which of its providers owe it notice, through what channel, and who at the bank receives it. During the first hour of June 12, Google’s own status page was impaired. A notice channel that depends on the affected provider may not work when it is needed.

In the EU, DORA adds a reporting duty of its own. Financial entities must classify ICT-related incidents and report major ones to their supervisors on set timelines, including incidents that begin at a provider. A bank can meet those deadlines only if it learns quickly what happened at its provider and which of its services were affected.

Reading a post-incident report

A report is evidence only if someone extracts the parts that matter. Five items are worth recording for each critical provider incident:

1. The trigger. Code, configuration, capacity, hardware or attack.
2. Detection and mitigation times. Google’s report said triage began within two minutes and the off switch was fully rolled out within about 40. Those figures are a benchmark for the next incident.
3. Why the blast radius was what it was. Here, a global component and global replication.
4. What slowed recovery. Here, a retry storm against a shared database.
5. Commitments, with dates where given. Google listed modularization, feature flags for critical changes, incremental propagation of replicated data, better error handling and backoff, and communications that survive the outage.

Kept in one place over several years, these records show whether a provider’s incidents are getting smaller and rarer, and whether its commitments are delivered.

Post-incident reports then become diligence evidence in their own right. A risk team that reads them, records the provider’s commitments and checks delivery at the next review is assessing change safety on evidence.

What the bank still controls on the day itself

Diligence and contracts shape the odds. They do not stop a provider having a bad afternoon. When it happens, the bank’s own preparation decides how much customers feel it. Three things are within its control.

Knowing quickly that it is not you. In the first minutes of a broad outage, internal teams often waste time hunting for a fault in their own systems. A bank that monitors its critical providers’ public status pages, and that has mapped which of its services depend on which provider components, can recognize the pattern faster. “Online banking, fraud scoring and the contact center all failed at once, and all depend on the same provider” is a diagnosis the dependency map makes possible in minutes rather than hours.

Degrading gracefully. Not every service has to fail completely. Engineers use the phrase “graceful degradation” for designs that keep the most important functions working, perhaps more slowly or with fewer features, when a dependency is unavailable. A mobile app that can still show balances from a recent copy, even if it cannot accept new payments, is far less alarming to customers than one that shows only an error. These choices have to be made in advance, during design, and the business should help decide which functions matter most.

Talking to customers and staff. Customers want to know three things: is my money safe, what can I still do, and when will it be fixed. A short, pre-written message for a provider outage, ready to post on channels that do not depend on the affected provider, answers the first two even when nobody yet knows the third. Branch and call center staff need the same message, so they are not guessing.

Deciding whether to fail over

The hardest call on a day like June 12 is whether to move a service to a backup or wait for the provider to recover. Failing over has costs of its own. Data written in the last minutes before the failure may not have reached the backup. Switching back later is another change, with its own risk. A backup that has never carried real load may not carry it now.

The June 12 timeline shows why the decision rule should be written down in advance. Most regions recovered within roughly an hour, while one large region took hours longer. A bank that failed over at minute twenty might have been right for a service in us-central1 and wrong for one elsewhere. A rule tied to the service, such as fail over if online payments have been unavailable for a set period and the provider has given no recovery estimate, removes guesswork. It also names who decides, which matters more than the threshold itself.

For many services the honest plan is to wait, degrade gracefully and communicate. That is a legitimate choice when it is made deliberately and recorded, with the business owner’s agreement.

Monitoring that survives the outage

June 12 showed a specific trap. Some customers’ monitoring ran on Google Cloud, so it failed with everything else, and Google’s status reporting was itself impaired for the first hour. A bank whose only signals come from inside the affected provider is waiting to be told.

A modest fix is external synthetic monitoring: scripted checks that log in, load a balance or submit a test transaction from outside the bank’s main provider, every few minutes. When those checks fail across several services at once, and the dependency map shows a common provider, the incident team has its first hypothesis in minutes. The monitoring vendor’s own hosting matters too. It belongs on the dependency map like everything else.

Each of these depends on the earlier work. The dependency map tells the bank which services will fail together. The fourth-party view tells it which vendors to call. The contract tells it what explanation it can expect afterward. On the day, preparation shows up as calm.

Rehearsal pays. A tabletop exercise, a structured discussion in which leaders walk through a realistic scenario, built around “our main cloud provider has a global control-plane failure lasting four hours” will surface gaps faster than any questionnaire. Who decides whether to switch services to a backup? Who approves the customer message? Which regulators need to hear from the bank, and how soon? Answering those questions in a conference room is far cheaper than answering them live.

Who takes the blame when a provider breaks

Customers notice immediately. When a bank’s app or card authorizations fail, customers do not care whose server broke. They blame the bank, and some will tell everyone they know.

Regulators expect ownership. Outsourcing a service does not outsource responsibility for it. Supervisors in the United States and the European Union expect banks to understand and manage the risks in their providers, including how those providers change their systems.

Uptime numbers hide the important part. A provider’s promised availability, often expressed as a percentage with several nines, is an average over a long period. It says little about how a single bad change could take everything down at once, which is the scenario that actually hurts.

The arithmetic makes the point. A service promised at 99.9 percent availability may be down for about 8.8 hours a year. At 99.99 percent the allowance is about 53 minutes. A single seven-hour event, concentrated on one weekday afternoon, can consume a year’s budget at the first level and many years’ at the second.

The usual contractual remedy for a missed availability target is a service credit, a discount on future fees. Credits are sized to the provider’s fees, not to the customer’s losses. For a bank whose card authorizations or online banking failed for an afternoon, the credit will rarely cover the cost of the day, let alone the complaints and regulatory questions that follow.

Boards need defensible answers. After an outage, directors will ask what management knew and what it asked. A written record of change-safety questions and answers is a far better answer than “the provider assured us it was reliable.”

Change safety as a standing topic

The stronger third-party risk programs treat change safety as a standing topic for every critical cloud and software-as-a-service provider. (Software as a service, or SaaS, is software you rent and use over the internet rather than install yourself.) They ask about flags, staged activation and configuration validation, and map dependence on global control-plane services. They record which vendors share a provider and require root cause reports by contract. At each annual review, they check whether promised fixes were delivered.

The case against asking

A skeptical CIO could object on three grounds. Hyperscalers will not change their engineering because one bank asks. Their answers will be standard text. And bank diligence teams do not have the expertise to judge a release pipeline, so the exercise produces paperwork rather than insight.

Each point has force. A single bank does not move a hyperscaler, and the first round of answers may well be boilerplate. The questions are still worth asking, for three reasons. They tell the bank where its own architecture must compensate for what the provider will not promise. They create a record that matters after the next incident. And collectively they reinforce what supervisors began asking of the largest providers directly.

The expertise gap is real and easier to close than it sounds. The questions here need an engineer in the room, usually from the bank’s cloud platform or site reliability team, rather than a new specialist function. The cost is a few hours per critical provider each year.

Where the argument weakens is in software bought by business units without central review. Change-safety questions only help if the provider is on the critical list in the first place. The inventory comes before the interrogation.

Measures that keep the program honest

A standing topic needs a few measures, reported to the risk committee once or twice a year:

• The share of critical providers with written answers on activation, configuration and global dependencies, and the age of those answers.
• The share of critical contracts that include a right to a written root cause analysis within a set period.
• The date the dependency map was last verified against the live architecture, rather than against diagrams.
• The number of provider remediation commitments recorded, and how many were confirmed as delivered.
• The provider-outage scenarios exercised in the past year, and the gaps each exercise found.

The measures require no new tooling. Each one turns a vague assurance into a fact the committee can check.

The usual blind spots:

• Counting regions, not dependencies. Assuming two regions means two independent systems.
• Reviewing vendors one at a time. Missing that several critical vendors share one underlying provider.
• Collecting reports without reading them. Filing incident reports without noting commitments or checking delivery later.
• Treating configuration as harmless. Governing code changes tightly at home while settings changes go through lightly, and not asking providers about either.
• Relying on goodwill for transparency. Having no contractual right to a root cause analysis.

Gaps a cloud review should close

• For each critical cloud or SaaS provider, have we asked in writing how new features are switched on, how configuration and policy changes are validated and staged, and who can pause a global change?
• Which of our critical services depend on a global control-plane component, even if they run in several regions?
• Which of our critical vendors run on the same cloud provider, and have we tested a day when they all fail together?
• Do our contracts give us a right to a written root cause analysis within a set period, and who reads each post-incident report, records the commitments and checks them at the next review?
• For which short list of services would a few hours of downtime breach a regulatory or customer commitment, and is a second provider justified there?

With hindsight

Google’s remediation commitments included modularizing Service Control, meaning breaking it into smaller, more independent pieces so one fault cannot take down the whole. It also committed to requiring feature flags for critical changes and changing how global policy data is replicated.

Later incidents followed the same pattern. In October 2025, a long AWS outage in its US-EAST-1 region began with a fault in automated DNS management. DNS, the domain name system, is the internet’s address book, translating names people type into the numeric addresses computers use. Later that month, Microsoft attributed an Azure Front Door outage to configuration changes that exposed a latent bug its pre-production validation had missed. In November 2025, a malformed Bot Management feature file pushed across Cloudflare’s network caused a global disruption. Each started with a change or configuration path.

The details reward a closer look. AWS said its disruption ran from late on October 19 to the afternoon of October 20, Pacific time. A timing fault between two components of DynamoDB’s automated DNS management left the regional endpoint with no addresses, and the automation could not repair it. AWS disabled that automation worldwide until it could fix the fault and add safeguards.

Microsoft said its Azure Front Door incident on October 29 lasted about eight and a half hours. Customer configuration changes made across two versions of its control plane produced incompatible metadata, which crashed edge servers. Because configuration deployed to all edge sites, the bad metadata reached most of them before the crashes appeared. Microsoft’s repair list included more rollout stages, longer bake times and splitting its data plane into smaller cells.

Cloudflare said its November 18 outage began with a database permissions change that roughly doubled the size of an automatically generated configuration file. The file exceeded a limit in the software that read it, and errors spread across the network. Its commitments included validating internally generated configuration with the same rigor as user input and adding more global kill switches.

The remediation lists read like one another, and like Google’s. Stage changes, validate configuration, limit blast radius, keep an off switch that works. That convergence is useful to banks. It gives diligence teams a shared vocabulary that providers have used themselves in public.

Taken together, these incidents at four of the most sophisticated technology companies in the world make the case plainly. The biggest risk at a major provider is less often a broken machine or an attacker and more often an ordinary change that travels too far, too fast. Even companies with deep engineering talent and careful processes get this wrong from time to time.

Supervisors moved in the same direction. In November 2025, the European Supervisory Authorities designated their first list of critical ICT third-party providers under DORA, including Google Cloud, AWS and Microsoft. Those providers now sit under direct European oversight, which gives banks a firmer basis for asking detailed questions and expecting real answers.

The designation came on November 18, 2025, the same day as the Cloudflare outage, by coincidence. Direct oversight does not move responsibility, though. Financial entities using those providers remain responsible for their own ICT risk under DORA, and U.S. banks remain responsible for their providers under the 2023 interagency guidance.

Some things did not change. Running critical services on two providers remained expensive and complex, and the case for it stayed narrow: a short list of services where hours of downtime would breach a commitment. The more realistic gains came from mapping dependencies, fixing retry behavior and preparing monitoring and communications that survive a provider’s bad day.

By this point in 2026, asking how a provider stages code activation and configuration has a firm footing in both incident history and supervisory practice. The practical shift for banks is modest but meaningful. Keep asking about uptime, but spend more of the diligence conversation on how the provider changes things, and keep a written record of what it says.