One region failed, many financial apps stalled: concentration risk sat in the dependencies

The week

On Monday, October 20, 2025, many people woke up to apps that would not load. Logins hung, payments spun and balances refused to appear. The cause was a major disruption at Amazon Web Services, the world’s best-known cloud provider, in its US-EAST-1 region in northern Virginia.

A cloud provider rents out computing power, storage and ready-made software services over the internet, so that companies do not have to run their own data centers. A “region” is a cluster of the provider’s data centers in one geographic area. US-EAST-1 is one of the busiest, and a great many services, including some of AWS’s own, depend on it.

In its post-event summary, AWS traced the trigger to a latent defect, a hidden bug that had been waiting for the right conditions, in the automated DNS management for DynamoDB. DynamoDB is an AWS database service that countless applications use to store and look up information. DNS, the Domain Name System, is the internet’s address book: it turns a name such as a service’s web address into the numeric address computers actually use to find each other. The defect left DynamoDB’s regional endpoint, the address applications use to reach it, unresolvable. The database was there, but the address book entry pointing to it had gone blank, so nothing could find it.

Because other AWS services rely on DynamoDB behind the scenes, failures spread. They included EC2 instance launches, meaning customers could not start new virtual servers, and Network Load Balancer health checks, the routine checks that decide which servers are healthy enough to receive traffic. When those checks misfire, healthy servers can be treated as broken.

AWS put the customer impact at roughly 14 hours, from late on October 19 to mid-afternoon Pacific time on October 20, in several overlapping phases. Because so many services depend on that region, the disruption reached a wide range of consumer and business applications, including financial ones. For a bank or payments firm, a day like that means customers unable to reach their money, staff unable to see what is happening and deadlines that do not move just because a data center in Virginia had a bad night.

Suppose you run operations at a mid-sized bank. Your technology team spent years following best practice. Your mobile banking app runs across several data centers in the same cloud region, so that losing one building would not matter. You even have a written plan to move to another region if the whole of US-EAST-1 fails. Then, on October 20, your app stops working anyway. Your core systems are healthy. What failed was the login service you buy from a vendor, the company that sends your push notifications and the monitoring dashboard your engineers use to see what is going on. All of them, it turns out, ran in the region that went down.

The easy conclusion from October 20 was that financial firms rely too heavily on one cloud provider. That is true and not very useful. Many affected firms were not running everything in one place by accident. Plenty had multi-availability-zone designs, and some had multi-region plans. What caught them out was narrower. Three distinct exposures explain most of the impact: dependencies firms had not chosen, a recovery tail that outlasted the fix, and governance that measured the provider instead of the firm.

Regions, zones and the services stacked on top

A cloud provider such as AWS divides the world into regions, each a geographic cluster of data centers. Within each region are availability zones: separate buildings, or groups of buildings, with their own power and networking, close enough to work together but far enough apart that a fire or flood in one should not take out another. A “multi-availability-zone” design spreads an application across several of those buildings. A “multi-region” design goes further and can run, or be moved, in a different part of the country.

On top of the raw computing power, providers sell managed services: databases, messaging systems, identity tools and much more. They are convenient because the provider runs them. They also create hidden links, because the provider’s own services often depend on one another. DynamoDB is one such building block. When its address could not be found, services that relied on it without advertising the fact also stumbled.

Two more terms matter. A “control plane” is the management layer of a cloud service, the part that creates, changes and monitors resources, as opposed to the “data plane” that does the everyday work. An airport makes the distinction clear: the data plane is the planes flying; the control plane is air traffic control. Planes already in the air can keep flying for a while if the control tower has trouble, but nothing new takes off. Some cloud features keep their control planes in a single region, which means a firm can be exposed to one region even if its own application lives elsewhere.

Finally, “software as a service,” or SaaS, refers to finished applications a firm rents rather than builds: an identity provider that handles logins, a messaging platform, a fraud-scoring service, a payment gateway. Each SaaS vendor runs on infrastructure of its own choosing, often the same big clouds and sometimes the same region.

Dependencies nobody chose

A firm can deploy its own application across regions and still depend on one region indirectly. Some cloud features keep their control planes in a single region. Software-as-a-service tools the firm uses, such as identity providers, messaging platforms, fraud scoring or payment gateways, may run in the same region. So may internal tooling: build pipelines (the systems that package and release new software), secrets management (the vault that stores passwords and keys), feature flags (switches that turn functions on and off) and monitoring.

The result is that failover for the core application can succeed while the services around it fail. If the login service, the push notification provider or the observability stack, the tools engineers use to see inside their systems, is down, a healthy core banking API is still unusable to customers and invisible to operators. An API, or application programming interface, is the doorway through which one piece of software asks another for something; a mobile app uses the bank’s API to fetch a balance. A healthy doorway is no use if nobody can log in to walk through it.

The practical response is a dependency map that goes one level beyond the firm’s own architecture. For each important customer journey, such as login, card authorization, bill pay or trade entry, list every service involved, internal and external, and record where each runs. Then ask vendors directly which providers and regions they depend on. Many will not know offhand. That answer is itself useful, and it belongs in the third-party risk file.

The map should also cover the tools used to respond. If the incident chat platform, the status page or the paging service shares the failed region, the firm loses its ability to coordinate at the worst moment. Keeping one out-of-band communication path, a channel on an independent provider that does not depend on the systems that are broken, is cheap.

Recovery is a queue, not a switch

The underlying DNS problem was fixed hours before many services returned to normal. In between, throttled requests, retries and stalled work had to drain. Applications that retried aggressively added load to a recovering system. Batch jobs, the large scheduled runs that process work in bulk, had stalled and restarted at once, competing for the same capacity. Clearing the fault opened the road; it did not clear the traffic, and every client that rejoined at the same moment made the jam worse before it got better.

Financial services feels this more than most sectors because so many processes are time-bound. Payment cut-offs, settlement windows, statement runs and overnight reconciliation all have deadlines. A system that is technically up at 3 p.m. but cannot clear its queue until evening may still miss a cut-off. Resilience tests that measure only time to restore service understate the business impact. Firms should also measure time to clear the backlog, and whether downstream deadlines were met.

Design choices shorten the tail. Each has a technical name and a simple idea behind it:

• Retries with exponential backoff and jitter. When a request fails, wait a little before trying again, wait longer each time, and add a small random delay so thousands of clients do not all retry at the same instant. This prevents clients from hammering a recovering service.
• Circuit breakers. Software stops calling a dependency that keeps failing, so a failing dependency cannot exhaust resources elsewhere.
• Idempotent transaction design. Build transactions so that sending the same request twice has the same effect as sending it once. That makes it safe to replay requests without paying someone twice.
• Load shedding. Deliberately drop non-essential calls, such as marketing messages or analytics, to keep critical traffic such as payments moving.
• A documented restart order. Agree with operations and finance in advance which batch processes restart first, which avoids an improvised sequence on the night.

Governance that measures the firm, not the provider

Regulators already ask about concentration risk, the danger of many firms, or many functions inside one firm, depending on the same supplier. The U.S. banking agencies’ 2023 interagency guidance on third-party relationships expects banks to plan for moving an activity if a provider fails and to manage the risks of a provider’s own subcontractors. In the European Union, the Digital Operational Resilience Act, known as DORA, has applied since January 2025 and provides for direct oversight of critical ICT providers, meaning the technology companies whose failure could disrupt the financial system. Board reports, however, often describe cloud resilience as a property of the provider: uptime commitments, certifications and regions used.

October 20 points to a better test. Can the firm deliver its important business services within defined impact tolerances when one region of its main provider is impaired? An impact tolerance is the most disruption the firm has decided it can accept for a given service, for example how long customers can go without making payments. That test is about the firm’s design. It leads to concrete exercises, such as failing over a customer journey together with its dependencies. It also forces explicit decisions about which services must run active-active across regions, meaning live in two places at once, and which can accept a degraded mode.

Degraded modes deserve more attention than they get. A degraded mode is a planned, reduced level of service that keeps the essentials working when something breaks. Showing read-only balances from a cached store, queuing payments for later release with clear customer messaging, or approving low-value card transactions under stand-in rules are all choices that keep customers served. Each needs a business owner, a risk limit and a rehearsal. Stand-in rules, for instance, mean accepting some credit risk on small transactions rather than declining every card at the checkout; someone in the business, as well as in IT, has to own that trade-off.

Smaller institutions face the same problem one step removed. A community bank or credit union rarely contracts with a hyperscaler, one of the very largest cloud providers, directly. Its exposure arrives through its core processor (the company that runs its central account system), digital banking provider and card processor. Oversight therefore means asking those providers for their own regional dependency maps and test results, and writing that disclosure into renewal terms. A provider that cannot describe its degraded modes is telling the institution something about its resilience.

Where an outage day costs a bank

A cloud outage looks like something that happens to engineers. For a financial institution, the costs land almost everywhere else.

Customers. People judge a bank by whether they can see their balance and move their money when they want to. An app that fails to load does not come with an explanation about DNS in northern Virginia; it simply looks as though the bank is down. Customers who cannot pay a bill, complete a purchase or confirm that their paycheck arrived will call, and call centers fill quickly. Clear, early messaging through a channel that still works is worth more on that day than almost any technical fix.

Money. Missed payment cut-offs and settlement windows have direct costs: late fees, penalty interest, manual work to reconcile what went through and what did not, and sometimes compensation for customers. A queue that clears at 9 p.m. instead of 3 p.m. can push work into the next business day, which in payments is a real difference.

Staff. On an outage day the operations team is doing two jobs at once: fixing the problem and explaining it. If their chat tool, paging system or dashboards are affected too, they are doing both with one hand tied. Burnout after a long incident is real, and recovery tails make incidents long.

Regulators. Supervisors on both sides of the Atlantic expect firms to understand their dependence on third parties, including those third parties’ own suppliers. After a public outage, examiners are likely to ask how a firm was affected, how quickly it recovered and what it learned. A firm with a dependency map and test results can answer. A firm that can only point to its provider’s uptime commitment cannot.

Reputation. In a widespread outage, nobody expects a single bank to have prevented the provider’s failure. What customers and the press notice is how each institution behaved: who told customers what was happening, who kept essential services running in reduced form and who went silent. Degraded modes and out-of-band communication are, in that sense, reputation tools as much as technical ones.

All of these costs are shaped by decisions made long before the outage: which dependencies were accepted, which degraded modes were designed and which tests were run. That puts the subject on the board and executive agenda as well as in the architecture review.

Where multi-cloud earns its cost

The same framing settles much of the multi-cloud debate. Running the same workload on two providers is expensive and adds failure modes of its own, from data synchronization to doubled skills requirements. For most institutions, multi-region designs within one provider, plus a few rehearsed degraded modes, deliver more resilience per dollar. Multi-cloud earns its cost for specific components where independence is cheap and valuable, such as DNS, identity, status pages and customer notifications.

Take a regional bank that has followed this approach. It runs its digital banking on one major cloud provider, across two regions. For each of its five most important customer journeys it keeps a one-page map listing every dependency, who provides it and where it runs. Its status page and customer text-message service sit with a different provider, so it can tell customers what is happening even if its main cloud is struggling. Its card program has stand-in rules, owned by the head of cards, for approving small purchases when the authorization service is unreachable. Once a year it rehearses losing a region and measures when the app comes back, when the payment queue is empty and whether the day’s settlement deadline was met.

The usual errors run the other way. Some firms assume that because their own application is multi-region, the firm is. Others treat the provider’s uptime commitment as if it were the firm’s own resilience, or measure recovery by the moment the dashboard turns green rather than the moment the backlog clears. Degraded modes get designed on paper without a business owner willing to accept their risks. And some reach for multi-cloud everywhere, spending heavily on duplication while the shared login vendor remains a single point of failure.

What a bank should know before the next regional outage

1. For our five most important customer journeys, what is every service involved, who provides it and in which cloud region does it run?
2. Which of our SaaS vendors and processors depend on the same provider and region as we do, and have we asked them in writing?
3. If our main cloud region failed tomorrow, what channel would we use to talk to each other and to customers?
4. When we test recovery, do we measure when the backlog clears and whether payment and settlement deadlines were met, or only when systems come back?
5. What degraded modes do we have, who owns each one, what risk limit applies and when did we last rehearse it?

With hindsight

Large incidents followed within weeks: an Azure Front Door outage on October 29 and a Cloudflare outage on November 18 each disrupted widely used services. Azure Front Door is Microsoft’s service for routing web traffic to applications; Cloudflare sits in front of a vast number of websites to speed them up and protect them. Together the three incidents showed the exposure was shared infrastructure rather than one provider. A firm that had fled AWS for another cloud in late October would not have escaped the November headlines. The outage made a stronger case for changing how institutions test cloud resilience than for changing how they buy cloud.

Supervisors moved too. In November 2025 the European Supervisory Authorities published their first list of critical ICT third-party providers under DORA, which included the EU entities of Amazon Web Services, Google Cloud and Microsoft. That put two of the three providers whose failures had been in the headlines under direct EU oversight. Regulators now treat a handful of technology companies as part of the financial system’s plumbing, and expect the institutions that use them to understand how that plumbing could fail.

Calls for a wholesale move to multi-cloud ran into the usual cost and complexity. The changes worth making are in due diligence, by asking fintech partners and SaaS vendors which regions they depend on, and in measuring recovery against business deadlines rather than service restoration alone. Those changes are less dramatic than switching providers, but they address what actually stopped apps working that Monday: the services around the core, and the long queue after the fix. A firm that knows what depends on what, and how long it takes to be truly back, is better placed for the next outage, whichever company it comes from.