S3’s February 2017 outage reached services built on it, including the console used to update the status page

On February 28, 2017, a command entered with a wrong input took down S3’s index subsystem in Amazon’s Northern Virginia cloud region. Other services built on S3 failed with it, and the status dashboard could not be updated normally until 11:37 a.m. Pacific time.

Looking Back revisits past weeks one at a time, up to the most recent, and asks what it meant for the banks, health systems and public agencies we serve. The series was written in 2026, with the benefit of hindsight, and each article says which week it looks back on.

Two-minute summaryThe dek and the first sentence of each section, quoted from the article

On February 28, 2017, a command entered with a wrong input took down S3’s index subsystem in Amazon’s Northern Virginia cloud region. Other services built on S3 failed with it, and the status dashboard could not be updated normally until 11:37 a.m. Pacific time.

  1. The week On the morning of February 28, 2017, Amazon’s cloud business had a service disruption in its Northern Virginia region. [1] The company later published a summary of the event.
  2. Direct dependency: the customer’s own use of S3 AWS said S3 was unable to service requests while the index and placement subsystems restarted. [1] Any application that read objects from or wrote objects to S3 in that region was affected.
  3. Inherited dependency: provider services built on S3 None of those services is S3.
  4. Observational dependency: the status channel shared the failure AWS said that from the beginning of the event until 11:37 a.m. Pacific time, it was unable to update individual services’ status on its Service Health Dashboard. [1] The cause was a dependency the dashboard’s administration console had on S3.
  5. Recovery dependency: the provider’s restart set the duration The longest part of the outage was not the mistake.
  6. How the dependencies reach different financial institutions The four kinds of dependency apply across the sector, but they weigh differently by type of institution.
  7. Objections, in order AWS did change the tool and add safeguards. [1] The error was the trigger.
  8. What technology and resilience leaders could check Each check produces a record: a list, a decision, a test result.
  9. With hindsight On December 7, 2021, AWS had another service disruption in its Northern Virginia region.

The week

On the morning of February 28, 2017, Amazon’s cloud business had a service disruption in its Northern Virginia region. [1] The company later published a summary of the event. The copy we cite was captured by the Internet Archive on March 2, 2017. [1]

The summary said the Amazon Simple Storage Service team was debugging an issue that was causing the S3 billing system to progress more slowly than expected. [1] At 9:37 a.m. Pacific time, an authorized team member using an established playbook executed a command intended to remove a small number of servers for one of the S3 subsystems used by the billing process. [1]

One of the inputs to the command was entered incorrectly, the summary said, and a larger set of servers was removed than intended. [1] The servers removed supported two other S3 subsystems. [1]

The first was the index subsystem. The summary said it manages the metadata and location information of all S3 objects in the region, and is necessary to serve all GET, LIST, PUT and DELETE requests. [1] The second was the placement subsystem, which manages allocation of new storage and requires the index subsystem to function. [1]

Removing a significant portion of capacity caused each of these systems to require a full restart, according to the summary. [1] While they were being restarted, S3 was unable to service requests. [1]

The disruption spread to other services. The summary said other services in the region that rely on S3 for storage were also impacted while the S3 APIs were unavailable. [1] It listed the S3 console, new instance launches in Amazon EC2, Amazon EBS volumes when data was needed from an S3 snapshot, and AWS Lambda. [1]

The company explained why recovery took as long as it did. S3 subsystems were designed to support the removal or failure of significant capacity with little or no customer impact. [1] But it said it had not completely restarted the index or placement subsystems in its larger regions for many years. [1] S3 had grown, and restarting and running safety checks to validate the integrity of the metadata took longer than expected. [1]

The summary gave the recovery times. By 12:26 p.m. Pacific time, the index subsystem had activated enough capacity to begin servicing GET, LIST and DELETE requests. [1] By 1:18 p.m., those APIs were functioning normally. The placement subsystem finished recovery at 1:54 p.m., and S3 was then operating normally. [1]

Other services that had accumulated a backlog of work during the disruption needed more time to fully recover, the summary said. [1]

The summary also described a problem with status reporting. From the beginning of the event until 11:37 a.m. Pacific time, the company said it was unable to update the individual services’ status on its Service Health Dashboard. [1] The reason was a dependency the dashboard’s administration console had on S3. It used its Twitter feed and banner text on the dashboard instead. [1]

The summary listed changes. It modified the capacity removal tool to remove capacity more slowly and added safeguards to prevent removal below a subsystem’s minimum required capacity. [1] It said it was auditing its other operational tools for similar safety checks. [1]

It also said it would improve the recovery time of key S3 subsystems. It described breaking services into small partitions it calls cells, and said the S3 team was reprioritizing planned further partitioning of the index subsystem to begin immediately. [1] It said it had changed the dashboard’s administration console to run across multiple regions. [1]

For banks, the relevant supervisory text at the time was an appendix to the FFIEC’s Business Continuity Planning booklet, issued in February 2015. It was titled “Strengthening the Resilience of Outsourced Technology Services.” [2] [3] The FDIC transmitted it to its supervised institutions on February 23, 2015, as FIL-9-2015. [3]

The appendix said a financial institution’s reliance on third-party service providers does not relieve it of responsibility for the resilience of outsourced activities. [2] It discussed four areas: third-party management, third-party capacity, testing with third-party service providers, and cyber resilience. [2]

The February 28 outage is easy to remember as a typo. We read it as a map of dependencies. AWS’s own summary described what we count as four different ways a customer depended on one storage subsystem, and only one of them was the obvious one. [1]

Our argument is that a bank’s outsourcing review in 2017 was built to see the first kind of dependency and to miss the other three. FFIEC’s appendix asked banks to look at subcontractors, single points of failure and recovery testing. [2] It did not anticipate that the provider’s own services, and the channel used to report on them, could depend on the same component a bank’s application used.

We take the dependencies in four parts. Each is a different path by which the S3 index subsystem reached a customer:

  1. Direct: the customer’s own data and applications that read from and wrote to S3.
  2. Inherited: other AWS services the customer used, which relied on S3 underneath.
  3. Observational: the status channel the customer would use to learn what was wrong.
  4. Recovery: the provider’s restart process, which set how long the outage lasted.

The four paths do not overlap. A bank can know its direct dependencies from its own architecture. It learns inherited ones only from the provider. Observational and recovery dependencies sit entirely on the provider’s side, and the bank sees them only when something fails.

The outage did not create these dependencies. It made them visible on one day, in one region, in a provider’s own words. [1] The analysis that follows applies to any provider, AWS included.

Direct dependency: the customer’s own use of S3

AWS said S3 was unable to service requests while the index and placement subsystems restarted. [1] Any application that read objects from or wrote objects to S3 in that region was affected. The index subsystem was necessary for GET, LIST, PUT and DELETE requests, so reads and writes both failed. [1]

What a bank would have known

A bank’s architecture records should show which applications use which storage. That is the dependency a vendor review is designed to see. A bank that kept those records could list its affected applications within minutes of learning that S3 was down.

The appendix asked institutions to assess the effectiveness of a service provider’s business continuity program, with emphasis on recovery capabilities and capacity. [2] It also asked them to review whether the provider’s plan aligned with their own. [2] For a direct dependency on a single region’s storage, the alignment question is simple to state: what does the bank do when that region’s storage is unavailable?

Region as a unit of failure

The disruption was in one region. [1] A bank that had copies of critical data in another region could, in principle, have read from them. Whether it could actually do so depended on design choices made long before the event.

Replication across regions has costs. It adds storage and transfer charges. It adds complexity to applications that must know where to read. It adds consistency questions when writes happen in two places. Those costs are why a single-region design can look reasonable on paper.

The outage showed the other side of the trade. A single region is a single point of failure for the services in it. The appendix asked institutions to identify single points of failure that would cause wide-scale disruption. [2] A cloud region fits that description for the workloads that live only there.

Recovery objectives and the provider’s clock

The appendix said a financial institution should be able to demonstrate the ability to recover critical systems and resume operations within established recovery time objectives, whether the process is in-house or at a provider. [2]

The S3 recovery took from 9:37 a.m. to 1:54 p.m. Pacific time by AWS’s account, before backlogs in other services cleared. [1] A bank whose recovery time objective for a function was shorter than that window, and whose function depended solely on the region, could not meet its own objective. The objective was the bank’s. The clock was AWS’s.

Testing a direct dependency

The appendix said testing should be sufficient in scope and rigor to demonstrate the ability to meet recovery objectives, whether a service is performed in-house or outsourced. [2] For a direct dependency on cloud storage, the test is to make the storage unavailable to the application and observe what happens.

That test is within the bank’s control. It does not need the provider’s participation. A bank can block access to a region’s storage from its own application in a test environment and measure whether its failover works and how long it takes.

Reading when writing fails

Not every function needs to write. A bank’s customer-facing pages may only need to read documents, rates or statements. Those functions can be designed to serve from a cache or a copy when the primary store is unavailable.

AWS said the index subsystem was needed for reads and writes alike. [1] A cache held outside that subsystem would have kept some read-only functions working. The design choice is to decide, function by function, whether stale data is better than no data, and to build the fallback before it is needed.

Backups held in the same failure domain

A bank that backs up data into the same region and service as its primary copy has two copies with one dependency. When the service fails, both are unreachable.

The appendix said cloud-based disaster recovery services carry unique data integrity risks, and that management should assess such services before implementation and periodically afterward. [2] The assessment should include where the recovery copy lives relative to the primary, and whether the same component failure would reach both.

What the vendor file should hold

The appendix asked for contracts reviewed by counsel and subject matter experts, with terms on audit rights, data governance, subcontracting and business continuity. [2] For a cloud dependency, the vendor file can also hold architecture facts: the regions used, the services relied on, and the recovery path if one region fails.

Those facts change as engineers change systems. A vendor file updated only at contract renewal describes last year’s architecture. Tying the file to change management keeps it current enough to use during an incident.

Inherited dependency: provider services built on S3

AWS said other services in the region that rely on S3 for storage were impacted while the S3 APIs were unavailable. [1] It named the S3 console, new EC2 instance launches, EBS volumes when data was needed from an S3 snapshot, and Lambda. [1]

None of those services is S3. A bank could use them without storing a single object in S3 itself. The dependency was inside the provider, where the bank could not see it.

Why inherited dependencies escape review

A vendor review asks which services the bank uses. It records EC2, EBS and Lambda as separate items. Unless the provider discloses its own internal dependencies, the review has no way to know that all three share a component.

The appendix pointed at a related idea. It said the failure of a subcontractor could result in the failure of the service provider’s ability to deliver contracted services, and asked institutions to understand the due diligence process a provider uses for its subcontractors. [2] It also asked them to review available audit reports on providers’ resiliency and interdependencies, such as subcontractors. [2] An internal shared component is not a subcontractor. It behaves like one, because its failure passes through.

Failover that needs the failed service

Inherited dependencies can defeat a recovery plan in a specific way. A plan to recover from a failed server may depend on launching a new one. AWS said new instance launches were among the impacted functions. [1]

A recovery design that assumes new capacity can always be launched in the affected region fails when launches depend on the failed component. The same applies to volumes restored from snapshots, which AWS also listed. [1] The recovery tool was itself downstream of the failure.

Asking the provider the right question

The appendix said contracts should empower an institution to request information from its provider about the provider’s response to supervisory guidance and other notices. [2] A bank can extend that idea to architecture. It can ask which of the provider’s services share critical internal components, and which recovery functions depend on them.

A provider may answer at a high level or not at all. Even a high-level answer helps. Knowing that launches, snapshots and serverless functions share a storage dependency in a region tells a bank not to rely on any of them to recover from a storage failure there.

Mapping inherited dependencies from incidents

Public post-incident summaries are one source of this knowledge. AWS’s 2017 summary named the affected services. [1] A bank that read such summaries and recorded the dependencies they revealed would build, over time, a map the provider never published as a single document.

That map is imperfect. It shows only dependencies that have failed in public. It is still better than a vendor inventory that treats each service as independent.

Control plane and data plane

The 2017 and 2021 summaries together point at a distinction that matters for recovery design. One part of a service handles running workloads. Another part handles creating, changing and managing them. [1] [4]

In 2021, AWS said running EC2 instances were unaffected while the APIs used to launch and describe instances had elevated errors. [4] A recovery plan that only needs running capacity can survive that kind of event. A plan that needs to create new capacity cannot. Knowing which of its recovery steps need the management side lets a bank keep them out of the failure.

Pre-provisioned capacity

One response is to keep recovery capacity already running, rather than launching it at the time of failure. That costs money every day the capacity sits idle. It removes the dependency on launches during an event.

The trade is the same one as multi-region storage. A bank decides, for each critical function, whether the cost of standing capacity is worth the independence it buys. The decision should be written down with the recovery objective it supports.

Software vendors on the same provider

A bank’s software vendors may run on the same cloud provider, in the same region, as the bank’s own workloads. A bank that spread its risk by using several vendors may find they share one region underneath.

The appendix discussed concentration among service providers and the risk that a critical failure at one could have large-scale consequences. [2] The cloud adds a layer: vendors that look independent can share a provider and a region. The only way to know is to ask each vendor where it runs and record the answers side by side.

Observational dependency: the status channel shared the failure

AWS said that from the beginning of the event until 11:37 a.m. Pacific time, it was unable to update individual services’ status on its Service Health Dashboard. [1] The cause was a dependency the dashboard’s administration console had on S3. AWS used its Twitter feed and banner text instead. [1]

For an incident commander, this dependency carries special weight. During the first part of the outage, the official per-service status page could not be updated through its normal console. A bank watching that page for confirmation would have seen less than the provider knew.

Why the observation path is a dependency

A bank’s incident response starts with knowing what is wrong. If the bank’s monitoring shows errors, it needs to know whether the cause is its own systems or the provider’s. One place to check is the provider’s status page.

When that page depends on the failing component, the bank loses its confirmation channel at the moment it needs it. The bank may spend time investigating its own systems for a problem that is not theirs. Or it may wait for a confirmation that arrives late.

Independent signals

The answer is a signal that does not depend on the provider’s status page. A bank can run its own synthetic checks against the provider’s services from outside the provider’s region. If storage requests from an independent location fail, the bank knows the provider has a problem, whatever the status page says.

The checks should mirror what the bank’s applications actually do. A storage check reads and writes a test object. A compute check launches and stops a small instance, since launches were among the functions AWS said were impaired. [1] A check that only pings an endpoint can pass while the operation the bank needs is failing.

Those checks are cheap. They need to be designed in advance, run continuously and wired into the bank’s alerting. Their value is highest in exactly the scenario the 2017 summary described, when the provider’s own channel is impaired. [1]

Communication plans that assume the channel

Banks have their own communication duties during an outage. Customers, regulators and internal staff all need information. A communication plan that waits for the provider’s status page to confirm a cause may delay those messages.

Who declares the incident

A bank needs a rule for declaring a provider incident without the provider’s confirmation. The rule names who can declare, on what evidence, and what follows: activation of the bank’s own response, customer messaging and escalation.

Without such a rule, the decision waits for the status page. The 2017 summary shows how long that wait could be for per-service status. [1] A bank whose own checks showed failures could have declared and acted while the provider’s page lagged.

Regulatory notice clocks

In November 2021, the OCC, Federal Reserve and FDIC issued a rule on computer-security incident notification. [8] It requires a banking organization to notify its primary federal regulator of a notification incident as soon as possible and no later than 36 hours after determining it occurred. [8] The rule took effect April 1, 2022, with a compliance date of May 1, 2022. [8]

The rule also requires a bank service provider to notify affected bank customers as soon as possible after it determines it has had a qualifying computer-security incident. [8] The incident must have caused, or be reasonably likely to cause, a material service disruption or degradation for four or more hours. [8]

The agencies said such incidents can result from non-malicious failure of hardware and software and from personnel errors, as well as from cyberattacks. [8] On our reading, an outage like the 2017 one could fall within the rule’s scope where its other conditions are met. Counsel should confirm how the definitions apply to a given provider and bank.

The rule gives the observational dependency a regulatory edge. A bank’s 36-hour clock starts when it determines a notification incident occurred. [8] A bank that cannot see what its provider is doing may make that determination late.

The appendix discussed the risk that business resilience strategies depend on functioning communication links, and listed reliance on a single communications provider as a possible single point of failure. [2] The 2017 event adds a variant: reliance on the provider’s own status channel as the single source of truth.

The provider’s fix and its limits

AWS said it had changed the dashboard’s administration console to run across multiple regions. [1] That fix addressed the specific dependency on S3 in one region. It did not, and could not, remove every shared dependency between a status system and the services it reports on.

The hindsight section returns to this point. A later AWS event showed a status tooling failover problem of a different kind. [4] A bank is better served by its own independent signal than by trusting that the provider has removed every shared path.

Recovery dependency: the provider’s restart set the duration

The longest part of the outage was not the mistake. It was the restart. AWS said it had not completely restarted the index or placement subsystems in its larger regions for many years, and that restarting and validating metadata integrity took longer than expected. [1]

A recovery path that had not been exercised

The summary described a recovery path that existed but had not been run at that scale. S3 had grown substantially, AWS said, and the process of restarting the services and running safety checks took longer than expected. [1]

A procedure that works at one size can behave differently at a larger one. Without a recent full exercise, no one knows how long it will take.

What the bank could and could not control

A bank has no control over a provider’s restart procedures. It cannot test them. It cannot speed them up during an event. It can only design its own systems so that they do not depend on the provider recovering quickly.

The appendix asked institutions to participate in their providers’ business continuity testing when possible, and noted that proxy tests may not fully capture an institution’s needs. [2] For a large cloud provider, participation in internal restart tests is not realistic. The bank’s remaining lever is its own architecture.

Cells and blast radius

AWS described its approach of breaking services into small partitions called cells, to make recovery processes testable and to reduce what it called blast radius. [1] It said it was reprioritizing further partitioning of the index subsystem. [1]

A bank can apply the same idea to its own systems. Partitioning workloads, so that one failure reaches part of the customer base rather than all of it, limits the damage when a dependency fails. It also makes the bank’s own recovery procedures testable at a size that can actually be exercised.

Backlogs after recovery

AWS said some services had accumulated a backlog during the disruption and needed more time to fully recover. [1] Recovery of the underlying component was not the end of the outage for every service.

Banks face the same pattern. Payment files, batch jobs and messages queue up during an outage. When the dependency returns, the queue must drain before operations are normal. A recovery plan that measures only when the provider is back understates the real duration for the bank.

Exercising the bank’s own restarts

The S3 summary is a warning about untested recovery at scale, and it applies to banks’ own systems as well as providers’. [1] A bank that has not restarted a large internal system from cold in years does not know how long it takes.

The appendix said testing should demonstrate the ability to meet recovery objectives and should disclose the adequacy of both organizations’ ability to recover, restore and resume. [2] A full restart exercise of the bank’s own critical systems, at production scale, is the internal half of that test.

Recovery objectives written with the provider in mind

A recovery time objective is a promise about how long a function can be down. When the function depends on a provider, the promise is only as good as the provider’s recovery, unless the bank has a path that does not need it.

Writing the objective with the provider’s history in view makes it honest. The bank can read the provider’s published incident summaries, note how long recoveries took, and decide whether its objective assumes something the record does not support. [1] [4]

Communicating during a long recovery

A recovery measured in hours needs a communication rhythm. Customers want to know whether to wait or use another channel. Staff want to know what to tell them. Regulators may need notice, depending on the rules that apply. [8]

The bank’s messages cannot wait for the provider’s timeline. They can say what the bank knows: which services are affected, what customers can do meanwhile, and when the next update will come. The provider’s summary arrives later and fills in the cause. [1]

How the dependencies reach different financial institutions

The four kinds of dependency apply across the sector, but they weigh differently by type of institution. We take the main types in turn.

Large banks

Large firms with their own engineering teams can design multi-region architectures, run independent monitoring and partition workloads. The interagency sound practices paper of 2020 was directed to the largest and most complex domestic firms, and cited a growing reliance on third-party service providers among the sources of operational risk. [5]

Community banks

Community banks may reach cloud services through a core processor or other provider rather than directly. Their direct dependency on a cloud region may be invisible to them, because it belongs to their provider. The appendix’s point about subcontractors applies: the bank’s provider is the customer of the cloud, and the bank inherits that dependency. [2]

For these banks, the practical question is what their providers disclose. A bank can ask its core and digital banking providers which cloud regions host its services, and what happens if one region fails.

Credit unions

Credit unions face the same structure as community banks. The NCUA is among the four agencies that issued the 2026 proposal discussed in the hindsight section. [7] The dependency questions apply to a credit union’s providers as they do to a bank’s.

Broker-dealers and trading firms

For firms whose operations are tied to market hours, the recovery window matters more than the cause. An outage during trading hours that lasts longer than the firm’s tolerance forces manual procedures or halts. The observational dependency also matters: a firm needs to know quickly whether to fail over.

Fintechs

Fintechs that run directly on a cloud provider have direct exposure, with no core processor between them and the cloud. When they partner with banks, the banks inherit the fintech’s cloud dependencies through the partnership. A bank reviewing a fintech partner can ask the same four questions about the partner’s architecture.

Payment and market utilities

Infrastructure providers that serve a large number of institutions concentrate the effect of a dependency failure. The appendix discussed concentration among service providers and said a critical failure at a provider could have large-scale consequences. [2] A utility that depends on a single cloud region carries that concentration one layer further down.

Wealth managers and advisers

Advisory firms that run client portals, trading tools and document storage on cloud services face the same four dependencies. Their exposure depends on whether they host directly or through a platform provider.

For firms on a platform, the questions go to the platform: which regions host client data, what fails over, and how the platform will tell the firm when something is wrong. The answers belong in the firm’s vendor file, beside the contract.

Bank service providers

Core processors, digital banking platforms and other providers to banks sit in the middle of the chain. They are customers of cloud providers and suppliers to banks. The 2021 rule requires a bank service provider to notify affected bank customers as soon as possible after it determines that a qualifying incident has occurred. [8]

That duty makes the observational dependency the provider’s problem as well as the bank’s. A provider that cannot see what its cloud is doing cannot tell its banks promptly. Independent monitoring at the provider level serves its banks and its own notice duty.

Objections, in order

1. This was a one-time human error that AWS fixed

AWS did change the tool and add safeguards. [1] The error was the trigger. The dependencies it revealed were structural: other services built on S3, a status console that depended on it, and a restart that had not been exercised at scale. [1] Those structures exist in any large provider, whatever triggers the next failure.

2. Cloud is more reliable than a bank’s own data center

That may be true for some workloads. The argument here concerns visibility, not cloud as such: the dependencies inside a provider are invisible to a review that treats each service as separate, and the bank still owns its recovery objectives. [2]

3. Multi-region designs are too expensive

They cost more. The cost is a choice about how much disruption a function can tolerate. For functions with recovery objectives shorter than a regional outage, the bank either pays for a second region or accepts that it cannot meet its objective when the region fails. [1] [2]

4. Banks cannot see inside a provider, so the inherited dependency is not their problem

The appendix placed responsibility for oversight on the institution’s management and board, whether systems are managed in-house or by a provider. [2] A bank cannot see everything. It can still ask, read incident summaries and avoid relying on recovery tools in the same failure domain.

5. The provider’s status page is good enough

On February 28, 2017, the per-service status page could not be updated through its console until 11:37 a.m. Pacific time, by AWS’s account. [1] A bank that relied only on it would have had less information than an independent check could provide.

6. Supervisory guidance has moved on

The guidance has changed, as the hindsight section shows. [6] [7] The dependencies have not. Any version of third-party guidance leaves the bank responsible for knowing what its critical services depend on.

7. Regulators never required multi-region designs

The guidance we cite does not prescribe an architecture. [2] [6] It asks institutions to meet their own recovery objectives and to identify single points of failure. [2] Whether that requires a second region is a conclusion each bank reaches from its own objectives, not a rule written into the guidance.

8. A provider’s own summary is not neutral evidence

The 2017 and 2021 summaries are AWS’s accounts of its own events. [1] [4] We rely on them as such, not as independent findings. They are still a detailed public record of what failed, and a provider has little reason to overstate its own failures. We give them weight on the points we use for that reason.

What technology and resilience leaders could check

The four kinds of dependency give a short list of checks:

  • List the critical functions that depend on a single cloud region, directly or through a provider.
  • For each, compare the recovery time objective with a realistic regional outage, and decide whether to pay for a second region or accept the gap.
  • Record which provider services share internal dependencies, using provider disclosures and published incident summaries.
  • Make sure recovery procedures do not rely on launching capacity or restoring snapshots in the region that failed.
  • Run independent synthetic checks against critical provider services from outside the provider’s region.
  • Measure recovery to the point where queued work has cleared, rather than to the point the provider is back.
  • Ask each software vendor which cloud provider and region it runs in, and record the answers side by side.

Each check produces a record: a list, a decision, a test result. Those records are what an examiner can review and what an incident team can use at short notice. A check that leaves no record helps only the people who did it, and only while they remember.

These checks fit an existing third-party risk program. They add architecture questions to contract and due diligence questions, which is where the 2017 outage showed the gap. [1] [2]

Where to start

A bank that has done none of this can start with one function. It picks the customer-facing function it would least like to lose, and traces its dependencies through all four paths. The exercise takes days, not months, for one function.

The result shows where the bank’s records are thin. Perhaps the region is known but the inherited services are not. Perhaps there is no independent check. Each gap found for one function is likely to exist for others, so the first trace sets the agenda for the rest. The trace should be repeated after any major architecture change, because the dependency paths move whenever the design does.

What a supervisor would ask

An examiner reviewing a bank’s cloud dependencies under the appendix would start from the bank’s own documents. [2] The questions follow the four kinds of dependency. Which critical functions use which region? Which provider services do those functions need to recover?

The other two questions concern the provider’s side: how the bank learns of a provider failure, and how long the bank can wait for the provider to recover.

A bank that can answer each question from a record is in a stronger position than one that answers from memory. The records also help the bank itself. They turn a provider outage from a surprise into a scenario the bank has already walked through.

With hindsight

On December 7, 2021, AWS had another service disruption in its Northern Virginia region. Its summary, dated December 10, 2021, traced the event to congestion on networking devices between its main network and an internal network. [4] AWS said that internal network hosts foundational services, including monitoring, internal DNS, authorization services and parts of the EC2 control plane. [4]

The 2021 summary described dependencies of several of the same kinds. Running EC2 instances were unaffected, but the EC2 APIs used to launch new instances had elevated errors, and customers of some other services could not create new resources. [4] Monitoring data was not visible to AWS’s own operations team, which impaired its ability to find the source of the congestion. [4]

The status channel was affected again, in a different way. AWS said the networking congestion impaired its Service Health Dashboard tooling from failing over to its standby region, and that it was updating the dashboard by 8:22 a.m. Pacific time. [4] It also said its Support Contact Center relied on the internal network, so the ability to create support cases was impacted from 7:33 a.m. until 2:25 p.m. Pacific time. [4]

The 2017 fix had moved the dashboard console across regions. [1] The 2021 event showed that a failover path can itself depend on the failing component. [4] The lesson about independent signals held up.

Supervisory guidance moved in several steps. In SR 20-24, a supervisory letter dated November 2, 2020, the Federal Reserve transmitted an interagency paper on sound practices to strengthen operational resilience, which it said the Federal Reserve, FDIC and OCC had issued. [5] It was directed to the largest and most complex domestic firms. [5] On June 2, 2026, the Federal Reserve revised the paper to remove references to reputational risk. [5]

In June 2023, the Federal Reserve, FDIC and OCC issued final interagency guidance on third-party relationships, final as of June 6, 2023. [6] It rescinded and replaced the Board’s 2013 guidance, the FDIC’s 2008 guidance, and the OCC’s 2013 guidance and 2020 frequently asked questions. [6]

The FDIC’s web page for FIL-9-2015, which transmitted the FFIEC appendix, is now marked inactive, with a note that its content has expired or been rescinded by the FDIC. [3] The appendix we cite therefore describes the supervisory expectations of its time, not current FDIC guidance. [2] [3]

On September 15, 2026, the OCC, Federal Reserve, FDIC and NCUA proposed new third-party risk management guidance. [7] They said any finalized guidance would replace the 2023 guidance and its supplemental resources, and that they planned to rescind and replace existing guidance. [7] The agencies asked for comments on or before November 16, 2026. [7]

The proposal emphasized aligning oversight with reasonably assessed risk levels for each relationship, and tailoring by size, complexity and risk profile. [7] It listed cloud processing and cloud storage among services that support banks’ essential operating functions. [7] It said the use of subcontractors alone does not typically create an independent third-party relationship or a presumption of direct oversight of subcontractors. [7]

That last statement differs in emphasis from the 2015 appendix, which asked institutions to understand providers’ due diligence on their subcontractors and to review audit reports on providers’ interdependencies, such as subcontractors. [2] [7] As checked on November 15, 2026, the proposal was open for comment and not final. Banks that plan around it should note that its text may change.

What held up from 2017 was the map of dependencies. Two AWS summaries, from 2017 and 2021, described services built on shared internal components and status tools that shared the failure. [1] [4] What changed was the supervisory frame, which moved from an FFIEC appendix to interagency guidance and now to a proposal centered on risk-based tailoring. [6] [7]

Under every version, the bank remains responsible for knowing what its critical functions depend on. The four kinds of dependency are a way to ask that question that does not depend on which guidance is in force. [2] [7]

Spotted an error? Tell us and we will check it and correct the article where needed.

  • cloud outage
  • AWS
  • third-party risk
  • operational resilience
  • FFIEC
  • concentration risk