A deleted file, a national ground stop, and the runbook nobody rehearsed

The week

Before any airliner pushes back from the gate, its pilots read a stack of short safety bulletins about the trip ahead: a closed runway, a crane near the approach path, a navigation aid out of service. These bulletins are called NOTAMs, and they flow through a system run by the Federal Aviation Administration. Pilots must review them before flight. If the system that distributes them stops working, flying safely becomes a much harder question.

The name itself was new. Until December 2021 the FAA called these bulletins Notices to Airmen; it adopted the gender-neutral Notice to Air Missions while keeping the familiar acronym. The content had not changed. NOTAMs cover everything from runway closures and lighting outages to temporary flight restrictions, and airline dispatch systems pull them automatically into the briefing packages crews read before departure.

Late on January 10, 2023, the FAA’s Notice to Air Missions system failed. Early on January 11 the FAA ordered airlines to pause all domestic departures until 9 a.m. Eastern time. Departures resumed later that morning, but thousands of flights were delayed or canceled.

The sequence mattered. The failure began in the evening, when traffic was light. Overnight, according to reports at the time, the FAA relayed urgent notices to pilots by telephone, a manual fallback that could cope with a handful of overnight flights. It could not cope with the first wave of weekday departures. The FAA also needed to confirm that the restored system held complete and correct information before relying on it. A pause on departures bought that time, at the cost of disrupting travel across the country.

The agency was led at the time by an acting administrator, Billy Nolen. Transportation Secretary Pete Buttigieg said publicly that he had directed an after-action review, and members of Congress from both parties called for explanations. A national ground stop caused by a software failure, rather than by weather or a security threat, was the kind of event that drew attention well outside aviation.

The first suspicion in many minds was a cyberattack. In its preliminary review, the FAA said it had found no evidence of a cyberattack or malicious intent. Instead, it said contract personnel had unintentionally deleted files while working to correct synchronization between the live primary database and a backup database.

A database is the organized store of information a computer system relies on; for NOTAMs, it holds the bulletins themselves. Critical systems usually keep two copies: the primary, which the system uses in real time, and a backup, kept ready in case the primary fails. Synchronization is the process that keeps the two copies matching. When they drift apart, someone has to bring them back into line. That was the job under way when files were deleted.

The FAA also said it had taken steps to make the system more resilient. Chief among them was a synchronization delay, so that bad data in one database could not immediately reach the backup. That change addressed the exact mechanism of the failure. It also implicitly confirmed the central weakness: the backup had been set up to mirror every change, including a harmful one, at once.

The timing sharpened the story. The outage came weeks after Southwest Airlines’ holiday scheduling collapse, when a single airline’s operations had unraveled over the holidays. In a filing on January 6, Southwest estimated that the disruption would have a negative pre-tax impact of $725 million to $825 million on its fourth-quarter results. The airline’s crew scheduling tools had been unable to keep up with mass cancellations, and recovery depended on staff reconstructing schedules by hand. Both events drew attention to aging technology in critical transportation systems, and to how much daily life depends on software most people never see.

The public conversation quickly settled on one answer: the FAA needed a new system. That was true. It was also an incomplete account of what the night revealed.

Every organization that runs an important computer system has a version of that night waiting for it. A routine task, scheduled out of hours to avoid disruption, carried out by capable people following the steps they have always followed. Then one step goes wrong, and the safety net turns out to be attached to the same rope.

The obvious conclusion from the NOTAM failure was that the FAA needed a new system, and it did. A new system alone would not have prevented that night, though. The trigger was a maintenance task on a primary database and its backup, performed in a way that damaged both. That kind of failure can happen on a platform of any age.

Three separate sources of risk converged: the procedure, the people performing it, and an architecture that let one mistake reach both copies. Each is something a leader can ask about and fund, without needing to understand the system’s code.

Copies, mirrors and snapshots

A mission system works like a busy reference library that must never close. The database is the collection on the shelves. Production is the library as the public uses it, live and in real time. Maintenance is the work of repairing, reorganizing and updating the collection while the doors stay open.

To survive a disaster, the library keeps a second, identical collection in another building. That is the backup. Replication, or synchronization, is the process of copying every change from the first building to the second as it happens, so that if the first burns down, the second can open immediately with nothing lost.

Replication copies everything, including mistakes. If someone tears pages out of a book in the first building, the second building faithfully tears out the same pages. A synchronized backup is excellent protection against a hardware failure, a power cut or a flood. It offers little protection against a bad change, because it is designed to mirror the original exactly.

Protection against human error needs something else: a point-in-time copy. This is a snapshot of the collection as it stood at a particular moment, say last night, stored separately and left alone. If today’s work goes wrong, the library can be restored to how it was before the work began. The snapshot is only useful if it is truly separate, recent enough, and if the team knows how to restore from it quickly.

Four kinds of copy, four kinds of protection

Resilience teams distinguish several kinds of copy, and each protects against something different.

• Synchronous or near-synchronous replicas mirror every change within seconds. They protect against hardware failure and site loss with almost no data lost. They copy errors just as fast.
• Delayed replicas apply changes after a set lag, perhaps an hour. They give a team a window to stop a bad change before it reaches the second copy, at the cost of a slightly staler fallback.
• Point-in-time backups or snapshots capture the data as it stood at a moment and keep it. They protect against bad changes and corruption, provided they are stored out of reach of the systems and people doing the work.
• Isolated or immutable copies cannot be altered or deleted for a set period, even by administrators. They were popularized as a defense against ransomware, which deliberately targets backups, and they protect equally well against an honest mistake.

A mission system usually needs more than one of these. The design question is which failure each copy is meant to survive, and whether any routine task can reach all of them at once.

Replication became the default for a sensible reason. For decades, the most common threats to availability were hardware failures, power cuts and site outages. Replication answered those well, and its success made it easy to forget the threats it does not cover. Operator error, faulty software updates and malicious deletion all travel through the replication channel unless something stops them.

Two more terms complete the picture. A runbook is the step-by-step instruction sheet for an operational task, like a pilot’s checklist. A rollback plan is the agreed route back to safety if a change goes wrong, with a clear point at which the team stops trying to fix things and restores the earlier state instead.

Change management is the wider discipline these terms belong to. It is the set of rules for how any change to a production system is proposed, assessed, approved, carried out and checked. Federal agencies will recognize it from the configuration management controls in NIST Special Publication 800-53, which also covers backup and recovery under its contingency planning family.

The procedure

Re-synchronizing a primary database with its backup is routine work. It is dangerous because it touches both copies of the data on which the system’s resilience depends.

A sound procedure for that work has several parts. It needs a written runbook that someone other than its author has reviewed. It needs a fresh, verified copy of the data taken beforehand and stored where the task cannot reach it. It needs a rehearsal in a non-production environment, a test copy of the system, that mirrors production closely enough to matter. And it needs a rollback plan with a fixed point at which the team stops and restores.

Legacy systems often lack some of these. Their procedures grew up informally over decades. Steps live in the memory of experienced staff, in personal notes, or in documents last revised before the most recent major change. Nobody decided to run critical maintenance this way. It is what remains when a system outlives several generations of its operators.

For a program director, the first useful question about a critical legacy system is therefore less its replacement date than whether every maintenance procedure that touches production data or its backups is written down, current, rehearsed and actually followed.

A short audit can answer that within weeks: list the procedures, record when each was last reviewed and last executed, and compare the document with what operators really do. The gap between the written runbook and real practice is often the most revealing finding. Experienced operators develop shortcuts that work well until the day they meet an unusual situation.

Rare tasks need different rules from routine ones

Most change management processes sort changes into categories. Standard changes are low-risk, pre-approved and repeated often. Normal changes go through review and approval. Emergency changes get expedited handling. The trouble is that a task like re-synchronizing a database can look standard because it has been done before. It is done rarely, it touches both copies and its failure is severe. Those three features should push it into a higher category automatically.

A simple test can help. A task deserves the full treatment if it is performed less often than every month, if it touches more than one copy of the data, or if its failure would stop the mission. Any one of those conditions should require a reviewed runbook, a fresh isolated copy, a second qualified person and a named stop point. All three together should require a senior engineer present and a rehearsal within the previous few days.

Pre-checks are part of the procedure, not an optional extra. Before touching anything, the team should confirm the current replication state, confirm that the latest point-in-time copy exists and can be read, and record a baseline count of key records. After the work, the same checks show whether anything was lost. Without a baseline, nobody can say with confidence whether the data is complete, which is precisely the doubt that can force a system offline for validation.

Why rehearsal environments drift

Rehearsal depends on a test system that behaves like production, and keeping one is harder than it sounds. Test systems often hold a smaller or older copy of the data. They may run on different hardware or a different software version. Configuration changes made in production during an incident may never be carried back. Over time the test system becomes a rough sketch of the real one.

Keeping them aligned costs money and attention. Production data may need to be masked before it can be copied into a test system, especially where it includes personal information. Infrastructure for a full-scale copy may double hosting costs. Many agencies compromise with a smaller test system and accept that some steps will behave differently. That is a reasonable trade-off, as long as the runbook flags which steps were not truly rehearsed, so the team on the night knows where to slow down.

An audit in practice

Suppose a state benefits agency runs an eligibility system that dates back decades. Its new chief operating officer asks for a list of every procedure that touches live data or backups. The list comes back with a column showing when each was last reviewed. Several have never been reviewed by anyone but their author; one refers to a server that was retired years ago. The most critical procedure, used a few times a year, has never been rehearsed on a test system. Little of this costs much to fix, and all of it was invisible until someone asked.

What a safe re-synchronization looks like

Done well, the same kind of task looks like this. Say an agency’s operations lead schedules a re-synchronization of a primary database and its backup. Two days earlier, the team rehearses the runbook on a test system that mirrors production, and notes one step that behaves differently from the document; the runbook is corrected and re-reviewed. On the night, before touching anything, the team takes a fresh point-in-time copy and stores it somewhere the task cannot reach, then confirms the copy can be read. A second qualified engineer watches each step and approves it before it runs, and the session is recorded.

The runbook names a stop point: if the copies do not match by a certain step, or if any unexpected deletion appears, the team halts and restores rather than improvising. Midway through, something looks wrong. The team stops, restores from the isolated copy, confirms the system is healthy and reschedules the work for daylight with more senior staff present. Users notice a brief pause, if anything. The next morning’s report is dull, which is exactly what a good maintenance report should be.

Nothing in that story requires new technology. It requires a written procedure, a rehearsal, a separate copy, a second pair of eyes and the authority to stop. Each of those is a management choice.

The authority to stop deserves emphasis. Engineers working at night on a task that is behind schedule feel pressure to finish, especially if users will notice a delay in the morning. A runbook that names a stop point, and a manager who has said in advance that stopping will be supported, changes that calculation. Without both, the incentive is to improvise, and improvisation on both copies of critical data is how small errors become national ones.

The people

The FAA said contractor personnel made the error. That needs careful reading. It does not show that contractors are riskier than federal employees. It does point to a common pattern in government IT. Detailed system knowledge sits with contract staff whose teams can change at each recompetition, the periodic reopening of a contract to bids. The federal program office holds accountability but not always the operational detail.

Consider how knowledge drains away under that model. A system is built by one contractor, maintained by a second and then recompeted again. Each handover loses a little of the unwritten knowledge about why things are done a certain way. After a few cycles, the people running the system may understand its daily routine well but not the rare, risky tasks that come up only a few times a year. Those rare tasks are where the danger concentrates.

That arrangement needs specific safeguards. Contracts for critical systems can make current runbooks a government-owned deliverable, updated after every significant change. Changes to production data can require a second qualified person to review and approve each step in real time, with the session recorded. This is two-person control, the same principle that requires two officers to open a bank vault. Transitions between contractors can include supervised handover of operational procedures, with outgoing staff observing incoming staff perform them.

What the contract can require

The contract is the main lever an agency has, and it is often underused. A contract for a critical system can list current runbooks, architecture diagrams and restore test results as formal deliverables, with due dates and acceptance criteria. Otherwise they remain informal artifacts on a contractor’s file share. It can require that key engineers for the riskiest tasks be named, with notice to the agency before they leave. It can make a successful, timed restore test a performance measure, tied to award fees or incentive payments where the contract type allows.

Transition clauses matter most. Federal contracts can already require an outgoing contractor to cooperate with its successor; the Federal Acquisition Regulation includes a continuity of services clause for that purpose. An agency can go further and specify the content of the handover. That could mean a joint walk-through of every high-risk procedure, a period of shadowing, and sign-off by the government’s own technical lead before the outgoing team leaves. Those terms are cheapest to write before award, when the agency has the most bargaining power.

Controlling privileged access

Two-person control has a technical counterpart in how privileged access is granted. Many organizations now use tools that issue administrative access only for a specific task and a limited time, record the session, and require approval from a second person. Those tools make the rules enforceable rather than aspirational. They also produce the evidence an inspector general or auditor will ask for: who had access, when, what they did, and who approved it.

The same records answer the first question the public asks after an outage. An agency that can show within hours exactly which commands were run, and by whom, can rule out an attack with confidence rather than with hope.

The government should also keep enough technical staff of its own to understand and challenge what the contractor is doing on its most important systems. An agency that cannot read its own runbooks cannot tell whether they are any good.

That in-house capacity does not need to be large. For each critical system, one government technical lead who can read the runbooks, attend the highest-risk changes and own the restore test results is often enough to change the dynamic. Agencies that cannot hire quickly have other routes. They can rotate engineers from other programs, draw on federally funded research and development centers, or bring in an independent verification contractor whose job is to check the operator’s work rather than do it. The FAA, for example, already relies on a federally funded center operated by MITRE for advanced aviation systems engineering.

Each route has a cost and a limit. In-house hires compete with private-sector salaries. Independent reviewers add another party to coordinate. But an agency with no one able to question its contractor has, in effect, delegated its accountability along with the work.

Timing matters as well. Maintenance on critical systems is often scheduled overnight to limit disruption. That reduces the operational impact of a planned outage but also means fewer experienced people are available when something goes wrong. Agencies can set rules for which tasks require senior staff on call, and for when a task must stop and wait for daylight.

These safeguards cost money. Two-person control slows maintenance. Government-owned runbooks require someone to review them. These costs are small next to a national ground stop, but they have to be funded in operations budgets that are usually squeezed to pay for modernization.

Learning without scapegoats

After an incident, the pressure to name a culprit is strong, especially when the person who made the final error was a contractor. Aviation itself offers a better model. The NASA-run Aviation Safety Reporting System lets pilots, controllers and mechanics report errors and near misses confidentially, so the system can learn from them. The premise is that most errors are produced by conditions that make them likely, and that punishing individuals discourages the reports that reveal those conditions.

Technology operations teams use the same idea under the name of blameless post-incident reviews. The review asks what about the procedure, the tooling and the staffing made the error possible, and what would stop the next person making it. Accountability still exists, but it attaches to fixing conditions. An agency that removes the individual and changes nothing else has only reset the clock.

The architecture

The third risk was that one maintenance error could affect both the primary and the backup. A backup continuously synchronized with production protects against hardware failure. It does not protect against corrupted data or a bad change, because the corruption is copied too. Protection against operator error needs a different mechanism: point-in-time copies held separately, with a tested way to restore them quickly.

Here legacy design genuinely matters. Older systems were often built when synchronized replication was the main resilience goal. Restoring from a point-in-time copy may be slow, poorly documented or never practiced.

A modernization plan should make fast restoration of known-good data an explicit requirement, with a target time tied to the mission. Specialists call this a recovery time objective: the longest the organization can tolerate being without the system. For a safety system that grounds aircraft when it fails, that target is measured in minutes or a few hours.

How much data can be lost, and how to know it is right

A second target stands alongside recovery time. The recovery point objective is the maximum amount of recent data the organization can afford to lose, expressed as time. A system restored from last night’s snapshot loses everything entered since. For a safety system that publishes notices, losing even a few hours of entries could mean a missing runway closure. The recovery point therefore has to be very recent, or the lost entries have to be re-entered and checked.

That explains why restoration alone is not enough. After any restore, someone has to confirm that the data is complete and correct before the system goes back into service. For a mission system, that validation step can take longer than the technical restore. It needs tools to compare record counts and checksums against a known baseline, reports that show what changed during the gap, and a defined owner who can declare the data trustworthy. Systems designed without that step leave leaders choosing between running on data they cannot vouch for and keeping the system offline while people check by hand.

When the backup shares the flaw

Engineers call it a common-mode failure when one cause defeats a system and its backup together. A mirrored database is one example. Others include a primary and backup running identical software with the same defect, or two sites that depend on the same network provider. A failover process that relies on the same credentials as the system it protects is another. A useful exercise is to list, for each critical system, everything the primary and the backup have in common. Each shared item is a path by which one failure could reach both.

The design also needs a degraded mode: a way to keep the most essential information flowing, even manually, while the main system is restored. Deciding in advance what that minimum service is, and who can declare it, avoids improvising a national decision in the middle of the night. A hospital that loses its electronic records falls back to paper forms it has practiced with; a mission system deserves the same forethought.

Until a replacement exists, the old system has to be run safely, and the replacement can be damaged by a careless maintenance step as easily as the original. The most useful test applies to both. Ask the operations team to demonstrate a restore from an isolated copy and time it. The result shows where exposure sits more clearly than the system’s age does.

Cloud migration does not remove these questions. Managed database services make snapshots and delayed replicas easier to configure, and many offer protected backups that cannot be deleted for a set period. They do not decide which of those features to turn on, how long to retain copies, who may delete them, or how long a restore and validation will take. A system moved to the cloud with the same assumptions about replication will fail in the same way.

Consequences beyond the server room

An outage like this has a technical cause and technical people to fix it. Its consequences are anything but technical.

• Mission. For the FAA, the system’s failure meant aircraft could not depart. For a benefits agency it might mean payments do not go out; for a court system, hearings cannot proceed; for a bank, customers cannot reach their money.
• Cost. Thousands of delayed or canceled flights carry costs far beyond the agency’s own budget. The price of the safeguards described here is modest by comparison.
• Trust. When a public system fails, the first question is often whether it was attacked. Being able to say quickly and credibly what happened, and that it was not malicious, depends on good records of who did what. Recorded maintenance sessions are as valuable for that as for prevention.
• Dependents. Mission systems feed other systems. The NOTAM outage reached airlines’ own planning and dispatch processes, which depended on a feed they did not control. Any agency whose data feeds partners, other agencies or the public should know who those consumers are and how they will be told when the feed is unreliable.
• Staff. The people who run legacy systems often carry a heavy and lonely burden: rare, high-stakes tasks performed at night with little backup. Clear procedures, a second qualified colleague and the explicit right to stop and wait for daylight protect them as much as the system. Staff who know they will be supported for halting a risky task are far more likely to halt it.
• Oversight. Legislators, inspectors general and the press will ask why a known weakness was not addressed. “We were waiting for the new system” is not a satisfying answer if the new system is years away.

The deeper problem is budgets. Modernization programs attract attention and funding because they promise a fresh start. Operational discipline on the existing system rarely does. Yet the old system is the one running today, and it will keep running until the day the new one fully replaces it.

How the same failure plays out in different organizations

The pattern is general, but the rules and stakes differ by sector. A federal civilian agency operates under FISMA and the NIST control catalog. An inspector general will expect to see evidence of change control, separation of duties and tested contingency plans for high-impact systems. A state or county agency often runs a critical system with a much smaller team and a single long-term vendor. Its knowledge drain is sharper, and one departing engineer can take a large share of operational memory. Where the system holds criminal justice information, the FBI’s CJIS Security Policy adds its own requirements on access and auditing.

In healthcare, an electronic health record outage moves clinicians to downtime procedures. Hospitals typically keep read-only copies of key patient information on designated computers and practice paper workflows, because care cannot wait for a restore. The equivalent of the NOTAM question is whether those downtime copies are current and whether staff have actually used them.

In financial services, examiners apply the FFIEC’s business continuity guidance. The federal banking agencies’ 2020 paper on sound practices for operational resilience also asks firms to identify critical operations and test their ability to deliver them through disruption. Broker-dealers must keep business continuity plans under FINRA Rule 4370. Exchanges, clearing agencies and certain other market infrastructure fall under the SEC’s Regulation SCI, which sets a two-hour resumption goal for their critical systems. Each regime asks, in its own language, the question the NOTAM failure raised: can the organization restore good data, quickly, after its own staff make a mistake?

Where the weak points usually sit

The agencies in better shape share a few habits. Every procedure that touches live data or backups is written, reviewed by someone other than its author, rehearsed on a test system and dated, so nothing depends on the memory of one person who may be on leave or moved to another contract. Point-in-time copies sit where routine maintenance cannot reach them, and restores are practiced and timed; a backup that has never been restored is a hope rather than a plan. Each critical system has a recovery target set by mission leaders, not by whatever the technology happens to allow, and a defined degraded mode with a named person authorized to declare it. The government owns current runbooks, contractor transitions include supervised handover, and in-house staff can challenge the contractor’s work.

Measures a program director can track

Leaders who cannot read the code can still track a handful of numbers each quarter. The first is the share of high-risk procedures reviewed and rehearsed in the past twelve months. The second is the date and duration of the last timed restore from an isolated copy, set against the mission’s recovery target. The third is the number of production data changes made without a recorded second approver, which should be zero. The fourth is turnover among the named engineers who perform the riskiest tasks.

Producing these measures requires no new systems. Most can be pulled from change tickets, access logs and contract records. Reported together on one page, they tell a program director more about the risk of a night like January 10 than any assessment of the system’s age.

The recurring errors run the other way. Replication gets treated as backup, though a mirror copies mistakes as faithfully as it copies data. Money moves from running the old system safely to building the new one, raising risk through the years of overlap. And after an incident, blame lands on the individual who made the final error, when the useful question is why a single mistake could reach both copies.

The case for putting replacement first

The strongest objection to this argument comes from people who run legacy systems. They point out that some old platforms cannot support modern controls at all. Snapshots may be impossible without taking the system down. Session recording tools may not work with its interfaces. The engineers who understand it may be retiring. Every dollar spent shoring up the old system, they argue, delays the one investment that would remove these problems for good.

Part of that is right. Some controls cannot be retrofitted, and a replacement designed with fast restoration in mind is the durable answer. But replacement programs for large federal systems routinely take years, and the old system carries the full load throughout. The procedures, staffing rules and restore tests described here are cheap compared with the replacement, and most can be applied to the old platform in months. Where a technical control is impossible, a procedural one can often stand in: a manual export before maintenance, a mandatory second engineer, a daylight-only rule. The argument here does not oppose replacement. It opposes treating replacement as the only safeguard.

Runbooks and restore tests for systems that cannot stop

1. Which of our systems would stop a core part of our mission if they failed tonight?
2. For each, is every procedure that touches live data or its backups written down, current and rehearsed, and does any routine task touch both the main copy and the backup at once?
3. When did we last restore from an isolated point-in-time copy, and how long did it take compared with what the mission can tolerate?
4. Which procedures depend on contractor staff, and do we own current copies of them?
5. How much of our operations budget is protected from being diverted to the replacement program?

These questions apply equally to a system built last year and one built decades ago.

With hindsight

Congress kept the system on its agenda. The FAA Reauthorization Act of 2024, signed on May 16, 2024, defined the agency’s pre-2000 legacy systems to include the Notice to Air Missions system. It required a plan to replace or enhance such systems, and set up a research program to inform modernization of the FAA’s aeronautical information systems, NOTAM included. Wider air traffic control technology problems drew sustained attention through 2025.

Those problems became highly visible. In late April 2025, controllers handling traffic for Newark Liberty International Airport briefly lost radar and radio contact with aircraft, and further outages followed in the weeks after. The Transportation Department announced plans for a new air traffic control system, and in July 2025 Congress provided about $12.5 billion for air traffic control modernization in that year’s budget reconciliation law. The scale of the commitment confirmed how far the underlying infrastructure had fallen behind.

That is a meaningful response, and it is the kind of response legislatures are good at: naming a problem, requiring a plan and directing research. What it cannot do on its own is change how a system is operated on a given night.

Replacement has taken years, and the old system had to be run in the meantime, which shows the limits of replacement as the main answer. This is the uncomfortable arithmetic of modernization everywhere. The years between announcing a new system and switching off the old one are often the riskiest, because attention and money flow to the future while the present keeps carrying the load.

The operational basics, rehearsed runbooks, two-person control for production data changes and isolated point-in-time backups, protect both old and new systems. Not one of them depends on the age of the technology. Southwest’s holiday collapse, the other technology failure in the news that week, led to a record $140 million civil penalty from the Transportation Department in December 2023. Regulators now treat operational failures that strand customers as a consumer protection issue too. A brand-new platform maintained without them can fail in exactly the same way.

Events elsewhere reinforced that. In August 2023, the United Kingdom’s air traffic provider, NATS, suffered a failure in its flight plan processing system on a busy holiday Monday. A single unusual flight plan triggered a fault, and the backup system, running the same software, failed in the same way. Airlines canceled large numbers of flights to and from the UK. It was a textbook common-mode failure: a backup that shared the primary’s flaw.

In July 2024, a faulty content update from the security vendor CrowdStrike crashed millions of Windows computers worldwide; Microsoft estimated about 8.5 million devices. Airlines, hospitals and emergency call centers were among those disrupted, and some airlines took days to recover. The update was routine. It reached every customer at nearly the same time, with no staged rollout to catch the defect early. CrowdStrike afterward committed to staged deployment and more customer control over updates. The parallel with the NOTAM failure is close: a routine change, applied everywhere at once, with no point at which a human could catch the error before it spread.

Both events affected modern systems as well as old ones. Neither was caused by an attacker. In each case the question that mattered was how the change process and the recovery design handled a single mistake.

A single test cuts through debates about system age. Agencies should ask whether they have actually restored known-good data from a separate copy within the time their mission allows. Not planned to, not documented how to, but done it, recently, with a stopwatch running. Organizations that can answer yes have reduced the risk that ended up grounding flights that January morning. Those that cannot should make that restore test their next priority.