Why CISA’s Ivanti disconnect order was decided by people, before it was issued

The week

The story began in early January, when Ivanti disclosed two vulnerabilities in its Connect Secure and Policy Secure gateways. These gateways are the devices that let staff working from home or in the field connect securely to an organization’s internal network from the open internet. A vulnerability is a flaw in software that an attacker can exploit. Attackers were chaining these two together for unauthenticated remote code execution. Someone on the internet, without any password, could make the gateway run their own commands. For a device whose whole job is to decide who gets in, that is about as bad as it gets.

On January 19, CISA, the Cybersecurity and Infrastructure Security Agency, issued Emergency Directive 24-01. An emergency directive is a binding order to federal civilian agencies. This one required them to apply Ivanti’s mitigation, a temporary configuration fix, and run its integrity checking tool, which looks for signs that a device has been tampered with.

On January 31, after more vulnerabilities were disclosed and attackers were seen working around the mitigation, CISA went further and issued supplemental direction. Agencies had to disconnect all affected products from their networks by 11:59 p.m. on February 2. Before reconnecting, they had to rebuild or factory reset the devices, apply patches, and revoke and reissue certificates, keys and passwords stored on them. They were also told to assume that associated domain accounts were compromised, with credential resets due by March 1. Told on a Wednesday, agencies had until Friday night to unplug the front door that thousands of remote staff used to reach work, rebuild it from scratch and change every key it had touched.

The directive bound only federal civilian agencies. State and local governments and private organizations running the same gateways were under no such order. They faced the same flaws and the same attackers, but each had to decide for itself what to do.

Emergency patching is usually treated as a problem of inventory and threat hunting: knowing what devices you have, and looking for signs that attackers are already inside. The January 31 direction exposed a different constraint. Most affected agencies knew they had Ivanti gateways; the devices were the front door for remote staff. Nobody needed a scan to find them.

The hard part was human. Meeting the deadline required three things of people, in sequence. Someone needed the authority to take remote access away from a workforce. Someone needed the skills to rebuild the device and reset what it touched. And a contractor had to put the agency first when every customer faced the same deadline. Each constraint was set months earlier, by decisions about delegations, staffing and contracts.

What a remote-access gateway is, and why it is a prime target

A remote-access gateway, often called a VPN gateway, sits at the edge of an organization’s network. VPN stands for virtual private network: a protected tunnel through the public internet that lets a laptop at home behave as if it were plugged in at the office. The gateway is at the end of that tunnel. It checks who is arriving, confirms their identity, and lets them through to internal systems.

Because of that role, gateways have three properties that make them attractive to attackers. They face the internet, so anyone in the world can try to talk to them. They are trusted by everything behind them, because their job is to vouch for visitors. And they are connected to the organization’s identity systems, so they handle passwords, certificates and session tokens all day long. A certificate is a digital ID card that machines use to prove who they are; a token is a temporary pass issued after someone has signed in.

When such a device is compromised, the damage is not limited to the box itself. An attacker may have copied the keys and passwords it holds, or used it to reach the accounts it connects to. That is why the supplemental direction went beyond patching. Fixing the software would close the hole, but it would not take back the keys an intruder might already have pocketed. Hence the instruction to rebuild, to reissue every secret, and to treat connected accounts as compromised.

Who decides: the authority to cut remote access

Disconnecting a VPN gateway means some staff cannot work until it returns. For an agency with field inspectors, benefits processors working from home, or contractors who connect only through that gateway, that is a mission decision. A security officer can recommend it. In most agencies, only a program executive or the deputy secretary’s office can accept the impact.

Where that decision had not been delegated in advance, the clock started late. The directive arrived, the security team briefed the CIO, the CIO briefed leadership, and leadership asked which operations would stop. Each step took hours that the deadline did not include.

The agencies that moved quickly had settled two things beforehand. They had a written delegation that let the CIO or CISO (the chief information officer or chief information security officer) disconnect an internet-facing device under a federal directive without further approval. They also had a documented degraded mode for remote work: a second access path for priority users, a list of staff who would come on site, and a message ready for the workforce. With those in place, the disconnect was a planned outage rather than an executive debate.

State and local governments faced a version of the same problem without the directive’s force. CISA had no power to order them to act. A county IT director who wanted to take the remote-access gateway offline over a weekend had to persuade department heads, a sheriff’s office or a court administrator, none of whom reported to IT. Where an incident response plan already named the IT director as the decision-maker for emergency disconnects, the conversation was short. Where it did not, the gateway often stayed up while the device was patched in place, which was exactly the approach the supplemental direction had judged insufficient for federal networks.

One county, two weekends

Take a county government whose gateway serves caseworkers who visit families, building inspectors in the field, and a handful of court staff who work remotely. On Wednesday afternoon the IT director reads the federal direction and concludes that the county should follow it, even though nobody has ordered it to.

In one version, the county’s incident response plan says nothing about who may disconnect shared systems. The IT director calls the county administrator, who asks the department heads. The court administrator objects because a filing deadline falls on Monday. The sheriff’s office wants to know whether patrol laptops are affected. By Friday, the meetings are still going, and the gateway is still online with a patch applied.

In the other version, the plan already names the IT director as the person who can order an emergency disconnect of an internet-facing device, with notice to the administrator. A one-page degraded-mode plan lists which caseworkers come into the office, and a pre-written message tells staff what to expect. The decision takes an hour, and the weekend goes to rebuilding the gateway.

Who does: skills to rebuild and reset

A factory reset or rebuild of a gateway is routine for an experienced network engineer. The full sequence is longer. It needs a known-good configuration to restore, new certificates issued and installed, integration with the identity provider reconnected, and testing before users return. The supplemental direction also told agencies to assume that domain accounts associated with the devices were compromised. On-premises passwords were to be reset twice, Kerberos tickets revoked, and then cloud tokens revoked in hybrid environments. That reached deep into Active Directory and cloud identity.

A short translation of those terms. Active Directory is the long-standing Microsoft system that manages user accounts and permissions inside most organizations’ own networks. Kerberos is the ticketing scheme it uses: once you sign in, you get tickets that let you into specific systems without retyping your password, much like wristbands at a festival. Revoking tickets means cutting off every wristband already handed out. A hybrid environment is one where the organization runs some identity services on its own servers and some in the cloud, so both halves must be cleaned. Resetting passwords twice is a standard precaution in Active Directory, because of how it keeps the previous password valid for a period.

Those steps span three skill sets: network engineering, public key infrastructure and identity administration. Public key infrastructure, or PKI, is the system that issues and manages digital certificates. In a large department these sit in separate teams that rarely work the same weekend. In a small agency they may rest with one or two people. A common failure point is more mundane: nobody can reach the one person who holds the device’s administrator credentials or the certificate authority access.

The workforce controls are simple and often missing. Each critical edge device should have at least two named people, staff or contractor, who have rebuilt it in an exercise within the past year. The configuration backup and rebuild procedure should be stored outside the device and outside the network it protects; a rebuild guide that lives only on the broken system is no guide at all. Identity teams should have a rehearsed runbook for revoking and reissuing credentials tied to a compromised appliance, including service accounts. A runbook is a step-by-step checklist; service accounts are the non-human accounts that software uses to talk to other software, and they are easy to forget.

Who else is waiting: the contractor queue

Many agencies do not operate their own edge devices. A managed service provider or a systems integrator runs them under a task order, which is a work order issued under a larger government contract. On January 31, those providers faced the same deadline across every federal customer at once, plus state, local and commercial customers dealing with the same vulnerabilities. Their engineers became the scarcest resource in the response.

Contracts rarely addressed this. Typical service levels covered incident response times for outages, not surge capacity for a directive that hits every customer simultaneously. An agency had little contractual basis to insist on going first. Agencies with small task orders were likely to wait longest.

Outsourcing moves the work to a provider, but it does not move the accountability. When the deadline passes, it is the agency that must report whether it complied. If the agency cannot act without the provider, and the provider is busy with larger customers, the agency’s security depends on its place in someone else’s queue.

There are practical fixes. Contracts for managed network and security services can name federal emergency directives as a priority event, with a defined response time and a requirement for the provider to report its plan within hours. Agencies can require that the provider’s staffing plan cover simultaneous demand. At a minimum, the agency should hold its own administrator access and configuration backups, so its staff can act if the provider cannot.

How the clock actually runs inside an organization

The roughly two days between the supplemental direction on January 31 and the 11:59 p.m. deadline on February 2 show where the hours go. Each step below is manageable on its own. The trouble is that they happen in sequence, and each one waits for the last.

First comes reading and understanding. Someone in the security office has to notice the new direction, read it carefully, and work out exactly which devices it covers. Directives are precise documents, and a misreading can mean doing the wrong work. Then comes the internal briefing: the security lead tells the CIO, and the CIO decides who else must know.

Next is the decision to disconnect, discussed above. If it needs to climb the hierarchy, it can easily consume most of the first day. Only after the decision is made can the organization tell its staff, and only after staff are told can the gateway safely go dark without causing confusion and a flood of help desk calls.

Then the technical work begins. The device is taken offline and reset or rebuilt. Patches are applied. New certificates are requested and installed. The connection to the identity system is restored. Testing confirms that everything works before users are let back in. Running alongside, the identity team begins the longer job of resetting connected accounts, which the direction allowed until March 1 but which still had to be planned and staffed.

Each of those handoffs is a point where a missing person, an unclear authority or a busy contractor can stall the whole chain. An organization that has rehearsed the sequence knows where its bottlenecks are. One that has not discovers them on the night, with the deadline approaching. That is why the most useful preparation is often a simple tabletop exercise: leaders and technical staff sitting around a table, walking through an imagined directive hour by hour, and noting every moment when someone says “we would have to ask.”

A clean test of what leadership controls

The Ivanti episode is a useful stress test because the instructions were unusually clear. There was no ambiguity about what to do. The only question was whether the organization could do it in time. That isolates the variables that leadership controls: who is allowed to decide, who is trained to act, and what suppliers have promised.

The costs of failing that test are concrete. A gateway left online and merely patched may already harbor an intruder. Credentials that are not reset may let that intruder return. For a federal agency, missing a binding directive is a compliance failure that must be explained. For a county or a private firm, the exposure is to data theft, service disruption and the reputational damage that follows a public breach. Meanwhile, the cost of preparing is modest: a signed delegation, two trained people per device, a stored backup, and a clause in a contract.

A boring day, by design

A well-prepared organization looks almost boring on the day. The security lead reads the directive, confirms which devices are affected, and invokes a delegation that already exists. A message goes out to staff from a template. Two named engineers, who rebuilt the device in an exercise last spring, pull the configuration from a backup stored elsewhere. The identity team opens its runbook for credential resets. The managed service provider, bound by a priority clause, calls within hours with its plan.

The ways it goes wrong are just as predictable:

• Undelegated decisions. The authority to cut remote access sits only at the top, so the clock runs while leaders are briefed.
• No degraded mode. Nobody has decided who works on site or through another path when the gateway goes dark.
• Single points of knowledge. One person holds the admin password or the certificate access, and that person is unreachable.
• Backups in the wrong place. The rebuild guide and configuration sit on the very network the device protects.
• Contracts written for ordinary outages. Service levels cover a single customer’s incident, not every customer’s emergency at once.
• Patching instead of rebuilding. Fixing the software on a device that may already be compromised, and leaving its keys in place.

What to fix before the next directive

1. If we had to disconnect our remote-access gateway tonight, who could make that decision without waiting for anyone else, and what is our plan for the staff and operations that depend on it?
2. For each critical internet-facing device, who are the two people who could rebuild it? When did they last practice?
3. Where is the configuration backup and rebuild procedure stored, and could we reach it if the device and its network were offline?
4. Do we have a rehearsed plan for resetting every credential tied to a compromised device, including service accounts?
5. If a contractor runs the device, what does our contract say about emergencies that hit all their customers at once, and where would we sit in the queue if every customer called on the same day?

These three constraints are independent. An agency can have clear authority and skilled staff yet still wait on a contractor, or a responsive contractor and no one empowered to approve the outage. Holding its own administrator access gives it a fallback either way. Each must be resolved before the next directive, because none can be fixed in 48 hours.

With hindsight

The Ivanti episode did not end with that weekend. On February 29, 2024, CISA and partner agencies warned that Ivanti’s integrity checker could miss compromise and that an attacker might keep persistence through a factory reset. Persistence means a foothold that survives attempts to remove it. Even organizations that followed the instructions to the letter could not be fully certain they were clean.

The flaws reached the defenders themselves. In March, CISA said two of its own systems had been compromised through Ivanti vulnerabilities. In April, MITRE, the nonprofit that operates federally funded research centers, disclosed that a research network had been breached through the same flaws in January. If the nation’s cyber defense agency and a respected research organization could be caught, no organization had grounds for complacency.

The problem also recurred. Ivanti Connect Secure faced another exploited zero-day in January 2025; a zero-day is a flaw attackers exploit before a fix is available. CISA issued further emergency directives in 2025, including one on Cisco firewalls that September.

Each of these later events reinforced the same point. The first set of instructions is rarely the last, and an organization that struggled to act once will struggle again when the guidance changes.

The pattern held: each order came with a deadline in days and assumed that agencies had the authority, people and contract terms to act. The technology changed from one directive to the next. The human prerequisites did not.

An agency that wrote down its delegations and named rebuild owners after the Ivanti weekend had, in effect, rehearsed for every directive that followed. In any sector, emergency readiness is mostly decided in calm periods. Signing a delegation, funding a second trained engineer and renegotiating a contract clause are dull tasks. They are also the tasks that decide whether an organization can act in two days when the next instruction arrives.