Storm-0558’s post-mortem described three key failures every agency development team can repeat

The week

In July 2023, Microsoft disclosed that a China-based actor it calls Storm-0558 had been reading email at about two dozen organizations, including U.S. government agencies, through Exchange Online and Outlook on the web. The method was unusual. The actor had obtained a consumer signing key and used it to forge authentication tokens.

That needs a word of explanation. When you log in to an online service, the service does not ask for your password every time you click. Instead, after you sign in, an identity system hands your browser a token, a small digital pass that says, in effect, “this is Jane, she is allowed in, signed by us.” The signature is made with a secret cryptographic key. Any application that sees the pass checks the signature, and if it matches, lets the holder in. Whoever holds the signing key can therefore print passes for anyone.

On September 6, 2023, Microsoft published the results of its investigation into how Storm-0558 obtained that key. The account was strikingly ordinary. The key had most likely ended up in a crash dump from a signing system. A crash dump is a snapshot of a computer’s memory taken when a program fails, so engineers can work out what went wrong; it can capture whatever the program was holding at the time, including secrets.

According to Microsoft, the dump was then moved from an isolated production environment, the tightly guarded systems that run the live service, into a debugging environment on Microsoft’s corporate network, where engineers examine problems. The actor later compromised an engineer’s account with access to that environment.

There was a second failure. Microsoft also described a validation flaw that let its mail system accept a token signed with a consumer key for enterprise accounts. A key meant for personal accounts should never have opened government and corporate mailboxes, but it did.

The September account turned a remote, nation-state story into something closer to home for agencies. Crash dumps, debugging environments and token checks are not unique to the world’s largest cloud providers. They exist wherever an organization writes software, and many government agencies and their contractors do exactly that.

Storm-0558 is usually discussed as a story about cloud provider accountability, and the federal review that followed in 2024 concentrated there. That framing is fair. But the September post-mortem also described failures that have nothing to do with being a hyperscale provider, the industry term for the handful of companies that run cloud services at enormous global scale.

Crash dumps, debugging environments and token validation exist in every organization that writes software, including federal and state agencies with their own development teams and the contractors who build systems for them. The post-mortem points to three ways a signing key turns into an attacker’s asset, and they follow the life of a key in order: how key material escapes, how far a stolen key reaches, and how long it stays useful. All three make sense only once it is clear what a signing key does and why it is so valuable.

Keys, tokens and wristbands

Most modern applications do not handle passwords themselves. They hand that job to an identity provider, a dedicated service whose only purpose is to confirm who someone is. Staff at an agency may sign in through a commercial cloud identity service. Members of the public may sign in through Login.gov or a state platform. Partner agencies may trust each other’s sign-ins through an arrangement called federation.

When the identity provider is satisfied, it issues a token. A token is a short block of data stating who the person is, which organization they belong to, which application the token is for, what they are allowed to do, and when the token expires. The identity provider signs the token with its private key, a long secret number. Applications check the signature with a matching public key, which can be shared openly. If the signature checks out, the application believes what the token says.

Concert wristbands work the same way. The venue prints wristbands for Friday, for Saturday and for the VIP area, each with a hologram that is hard to fake. Staff at each door check the hologram and the color. If a counterfeiter gets hold of the hologram machine, they can print any wristband they like. And if the door staff at the VIP area only check the hologram and not the color, a Friday wristband gets you backstage on Saturday.

The wristband story closely matches what went wrong. The attacker obtained the machine, the signing key. And a door, the enterprise mail system, checked the hologram but not whether the wristband was the right kind.

Three facts follow, and they shape everything below. First, a signing key is one of the most valuable secrets an organization holds, because it can impersonate anyone who trusts it. Second, the signature is only one of several checks an application must make. Third, a key is useful to an attacker for as long as applications keep trusting it.

Failure one: how key material escapes

The most striking detail was mundane. Microsoft later said it never found the dump in question, so this path remains its best theory rather than an established fact, but the scenario it described is common. A system crashed, a memory dump was captured for diagnosis, and the dump contained key material that should have been excluded. Automated checks meant to catch secrets in such dumps missed it. The dump then moved, as diagnostic data routinely does, to an environment where engineers could examine it, which was less protected than the production system it came from.

Nothing in that chain requires carelessness by any single person. Engineers need to diagnose crashes. Diagnostic data has to go somewhere they can look at it. The weakness is structural: secrets flow downhill, from well-guarded systems to less-guarded ones, inside the routine work of keeping software running.

Agency development teams move diagnostic data this way every week: core dumps, heap snapshots, verbose application logs, database copies made for troubleshooting, and browser session recordings attached to vendor support tickets. Each can carry secrets, session tokens or personal data. Suppose a state benefits agency’s eligibility system slows down during an enrollment period. To find the cause, a contractor copies the production database to a test server and turns on detailed logging. That copy now holds the personal data of every applicant, and the logs may hold session tokens. The test server, built for convenience, has broad access and weaker monitoring. Nobody intended to create a second, softer copy of the crown jewels, but that is what happened.

Three controls address the path.

• Keep keys out of memory entirely. Keep signing keys in hardware security modules or managed key services, so that the private key never sits in application memory and cannot appear in a dump. A hardware security module is a tamper-resistant device that holds keys and performs signing on request without ever releasing the key itself. Cloud providers offer managed key services that do the same job as a service.
• Clean data before it leaves. Scrub diagnostic artifacts automatically before they leave production, and test the scrubber against seeded fake secrets as part of each release. Seeding means deliberately planting fake passwords and keys in test data to confirm the scrubber catches them.
• Treat the destination as production. Treat any environment that receives production diagnostic data as production, with the same access reviews, phishing-resistant authentication and monitoring.

The third control is the one most often skipped, because debugging environments are where engineers expect freedom. An agency can keep that freedom by giving engineers synthetic or scrubbed data by default and making access to raw production artifacts a logged, time-limited request approved by the system owner.

Failure two: how far a stolen key reaches

A consumer key should never have signed a token that an enterprise mail service accepted. Microsoft attributed the failure partly to developers assuming that a shared library performed checks it did not perform. A library is a ready-made package of code that developers reuse rather than write from scratch. Libraries save enormous time, but they also invite assumptions about what they do. One key therefore opened doors well beyond its intended scope.

Agencies run many applications that accept tokens from identity providers: a commercial cloud identity service for staff, Login.gov or a state platform for the public, and federation with partner agencies. Each application has to check more than the signature. It must confirm the issuer, the audience, the tenant and the scopes, and reject a token that is genuine but meant for a different application or population.

In wristband terms, the issuer is who printed the wristband; the audience is which venue it is for; the tenant is which organization the person belongs to; and the scopes are which areas they may enter. A token can be perfectly genuine and still be wrong for this door. Libraries differ in what they verify by default, and their documentation is often vague on exactly this point.

The practical fix is to make validation testable. A security test suite for each application can include forged tokens, tokens from the wrong tenant, tokens for another audience and expired tokens, and every one should fail. Development contracts can require that suite and its results as a deliverable at each major release. Code review checklists can name the validation function and the parameters it must receive, so that a reviewer can confirm it in minutes instead of trusting a wrapper.

This matters for program leaders because it converts an invisible property into something a contract can require and a reviewer can confirm. Without it, the agency is relying on each developer’s understanding of a library’s defaults.

Ownership matters here too. When an application team assumes the identity team validates tokens, and the identity team assumes each application does, nobody does. The system security plan for each application should state which component performs each check, and the authorizing official should see that statement before signing an authorization to operate. The authorization to operate is the formal sign-off, by a senior official, that a federal system’s risks are understood and acceptable. It is the natural place to ask who checks what.

Failure three: how long a key stays useful

The key Storm-0558 used had been created years before the intrusion. A long-lived key gives whoever obtains it a long window, and a key nobody tracks may never be revoked. Rotation limits the window. An inventory makes revocation possible at all.

Rotation means replacing a key with a new one on a schedule. Revocation means declaring a key no longer trusted, immediately, because it may have been stolen. Neither is possible for a key nobody knows exists.

Many agencies already track TLS certificates because expired certificates cause visible outages. TLS certificates are what make the padlock appear in a web browser; when one expires, the website shows an alarming warning and the help desk phones ring, so somebody tends to keep a list. Token signing keys, API keys used for service-to-service calls and the certificates trusted for federation rarely get the same treatment. An API key is a password that one piece of software uses to call another, an API being the defined doorway through which programs talk to each other. These keys tend to fail silently, which is exactly why they are forgotten.

A useful inventory records each key’s purpose, the applications that trust it, where it is stored, its owner and its expiry, and it covers keys held by contractors and software-as-a-service providers on the agency’s behalf. Rotation should be scheduled and automated where possible.

The harder test is emergency rotation. If a key is suspected compromised on a Friday night, can the agency replace it and have every dependent application accept the new key without an outage? That procedure needs a written runbook and a rehearsal, ideally in a non-production environment that mirrors the real dependencies. A runbook is a step-by-step checklist for a specific situation, written in advance so nobody has to improvise under pressure. A program director can measure progress with two numbers: the share of signing keys in the inventory with a named owner, and the time taken in the most recent rotation exercise.

One county portal, three gaps

To see how the three failures combine, consider a hypothetical county that runs a resident portal built by a contractor. Residents sign in through a state identity platform to pay property taxes and apply for permits. County staff sign in to a separate case management system through the county’s commercial cloud identity service. The portal also calls the county’s payments system using an API key.

Suppose the portal crashes during a busy tax week. The contractor captures a memory dump and attaches it to a support ticket with a software vendor. Nobody checked whether the dump held the API key or residents’ session tokens. The first gap, then, is key material escaping through routine diagnostics.

Now suppose the case management system, built by a different team, checks that a token is properly signed by a trusted identity provider but does not check the audience or the tenant. A token issued for the resident portal, perfectly genuine, might then be accepted by the staff system. The second gap is a credential reaching far beyond its purpose.

Finally, suppose the payments API key was created when the portal launched and has never been changed, and no one at the county can say which applications rely on it. If it leaked, the county would not know what to revoke or what would break when it did. The third is a key that lives too long and belongs to no one.

No step here involves a sophisticated adversary or an exotic flaw. Each is a reasonable shortcut taken by a busy team. Together they describe a path very like the one in Microsoft’s account. The fixes are equally ordinary: a scrubber on diagnostic data, a validation test suite in the contract, and a key inventory with owners and a rehearsed rotation.

More than engineering detail

These can sound like engineering details, but four reasons put them on a leader’s desk.

Mission and public trust. Government email and citizen-facing systems hold sensitive information about policy, investigations and people. A forged token does not trip the alarms that a stolen password might, because to the system it looks like a legitimate user.

Shared responsibility. Much of an agency’s software is built or run by contractors and software providers. The controls above only happen if contracts require them and someone checks the deliverables.

Accountability. When something goes wrong, oversight bodies, inspectors general and the public will ask who was responsible for each control. A system security plan that names the component performing each check gives a clear answer.

Cost of delay. Building a key inventory and rehearsing rotation is ordinary work in a calm month. Doing it for the first time during an incident is slow, risky and expensive.

Traits of a well-run program

A well-run agency program in this area has a few visible traits. Signing keys live in hardware security modules or managed key services. Diagnostic data is scrubbed automatically, and the scrubber is tested each release. Debugging environments that see production data are governed like production. Each application has a token validation test suite that a contractor delivers and a reviewer reads. Every key appears in an inventory with an owner and an expiry, and the agency knows how long its last emergency rotation exercise took.

The failures are familiar:

• Treating debugging environments as informal. They often hold the most sensitive copies of data with the least protection.
• Trusting the library. Assuming a shared component performs checks it does not is precisely the failure Microsoft described.
• Splitting ownership by assumption. If two teams each think the other validates tokens, neither does.
• Tracking only what breaks loudly. Certificates get inventoried because outages are visible; signing and API keys fail silently and are forgotten.
• Writing the runbook but never rehearsing it. An untested procedure tends to fail at the step nobody remembered.

Where the keys are, and who checks the tokens

If you lead an agency, oversee a technology program or sign authorizations to operate, these are fair questions to put to your chief information officer, chief information security officer and lead contractors:

1. Where do our token signing keys live, and could any of them appear in application memory, and therefore in a crash dump?
2. When diagnostic data leaves production, what removes secrets and personal data from it, and how do we know it works?
3. For each application that accepts sign-ins, which component checks issuer, audience, tenant and scopes, and is that written in the system security plan?
4. Do we have an inventory of signing keys, API keys and federation certificates, including those held by contractors and software providers, with owners and expiry dates?
5. If we suspected a key was stolen tonight, how long would it take to replace it everywhere, and when did we last practice?

With hindsight

The Cyber Safety Review Board’s report in April 2024 concluded that the intrusion was preventable and pointed to a cascade of security failures at Microsoft. That language mattered. It placed the incident in the category of things that sound controls could have stopped, not in the category of unavoidable bad luck.

Before the report was published, Microsoft updated its account to say it had not found a crash dump containing the key, which left the exact path uncertain while leaving the general weaknesses clear. That is a useful reminder about post-mortems in general. The precise route an attacker took may never be fully established, but the weaknesses an investigation uncovers along the way are real whether or not they were the route used. Leaders should act on the weaknesses rather than waiting for certainty about the path.

Microsoft’s Secure Future Initiative, announced in November 2023 and expanded in 2024, committed to moving signing keys into hardware-backed protection, automating rotation and standardizing token validation, close to the agenda above. When the company at the center of the incident chose those three areas for its own program, it was a strong signal that they were the right places to look.

CISA, the federal Cybersecurity and Infrastructure Security Agency, finalized its secure software development attestation form in March 2024 for software sold to federal agencies, which asks producers to attest that they follow secure development practices. Those rules bind vendors. They put the burden on the companies that sell software to the government to vouch for how it was built.

The applications agencies build and commission themselves still depend on the agency’s own key inventory, validation testing and rotation drills. An attestation from a vendor does not cover the case management system an agency’s own team wrote, or the custom portal a contractor built to the agency’s specification. For those, the agency is the producer, and the same discipline it now expects from vendors applies to its own work.

The practical reading is simple. Ask where your keys are, who checks the tokens, and how fast you could replace a key tonight. If the answers are vague, the work is ordinary and well understood, and it is far cheaper to do now than during an incident.