The tool that checked for supply-chain risk was the supply-chain risk

The week

In the second half of March 2026, a group tracked as TeamPCP ran a cascading software supply-chain campaign, and its first weapon was a security tool. A software supply chain is the long line of outside components, tools and services that go into building a modern application; most organizations write only a small fraction of the code they run. Attacking the supply chain means poisoning one of those components so that it reaches everyone who uses it.

On March 19, attackers published a malicious release of Trivy, a widely used open-source vulnerability scanner. A vulnerability scanner is a security tool that inspects software for known weaknesses; “open-source” means its code is public and free to use. The attackers also repointed nearly all version tags of its official GitHub Action to credential-stealing code. A GitHub Action is a ready-made step that development teams plug into their automated build process, and a version tag is the label they use to say which version they want. Because the labels had moved, teams asking for a familiar version silently received the attackers’ code instead, code designed to steal passwords and digital keys.

Aqua Security, which maintains Trivy, published safe versions on March 20 and a formal advisory, CVE-2026-33634, on March 21. A CVE is a public, numbered entry in the global catalog of known security flaws. Researchers later traced the compromise to credentials stolen in late February and only partly rotated: after an earlier break-in, not every key had been changed.

The campaign then moved from tool to tool. On March 23, the same group poisoned Checkmarx’s KICS GitHub Action through a compromised publishing account. On March 24, malicious versions 1.82.7 and 1.82.8 of LiteLLM appeared on PyPI. LiteLLM is a popular library, a reusable building block of code, that routes requests to many AI model providers. PyPI is the main public warehouse from which programmers using the Python language download such building blocks.

LiteLLM said the compromise was likely linked to the Trivy scanner in its own build pipeline, which would mean the poisoned security tool had helped the attackers poison the next product down the line. PyPI quarantined the malicious versions within about 40 minutes.

The irony is easy to state. Teams add scanners to their build pipelines to catch dangerous dependencies, and here the scanner was the dangerous dependency. The more useful observation is about credentials. Every hop in the campaign used a secret stolen at the previous hop, and the campaign began with a secret that should already have been dead.

So we look at exposure in three parts, following the life of a stolen credential. First, the tools that could read it. Second, the systems that pooled many credentials in one place. Third, the time each one stayed valid after theft. An organization’s risk in March was the product of all three.

The framing turns a bewildering technical story into three things any organization can check about itself. First, though, a short tour of the machinery.

Pipelines, tokens and the labels that can move

Dependencies. Modern software is assembled more than it is written. A bank’s mobile app or a hospital’s patient portal may rely on hundreds of outside components, many of them open-source libraries maintained by small teams or volunteers. Each component the software relies on is a dependency. If one is poisoned, everything built on it can be too.

Build pipelines. When a developer finishes a change, it does not go straight to customers. It goes through a build pipeline, an automated assembly line that gathers the code and its dependencies, runs tests and security scans, packages the result and ships it. Pipelines run hundreds of times a day in a large organization, unattended. To do their job they are given powerful access: to cloud accounts, to the systems that store finished software, and to the places where it is published.

Credentials and secrets. That access is granted through credentials, often called secrets: passwords, tokens and keys that prove to another system that the pipeline is allowed in. A token is a long string of characters, and whoever holds it is treated as the authorized user, no questions asked. An API key does the same job for a specific service, such as an AI model provider; an API (application programming interface) is the standard doorway through which one program asks another for something.

Tags and hashes. When a pipeline uses an outside step, it names which version it wants. It can do that with a tag, a friendly label like “version 2,” or with a commit hash, a long fingerprint that identifies one exact, unchangeable snapshot of the code. The difference decided who was exposed. A tag works like a sign on a shelf that the shop owner can move to a different box, while a hash is the box’s serial number: order by serial number and nobody can swap the box.

AI gateways. As organizations started using several AI models, many adopted a gateway: a single switchboard through which their applications send requests to whichever model provider is wanted. LiteLLM is one such switchboard. It holds the keys for every provider it connects to.

With that vocabulary, the campaign reads like a chain of stolen keys, each opening the next door.

How the campaign hopped from tool to tool

Laid out as a sequence, the campaign shows why a credential-centered view is the right one.

1. The first key. In late February, credentials connected to Trivy were stolen. The intrusion was detected, but the secrets were only partly rotated, so some stolen keys kept working.
2. The poisoned scanner. On March 19, using that access, the attackers published a malicious Trivy release and moved the action’s version tags to credential-stealing code. Every pipeline that ran the scanner by tag now handed its secrets to the attackers.
3. More doors. On March 23, the group poisoned Checkmarx’s KICS GitHub Action through a compromised publishing account, another security tool trusted inside pipelines.
4. The gateway. On March 24, two malicious LiteLLM versions appeared on PyPI. LiteLLM said the compromise was likely linked to the Trivy scanner in its own build pipeline. Those versions went after the keys that AI gateways pool.

At no point did the attackers need to defeat a firewall or guess a password. Each step used a key stolen in the step before. Stop any one theft, or make any one stolen key useless quickly, and the chain breaks. Which is why who can read keys, where they are pooled and how long they last matter more than the details of any one tool.

Tools that could read the keys

Vulnerability scanners run in the most sensitive part of the software process. To scan a repository or container image, they run inside build jobs that often hold cloud credentials, signing keys, registry passwords and package-publishing tokens. (A container image is a packaged, ready-to-run copy of an application; a signing key is what proves that software really came from its publisher.) Most teams add a scanner by referencing a public action by name and version tag, then never revisit the decision.

A version tag is a pointer the publisher can move. When the attackers force-pushed the Trivy action’s tags, every pipeline that referenced a tag pulled the new code on its next run, with all the access that job had been granted. Pipelines that referenced the action by full commit hash kept running the old, clean code.

Three controls follow, and none is exotic. Reference third-party actions and images by immutable hash, and update them deliberately. Give each pipeline step only the secrets it needs, so a scanning step never sees a publishing token. And restrict outbound network access from build runners to known registries and code hosts. A build runner is the computer that actually executes the pipeline, and outbound access, often called egress, is its ability to send data out to the internet.

The reported payloads sent stolen data to look-alike domains, web addresses designed to resemble legitimate ones at a glance. Egress rules would have blocked that, and alerting on denied connections would have turned a silent theft into a visible event.

Security tools deserve particular scrutiny because organizations grant them broad read access by design and exempt them from the suspicion applied to other code. A security tool in a pipeline should be inventoried, owned and reviewed like any privileged service account.

Systems that pooled the keys

LiteLLM is popular because it solves a real problem. It gives applications one interface to many model providers and manages the keys for each. That convenience makes a gateway host unusually rich. It may hold API keys for several AI providers, cloud credentials, database connection strings and the prompts and responses that pass through it.

Researchers reported that the malicious versions targeted exactly that: provider keys, cloud and Kubernetes credentials, SSH keys and environment files. Kubernetes is widely used software for running applications across fleets of servers; SSH keys let administrators log into servers remotely; environment files are configuration files where developers often store secrets in readable text for convenience.

For a bank, health system or agency, the data side matters as much as the keys. Prompts can contain customer records, clinical notes or case files. Yet AI gateways were often installed by a development team as a library, outside the controls applied to an identity provider or payment switch. The gateway had turned into tier-one infrastructure without anyone deciding it should. (“Tier-one” is the label organizations give their most critical systems, the ones that get the strictest controls and the closest monitoring.)

Governing it as tier-one infrastructure starts with a named owner and a hardened, pinned build. Secrets should be fetched from a vault at runtime rather than read from environment files. A vault is a dedicated, tightly controlled system for storing secrets, which hands them out only when needed and logs every request. Egress should be limited to approved model endpoints, and a security team should review the logs.

LiteLLM said users of its official proxy container, which pinned its dependencies, were not affected, a direct argument for consuming such components as pinned, maintained builds rather than installing the latest version on demand.

How long stolen keys stayed valid

The campaign was possible because a credential stolen in late February still worked in March. Researchers reported that the maintainers detected the earlier intrusion but rotated secrets only partly. Rotating a secret means issuing a new one and cancelling the old, much as a building rekeys its locks after a lost key. Rotating only some of them leaves the back door on the old key.

The same pattern repeated downstream. LiteLLM traced its incident to the scanner in its own pipeline, where the means to publish to PyPI were evidently within reach and still valid long enough to push two malicious releases.

Two practices shorten that window. On the publisher side, package registries now support short-lived, workflow-bound publishing credentials, often called trusted publishing, which removes standing tokens from build environments. Instead of a permanent key left lying in the pipeline, the pipeline receives a one-time pass that works for a single approved job and then expires. On the consumer side, a secrets inventory that records which pipelines can reach which credentials makes rotation targeted and complete. Organizations that had one could rotate the right keys within hours. Those without one faced a choice between rotating everything, which is slow and disruptive, and guessing, which is how the original incomplete rotation happened.

Rotation also needs verification. After an incident, someone should confirm that old credentials are actually rejected; issuing new ones does not prove that. Lock files, hash checking and an internal package mirror that delays brand-new releases by a short period add a further buffer. A lock file records exactly which version of each dependency was approved, so a build cannot silently pick up a newer one. An internal mirror is the organization’s own copy of the public warehouse, which can hold new arrivals in quarantine for a while before releasing them to developers. Many organizations would never have installed a version that was public for less than an hour.

The response week also tested a less visible capability: build records. To know whether it was exposed, an organization had to show which pipelines ran an affected action or installed an affected version between March 19 and the cleanup. That requires build logs and dependency resolution records kept long enough to search. Teams that retained them answered in an afternoon. Teams that kept a few days of logs had to assume exposure and rotate on that basis.

One insurer, two possible Tuesdays

Suppose a regional insurer has spent the past year building an AI assistant for its claims handlers. Its developers use Trivy in their pipelines, referenced by tag, and run LiteLLM as a library inside the assistant to switch between two AI providers. The library pulls its keys from an environment file.

When the advisories appear, the chief information security officer wants three answers, matching the three parts above: which pipelines ran the affected scanner and what secrets those jobs could see; whether any system installed the affected LiteLLM versions and what keys sat on those hosts; and, for every key in either place, whether the old version is now rejected.

The answers depend almost entirely on preparation. If the insurer keeps a month of build logs and a secrets inventory, the security team can list the affected pipelines by lunchtime, identify a dozen keys, rotate them, and confirm by evening that the old ones fail. If it keeps a few days of logs and no inventory, it must assume every pipeline and every key is exposed, rotate hundreds of secrets across cloud accounts and AI providers, and accept days of disruption to development, with lingering doubt about whether it found them all. Meanwhile the claims prompts that passed through the gateway, which may include policyholder details, raise a separate question for the privacy team.

The incident is the same in both versions; the week is not. The difference was decided months earlier, by choices nobody outside engineering had noticed.

What a stolen key costs outside engineering

Risk. Stolen cloud and AI keys can be used to run up bills, reach data or launch further attacks. Because each hop in this campaign used keys stolen at the previous one, a single unrotated secret can expose partners and customers as well as the organization itself.

Customers and regulators. AI gateways carry prompts, and prompts carry customer, patient and case data. A compromise of the gateway can become a data incident with notification duties, even if no database was touched.

Money and productivity. Blanket rotation is expensive. Every key changed means systems reconfigured and tested, and developers diverted from planned work.

Governance. The deepest issue is that critical infrastructure arrived without a decision. An AI gateway installed as a convenience by one team became the place where an organization’s most valuable keys and data met, and nobody can govern a system they do not know they have.

The habits that broke the chain

The organizations that came through March easily had pipelines that used outside steps by fixed fingerprint, gave each step only the secrets it needed and could talk only to approved destinations. They treated AI gateways as critical systems, with owners, vaulted secrets and limited egress. They used short-lived publishing credentials, kept a secrets inventory, and held build records long enough to settle the exposure question in an afternoon.

The weaker ones tended to share a handful of habits:

• Trusting security tools by default. Giving scanners broad access and never reviewing them.
• Ordering by label rather than serial number. Referencing outside components by movable tags.
• One big key ring. Letting every pipeline step see every secret.
• Partial rotation. Changing some keys after an incident and assuming the job is done.
• Short memory. Keeping build logs for days, so exposure cannot be ruled out.
• Accidental infrastructure. Letting an AI gateway become critical without anyone deciding it should.

Where the exposure sits now

1. Which security tools run inside our build pipelines, who owns each one, and are they referenced by fixed fingerprint or by movable label?
2. Can our build machines send data anywhere on the internet, or only to approved destinations?
3. Where do we run AI gateways, what keys and data pass through them, and who owns them?
4. After our last rotation, did anyone confirm the old keys stopped working?
5. If we needed to know tomorrow whether a pipeline ran a poisoned component three weeks ago, could we?

With hindsight

The main packages were cleaned up within days, but the campaign did not stop with them. Researchers documented further waves, including malicious releases of a communications provider’s Python SDK on March 27. An SDK, or software development kit, is a bundle of code a company provides so that customers’ programmers can connect to its service easily, which makes it another natural carrier for a supply-chain attack.

CISA added the Trivy vulnerability to its Known Exploited Vulnerabilities catalog on March 26. That catalog is the federal government’s list of flaws confirmed to be used by attackers in the real world, and federal civilian agencies are expected to fix listed flaws on a deadline. Its inclusion signaled that the risk was far from theoretical.

Later research notes reported that credentials harvested in the campaign were being used by a ransomware operation, a reminder that stolen secrets have a long shelf life. A key stolen in March can be sold, shared and used months later by a different group with different aims. Rotation therefore has to be complete and verified as well as prompt.

Published accounts differed on exact times and on the number of organizations exposed, which is typical of fast-moving incidents, and we would not rely on any single estimate.

What held up is the credential-centered view. Organizations that pinned actions by hash, used short-lived publishing credentials and knew which secrets each pipeline could reach read about the campaign. Others spent weeks rotating keys they could not fully enumerate.

The software assembly line is now part of the organization’s attack surface, and the tools inside it, including the security tools, deserve the same scrutiny as any other privileged system. As AI gateways spread, the question of where keys and sensitive prompts are pooled will only become more pressing. Knowing the answer before an incident is the cheapest protection available.