Log4Shell was a search problem before it was a patching problem

The week

Almost every piece of business software keeps a running record of what it does. Each time a customer logs in, a payment is processed or an error occurs, the software writes a short note to a log file so engineers can later work out what happened. Very few organizations write that logging code themselves. They borrow it, for free, from open-source projects: software written and maintained by volunteers and shared publicly for anyone to use. One of the most popular of these for programs written in the Java language is called Apache Log4j.

On December 9, 2021, the world learned that Log4j 2 had a serious flaw. It was a remote code execution vulnerability, meaning an outsider could make someone else’s computer run instructions of the outsider’s choosing, from anywhere on the internet. Proof-of-concept exploit code, a ready-made demonstration of how to carry out the attack, became public at the same time.

The disclosure did not go as planned. According to the federal Cyber Safety Review Board’s later review, a security engineer at Alibaba Cloud had reported the flaw privately to the Apache Software Foundation on November 24. The volunteers who maintain Log4j were still working on a fix when another party disclosed the vulnerability publicly. Defenders and attackers learned about it at roughly the same moment, before an upgrade was generally available.

The flaw itself was old. The Board traced it to a feature accepted into the project in July 2013, before Log4j 2 was generally released. That feature let the library look up values from directory services while writing a log entry. For eight years it sat in thousands of applications without anyone treating it as dangerous.

The trick was startlingly simple. An attacker only had to get a vulnerable system to write a specially crafted string of text into its log. Systems log much of what arrives from the outside world, so that string could be slipped in through ordinary channels. A web request would do, or a field on an online form, or the user-agent header, the small label every web browser sends to identify itself.

The flaw was tracked as CVE-2021-44228, the catalog number the security industry uses for publicly known vulnerabilities, and was soon nicknamed Log4Shell. It received the maximum severity score of 10.0. Within hours, attackers were running mass scans across the internet, knocking on millions of doors to see which ones would open. Early public demonstrations used servers for the game Minecraft, which runs on Java, a sign of how far the library had spread beyond business software.

The Apache Software Foundation released version 2.15.0 on December 10. On the same day, CISA, the federal government’s cybersecurity agency, added the flaw to its catalog of known exploited vulnerabilities, which federal civilian agencies are required to remediate by set deadlines. CISA also began maintaining a public, community-sourced list of affected vendor products.

The first fix proved incomplete. A second vulnerability, CVE-2021-45046, was published on December 14 and later rated critical as well. Version 2.16.0 disabled the risky lookup feature by default. A denial-of-service flaw led to 2.17.0 a few days later, and a fourth, medium-severity issue to another release at the end of December. Separate fixes were issued for older branches still running on Java 6 and Java 7. Early workarounds, such as a configuration flag that disabled lookups, were found not to cover every case and were withdrawn from the recommended list.

On December 17, CISA issued Emergency Directive 22-02, ordering federal civilian agencies to find and patch or mitigate affected systems before Christmas and to report on their progress. The directive did not bind state and local governments or private firms, although CISA urged all organizations to follow the same steps.

Across the industry, software vendors published advisories listing which of their products were affected, often revising them several times as investigations continued. On January 4, 2022, the Federal Trade Commission warned companies that failing to remediate Log4j could lead to enforcement under the FTC Act. It cited the Equifax breach as an example of the cost of leaving a known flaw unpatched. For the people running technology at banks, hospitals and agencies, the search for affected systems ran through the holiday season. It began in the same week that many of them were preparing year-end change freezes, and it ran into the Kronos ransomware outage that struck days later.

Most serious security flaws affect a product you can name: a particular firewall, a mail server, a version of an operating system. When one is announced, an organization asks a simple question, “Do we run that?”, and gets to work. Log4Shell was different, and that difference is the whole story. It lived in a component buried inside thousands of products, many of which did not advertise that they used it.

Some applications included Log4j directly. Others included it inside another library, which was inside a vendor’s application, which ran on a sealed appliance in a server room. The fix, once you found the vulnerable copy, was usually straightforward: an upgrade or a configuration change. Finding every copy was the crisis.

That search ran very differently in three places, each with a different owner and a different source of truth: software an organization builds, software it buys, and software it consumes as a service. We take them in that order, after a first section on what kind of thing Log4j is and why it was so hard to see. Two sections then follow the three places: what the search cost, and the habits that kept it short. A last one asks how long the same search would take today.

Software assembled from borrowed parts

Modern software is assembled more than it is written. A development team building, say, an online banking portal will write the parts that make their product distinctive, such as the screens customers see and the rules for moving money. It then pulls in dozens or hundreds of ready-made building blocks for everything else. These building blocks are called libraries. One library handles dates, another talks to databases, another writes logs. Log4j is a library for writing logs, and it is used by an enormous number of Java programs.

A useful analogy is a car. A carmaker designs the body and the engine, but buys brake pads, wiring harnesses and airbag sensors from suppliers, who in turn buy chips and springs from their own suppliers. If a particular spring turns out to be defective, the hard question is which cars, built in which years, by which makers, contain that spring somewhere inside a part that came from somewhere else. Replacing it is the easy part.

Software has the same layered structure. When a team adds a library to its project, that library often brings its own libraries along with it. These are called transitive dependencies: things you rely on because something you chose relies on them. A team might never have typed the word “Log4j” and still be running it three layers down.

Then there is open source. Log4j is maintained by volunteers under the Apache Software Foundation and is free for anyone to use. This is a genuine good: it means companies do not each have to reinvent basic plumbing. But it also means the code spreads unnoticed, without purchase orders, contracts or invoices that would leave a paper trail. Nobody in procurement ever signed for Log4j.

Who maintains the plumbing

The people behind Log4j were a small group of volunteers working under the Apache Software Foundation’s logging project. They had written a widely used library, given it away, and now had to produce several emergency releases in a few weeks while the world watched. Each release was reviewed and published under intense pressure, which helps explain why the first fix was incomplete.

For buyers, the uncomfortable fact is that a component in their most important systems may depend on a handful of unpaid people. Commercial software vendors that ship open-source components take on responsibility for them in their own products. Organizations that build their own software carry that responsibility directly. Some large users responded after 2021 by funding the projects they depend on or assigning staff to contribute. That is a choice each organization can make about its most critical dependencies, and it is far cheaper than the alternative.

How the flaw worked

Log4j had a helpful feature called message lookups. A log message could contain a short placeholder that told the library to fetch a value and substitute it, such as an environment setting or the name of the server. One kind of lookup used the Java Naming and Directory Interface, or JNDI, a standard way for Java programs to query directory services over the network.

Attackers realized they could write a message whose lookup pointed to a directory server they controlled. The vulnerable application would contact that server, which could answer with a reference to code. In many configurations the application would then load and run it. Since applications log things that arrive from outside, such as the text a user types or the label their browser sends, an attacker could trigger this from the open internet without logging in.

The string did not even have to reach the internet-facing system that received it. Data travels. A customer name typed into a web form might be stored, passed through a message queue and logged hours later by a back-office system with no internet connection of its own. Any Java system in that chain that logged the field with a vulnerable Log4j could be triggered. That made “we patched our internet-facing servers” an incomplete answer from the start.

That combination, trivial to trigger and devastating in effect, is why the flaw received the maximum severity score and why scanning began within hours.

Why it was hard to see

Several technical details made Log4j unusually hard to find, even for teams that went looking.

• Nested packaging. Java applications are distributed as archives, and archives can contain other archives. A web application archive might hold a library archive that holds Log4j. Simple file searches that looked for a file named “log4j” missed copies two or three levels down.
• Repackaged copies. Some products copy library code into their own archives under a different name, a practice called shading. The vulnerable code is present, but the file name gives no hint.
• Two different parts. Log4j 2 ships as separate pieces, an interface and a core. Only the core contained the vulnerable code. Scanners that flagged every piece produced false alarms, and teams spent time chasing them.
• An older generation. Log4j 1.x, a separate older version, was not affected by this particular flaw. It had been declared end of life in 2015, however, and had known weaknesses of its own. Finding it raised a different question rather than closing the case.

Each of these turned a yes-or-no question into a matter of judgment. Every judgment took a skilled person’s time, and there were not many such people to go around in December.

Software an organization builds

For applications developed in-house or by contractors, finding Log4j was possible in principle. Developers use build tools, such as Maven and Gradle, that assemble an application from its code and its libraries. They also keep a record of which libraries each project pulls in.

In practice, many organizations had no central view across their code repositories, the filing systems where software source code is stored. Code lived in many places, owned by many teams. Some older applications had been built by teams that had since moved on, so nobody in the building remembered what was inside them.

Searching repositories and build files found direct dependencies within hours. Transitive dependencies, pulled in by other libraries, took longer. And some applications defeated the search entirely. Once software is compiled and packaged for use, libraries can be bundled inside it, and so-called “fat” archives repackage all their dependencies into one large file. For these, the build record no longer told the full story, and teams had to scan the deployed artifacts themselves: the actual finished files running on servers.

Here the gap between prepared and unprepared organizations was stark. Those with dependency scanning built into their pipelines, the automated assembly lines that test and release software, and with a record of what was deployed where, produced a list within a day or two. Those without relied on developers’ memories and manual searches, which took weeks and were never complete.

Suppose a regional bank runs about two hundred internal applications. At one end of its estate, a modern mobile banking team has automated scanning and can answer the question by lunchtime. At the other end is a loan-servicing tool written by a contractor a decade ago, compiled into a single archive, and running on a server nobody has logged into for months. The bank’s security team cannot declare victory until it has opened that box too, and the only way to do so is to inspect the running file itself.

Triage before patching

With hundreds of candidate systems and a moving fix, teams had to decide what to do first. The sensible order ran from exposure outward. Internet-facing systems that logged request data came first, because they could be hit by mass scanning that day. Systems that logged data originating outside the organization, even indirectly, came next. Internal tools that logged only internal data came last.

That order depended on knowing which systems logged what, which most teams did not have written down. Many fell back on a simpler rule: anything Java that touched the internet first, everything else after. The rule was crude, but it was defensible, and it let work start while the inventory caught up.

Mitigations and their trade-offs

Upgrading was the clean fix, but it was not always possible on day one. An upgrade means rebuilding, testing and redeploying, and some applications could not be rebuilt quickly. Teams reached for interim measures, each with a cost.

• Removing the lookup class from the library file on disk disabled the dangerous feature without a rebuild. It worked, but it was a manual change to a vendor artifact, easily undone by the next deployment.
• Firewall rules that blocked requests containing the attack string helped in the first hours. Attackers quickly found ways to disguise the string, and rule sets became a race.
• Blocking outbound connections from servers to the internet cut off the most common route by which a vulnerable system fetched the attacker’s code. Organizations that already restricted outbound traffic from servers had a strong safety net. Those that allowed servers to reach anywhere did not, and turning on restrictions in a hurry broke legitimate integrations.

The search also had to be repeated. The first fixed release was found to be incomplete in certain configurations, and further issues led to additional releases within weeks. Interim mitigations, such as the configuration flag, were recommended and then superseded. Each new release reset part of the work.

Contractor-built and inherited code

Applications built by contractors raised a separate problem. An agency or bank might own the application, and even hold its source code, while the build environment that produced it remained with a contractor whose engagement had ended. Rebuilding the application with a fixed library then meant recreating that environment, or rehiring the contractor during the busiest week of the year.

Some organizations discovered that they held only the compiled application, with no source code at all. For them, the choices were to modify the deployed archive by hand, to restrict the system’s network access, or to switch it off. Each of those was a risk decision someone senior had to make and record.

The fix is contractual and dull. Statements of work for custom development can require delivery of source code, build scripts and a component list with every release, and can require that the application build in the customer’s own pipeline. Those terms cost little when written at the start and a great deal when missing at the end.

Here the quality of an organization’s engineering machinery showed up as a business outcome. Teams with automated build and deployment pipelines could rebuild and redeploy every affected application several times. Teams deploying by hand could not keep pace, and some ran partly patched estates into January. The difference lay less in the talent of the people than in whether the pipeline could run the same job again, quickly and safely, on short notice.

Software an organization buys

For commercial software, the organization depended on its vendors. Each vendor had to work out whether its products were affected, publish an advisory and deliver a fix. Responses varied widely in speed and clarity. Some published comprehensive product and version lists within days. Others were slow, or issued statements that were vague about specific versions. A statement that a product “is being investigated” could stand for weeks.

Banks, health systems and agencies spent much of December tracking vendor advisories in spreadsheets, matching them to their own inventories and chasing vendors that had not answered. A hospital technology team might have a list of several hundred vendors. It could spend its days sending emails, refreshing support pages and updating a shared spreadsheet column by column, while also running its own patching work.

The exercise exposed how little most buyers knew about the components inside products they owned. Medical devices, building management systems (the software that runs heating, cooling, lifts and door access) and specialized line-of-business applications were hardest, because their vendors were smaller and their update processes slower. Scanning helped at the margins. File-system scanners could find Log4j archives on servers the organization controlled, but not inside sealed appliances, the boxes a vendor ships with software preinstalled and locked down.

Appliances and regulated devices

Sealed products created a particular bind. A hospital cannot patch a medical device itself, because changes to the software can affect its validated function and the manufacturer’s regulatory obligations. A bank cannot modify the operating image of a vendor’s appliance without voiding support. In both cases the buyer had to wait for the vendor, while still carrying the risk.

The available response was compensating controls: measures that reduce exposure without fixing the flaw. Network segmentation could limit which systems a device could talk to. Outbound restrictions could stop it reaching an attacker’s server. Monitoring could watch for the telltale lookup strings in traffic. Health systems that had already segmented clinical devices onto separate networks applied these controls in hours. Those with flat networks had to design segmentation in the middle of an emergency.

Reading a vendor advisory

A useful advisory answered four things for each product: which versions were affected, which were not, what the fix was and when it would be available, and what to do in the meantime. Many early advisories answered only one or two.

Buyers learned to read “not affected” carefully. Sometimes it meant the product did not contain Log4j. Sometimes it meant the product contained a vulnerable copy that could not be reached in its configuration. The second is a reasonable claim, but it depends on analysis the buyer could not check. A format for exactly this kind of statement, the Vulnerability Exploitability eXchange or VEX, was developed in U.S. government-convened working groups around this time. It lets a supplier say, in machine-readable form, whether a known flaw in a component actually affects a product.

Holding vendors to dates

Chasing vendors worked better with structure. Some organizations ranked vendors by criticality, chased the top tier daily through named contacts and escalated to an executive sponsor after a set number of days. They got answers faster than those that emailed everyone equally.

Where a vendor could not give a fix date, the buyer still needed a record. A short risk acceptance, signed by the system’s business owner, named the compensating controls in place and set an expiry date for review. That record mattered later. Examiners and auditors asking about Log4j wanted to see that a gap had been noticed, assessed and owned. An unexplained gap in a spreadsheet tells them none of that.

The same discipline applied in reverse. Banks and software companies that were themselves vendors to others received hundreds of questionnaires. Those that published a single, dated public statement, and pointed every inquiry to it, saved their own staff a great deal of time.

The case for component lists

The episode moved the software bill of materials, or SBOM, from niche idea to procurement topic. An SBOM is a machine-readable list of the components in a piece of software. If every product had shipped with an accurate one, the question of where Log4j was could have been answered by a search rather than a survey.

The May 2021 executive order on cybersecurity had already called for SBOMs in federal procurement, and the Commerce Department had published minimum elements for them that July. Log4Shell supplied the example that made the case to everyone else. But an SBOM does not fix anything by itself. It is useful only if it is accurate, if it is updated when the software changes, and if someone can check every SBOM on file quickly when a new vulnerability is announced. That takes a system that can match the two.

Software an organization consumes as a service

Much of today’s software is rented and used over the internet rather than installed: email, customer relationship management, payroll, and the computing platforms themselves. This is software as a service and, more broadly, cloud computing, where a provider runs the machines and the customer uses them.

For these services, the provider was responsible for fixing its own systems. That was a real advantage, because customers had nothing to patch. But customers still needed assurance that providers had acted, and some had to consider whether their data had been exposed before fixes went in. Many providers published status statements within days. Few told individual customers whether exploitation had been attempted against their tenant, meaning that customer’s own walled-off portion of a shared service, or what logs the customer could review to check.

Where the provider’s job ends

The boundary of responsibility needed care. Cloud services come in layers. With infrastructure services, the provider rents out raw computing power, such as virtual machines and containers, which are software-defined computers and lightweight packaged environments. The provider patched the platform, but Java applications the customer ran on those machines were still the customer’s to find and fix. Platform services were in between: a managed runtime might be patched by the provider while libraries packaged inside the customer’s own application were not.

Teams that assumed “it’s in the cloud, so the provider handles it” missed workloads that were plainly their own. A shared-responsibility matrix, a simple table showing who does what, kept per service rather than per provider, answers that question before an incident rather than during one.

Container images added a twist. Teams often build their applications on top of base images published by others, which bring their own software. A container built months earlier from an image that included a vulnerable Java application kept that copy until someone rebuilt it, even if the original image had since been fixed. Registries full of old images were another place to search.

The contract gap

The gap was contractual as much as technical. Most service agreements committed providers to notify customers of confirmed breaches of customer data. Few committed them to tell customers, within a set time, whether a widely exploited vulnerability had affected the service and what had been done. Buyers who added that obligation at renewal, along with rights to relevant logs, had better answers the next time. The trade-off was negotiating capital. Large providers resist bespoke terms, so the request works best when buyers in a sector ask for the same thing.

Regulated buyers had particular reasons to press. Banks are expected by their supervisors to oversee critical service providers, and an examiner may ask what the bank knew about a provider’s exposure and when. Health systems rely on business associate agreements under HIPAA, which oblige a vendor handling protected health information to report breaches of it. Those agreements address confirmed breaches, not exposure to a vulnerability, so a provider could be silent on Log4j and still be within the contract. Federal agencies had a further channel through the government’s cloud authorization program, but county and city governments generally relied on their contracts alone.

Providers behind the provider

Service providers run on other services. A payroll platform may use a third party for logging, another for customer support chat and a cloud provider underneath it all. Each of those could have been exposed, and the customer had no direct relationship with any of them.

Banking supervisors and many vendor risk programs call these fourth parties. Log4Shell showed why they matter: a provider could truthfully say its own code was patched while a subcontractor that handled the customer’s data was still investigating. Useful contract language asks the provider to cover its material subcontractors in any vulnerability statement, and to name them on request. The provider will resist naming all of them. A list of the ones that process customer data is a reasonable middle ground.

What the search cost

Log4Shell revealed a good deal about the organization as a whole, well outside the IT department.

Money. For most organizations, the cost came as weeks of skilled staff time spent searching, chasing and re-patching, rather than as a ransom or a fine. Much of it fell during the holidays, and much of it was avoidable with better records. The Cyber Safety Review Board reported that one federal cabinet department alone dedicated 33,000 hours to its response. Every hour spent building a spreadsheet of vendor responses was an hour not spent on planned work.

People. The Board also noted burnout among defenders who worked through weekends and holidays, and confusion and “patching fatigue” as one fix followed another. The cost of an emergency like this falls on a small number of experienced people, and some of them leave afterward.

Risk. The time between disclosure and the first mass scans was measured in hours. An organization that needed weeks to know where it was exposed was, in effect, betting that attackers would get to it last.

Customers and patients. The systems at stake included the ones customers touch directly, such as web portals, and the ones that keep services running behind the scenes, such as clinical devices and building controls. Exposure in either could mean disruption or data loss.

Regulators. Federal agencies received a directive with deadlines. The FTC told companies it would use its authority against those that failed to protect consumer data from Log4j. Regulated firms in finance and healthcare could expect supervisors and auditors to ask how quickly they had established their exposure. “We are still checking” is an uncomfortable answer to give a regulator twice.

Reputation. Customers and partners asked the same question organizations were asking their own vendors: “Are you affected?” A clear, prompt answer built trust. A vague one invited doubt. Banks and software vendors that published a dated statement early, then updated it as facts changed, spent less time answering individual inquiries.

How the cost differed by sector

Federal civilian agencies had the clearest instructions and the least discretion. The emergency directive set deadlines and reporting formats, and agencies had to meet them whatever their inventory looked like. The burden was heavy, but the priorities were decided for them.

State and local governments had no such directive. Many relied on small IT teams, on state-level support and on the Multi-State Information Sharing and Analysis Center for advisories. Their exposure was often concentrated in vendor-supplied systems for permitting, courts and public safety, where they depended entirely on the vendor.

Banks generally had larger security teams and established vulnerability management programs. Their difficulty was scale: thousands of applications and hundreds of vendors, many of them technology service providers that serve the whole industry. Health systems faced the device problem in its sharpest form, with clinical equipment that could not be patched by the hospital and could not be taken offline without affecting care.

Why the calendar made it worse

The timing compounded every cost. Many banks, retailers and agencies impose change freezes in December to protect year-end processing and holiday trading. Emergency changes are still possible, but they need extra approvals, and change advisory boards meet less often when senior staff are on leave.

Organizations with a fast, well-understood emergency change process could push fixes through the freeze with a documented risk decision. Those whose emergency process was rarely used found that nobody was sure who could approve what. The vulnerability did not respect the freeze, and neither did the attackers. A quick rehearsal of the emergency change path, before the next December, costs very little.

Habits that kept the search short

Organizations that came through Log4Shell well tended to share a handful of habits. The Cyber Safety Review Board reached a similar conclusion: the most effective responders understood their use of Log4j and had mature processes to manage assets, assess risk and mobilize partners. None of the habits is exotic.

• An inventory that reaches below the product name. Beyond “we run this application”: “this application contains these components, at these versions, on these servers.”
• Scanning of what is actually running as well as source code, so that bundled libraries and old archives are visible.
• Automated pipelines that can rebuild and redeploy every in-house application several times in a week without heroics.
• A vendor register with named contacts and a habit of asking for component lists at purchase, not during an emergency.
• A shared-responsibility matrix per cloud service, so nobody has to debate during an incident whose job a given fix is.
• Restricted outbound traffic from servers, so that a vulnerable system cannot easily fetch an attacker’s code.
• Contract terms that require service providers to report on widely exploited flaws within a set time and give access to relevant logs.

The failures were mirror images of these.

• Treating the first patch as the end. Several releases followed the first one. Organizations that stood down their response team after round one had to reassemble it.
• Searching only source code. The build record is not the deployed application. Compiled and repackaged applications hid their contents from simple searches.
• Assuming the cloud provider covers everything. Applications a customer runs on rented infrastructure remain that customer’s responsibility.
• Accepting “under investigation” indefinitely. Vendor statements without dates or versions need follow-up, escalation and, if necessary, compensating controls, such as limiting what the affected system can reach on the network.
• Collecting SBOMs and never reading them. A drawer full of SBOMs is no help unless something can search them all at once when the next flaw appears.

Who owns which piece

Each habit needs an owner, or it decays. The application security team, or whoever runs the build pipeline, owns component lists for in-house software and the scans of running systems. Infrastructure owns outbound traffic rules. Procurement and vendor risk own the contract terms and the collection of supplier SBOMs. The configuration management team owns the map from applications to servers. The chief information security officer owns the response playbook that ties them together.

The measures that matter are few. One is the time it takes to say whether a named component is present. Others are the share of in-house applications that generate a component list on every build, and the share of critical vendors that have supplied a current one. Those numbers can be reported quarterly, and they move only when the underlying habits change.

The case against deep inventory

The strongest objection is cost. Component-level inventory, runtime scanning and SBOM management take tools, staff and attention every week, to prepare for an event that may come once every few years. A mid-sized credit union or county may reasonably ask whether its money is better spent elsewhere.

A second objection comes from the Board’s own finding. Exploitation of Log4j turned out lower than many experts predicted, and the Board was not aware of significant Log4j-based attacks on critical infrastructure when it reported. On that reading, the scramble was an overreaction.

Neither objection holds up well. The lower exploitation followed one of the most intense defensive responses in recent memory. It is weak evidence that the response was unnecessary. Inventory also pays for itself in less visible ways, through license compliance, end-of-life tracking and faster answers to auditors. The cost objection does argue for proportion: a small organization should buy these capabilities inside its platforms and contracts rather than build them.

Sequencing the investment

Not every organization can build all of this at once, and the order matters. The cheapest step with the largest payoff is usually scanning the servers the organization controls, including inside nested archives, so that it can see what is actually running. Generating component lists in build pipelines comes next, since open-source tools for this are widely available and the work is mostly configuration.

Restricting outbound traffic from servers takes longer, because it requires knowing which legitimate connections each system needs. Contract terms come at renewal, which sets their pace. A system to ingest and search supplier SBOMs belongs at the end of the list, once there is something worth searching.

A large bank may run all five tracks in parallel. A credit union or a small county may rely on its managed security provider for scanning and on its core or platform vendors for component lists. It may get standard contract clauses through a trade association or a purchasing cooperative. The capability is the same. The route to it differs with scale.

Suppose a county government runs a mix of in-house systems, a commercial permitting package and several cloud services. Before Log4Shell, its records listed applications by name and owner. Afterward, it adds a component list for each in-house application, generated automatically each time the software is built, and begins requesting SBOMs in new contracts. When the next widely publicized flaw arrives, its first report to the county administrator can say which systems are affected, which vendors have answered and which have not, rather than simply “we are looking.”

How long the search would take now

These questions are deliberately plain. A confident, specific answer is a good sign. A long pause is useful information too.

1. If a flaw like Log4Shell were announced tomorrow morning, could we produce a list of every affected system by the end of the day, and who would produce it?
2. Do we know which open-source components are inside the applications we build, including the ones pulled in by other components?
3. Do we scan what is actually deployed and running, or only our source code?
4. For our most important purchased products, do we have a software bill of materials, and do we have any way to search across them?
5. Do our service contracts require providers to tell us, within a set time, whether a widely exploited flaw affected us, and do they give us access to the relevant logs?

The answers are best tested rather than asserted. A tabletop exercise can name a widely used component, declare a flaw in it, and time how long each team takes to produce its list. The exercise costs a morning. It shows which records are current, which vendors answer, and which systems nobody can vouch for.

The result belongs in the regular risk report to the board or its risk committee, expressed as time: how many hours to a complete list, against a target the committee has agreed. Directors do not need to understand Java archives to see whether an answer that took three weeks in 2021 now takes a day.

Log4Shell was easy to exploit and, in most places, easy to fix. It was hard because so many organizations could not see inside their own software. That is a solvable problem: component lists, scans of running software and contract terms can all be in place before the next emergency.

With hindsight

Log4Shell did not disappear after December 2021. The federal Cyber Safety Review Board, which examines major cyber incidents, reported in July 2022 that Log4j was an “endemic vulnerability.” It expected vulnerable instances to remain in systems for many years, perhaps a decade or longer. The same report found that exploitation in the first months had been lower than many experts predicted. Both findings have held up.

Exploitation continued well beyond the first wave. Log4Shell appeared on the joint lists of routinely exploited vulnerabilities that CISA and allied agencies published for 2021 and for 2022. In November 2022, CISA and the FBI reported that Iranian government-sponsored actors had compromised a federal civilian agency through an unpatched VMware Horizon server, exploiting Log4Shell to install cryptocurrency mining software and steal credentials. That was nearly a year after the emergency directive. A famous vulnerability is not over when the headlines stop. Forgotten servers and vendor products that never received an update keep old flaws alive long after most organizations have moved on.

The policy response was substantial, and it pointed in a consistent direction: buyers and regulators want to know what is inside the software they rely on. In January 2022 the White House convened government and industry on open-source security. OMB guidance issued in September 2022 required software producers selling to agencies to attest to secure development practices, with a common attestation form in use from 2024.

Sector rules followed. Congress required medical device makers to include cybersecurity information, including a software bill of materials, in premarket submissions from March 2023. Version 4.0 of the PCI Data Security Standard applies to organizations that store, process or transmit payment card data. It added a requirement to inventory bespoke and custom software and the third-party components in it. That requirement became mandatory in 2025. The EU’s Cyber Resilience Act entered into force in December 2024. Its vulnerability reporting obligations for manufacturers apply from September 2026, and most of its other requirements from December 2027.

CISA pressed from another direction. In April 2023 it published, with partner agencies from several countries, principles for products that are secure by design and by default, urging manufacturers to take ownership of their customers’ security outcomes. In May 2024 it launched a voluntary Secure by Design pledge for software makers. Neither carried legal force, but both moved expectations toward suppliers.

Together these changes shift some of the burden of the December 2021 search from buyers to the people who make software. A vendor that must list its components, and attest to how it builds them, is in a much better position to answer “Are we affected?” quickly, and so is its customer.

The next major open-source scare tested the new habits from a different angle. In March 2024, an engineer investigating a small performance anomaly discovered a deliberately planted backdoor in xz Utils, a compression library included in many Linux distributions. The malicious versions had reached only some test and pre-release distributions before discovery. Organizations with component inventories confirmed within hours that their production systems were unaffected. Those without repeated, on a smaller scale, the December 2021 survey. The episode also showed a risk that inventory alone does not address: a lone, overstretched maintainer can be targeted over years by someone posing as a helper.

By this point in 2026, SBOMs are routinely requested in regulated procurement, though their quality and usefulness still vary. Some are thorough and kept current. Others are a one-time document produced to satisfy a contract and never updated. Receiving an SBOM is different from being able to use one, and many buyers still lack a system that matches a new vulnerability against every SBOM on file.

Inventory that reaches below the product name remains the precondition for every fix. The organizations that will handle the next Log4Shell well are the ones that can search their own estate, and their suppliers’ declarations, in hours. That capability is built in the months between crises, through procurement terms, automated pipelines and records someone keeps up to date.