The control system map most operators did not have

The week

In late July 2020, two of the most senior security agencies in the United States warned that a class of computers most executives rarely consider was increasingly in attackers’ sights. Cybersecurity, to most boards, means email, laptops and customer data. Some of the most consequential computers in any organization never show up on a desk. They open valves at the water plant, keep operating rooms at the right temperature and stop a data center from overheating.

The National Security Agency and the Cybersecurity and Infrastructure Security Agency, known as CISA, issued a joint alert, dated July 23, 2020, recommending immediate actions to reduce exposure across operational technology and control systems. Operational technology, often shortened to OT, is the hardware and software that monitors and controls physical equipment: pumps, motors, valves, boilers, chillers, generators. Information technology, or IT, handles information; OT handles the physical world.

The alert was addressed to the Department of Defense, national security systems, the defense industrial base and US critical infrastructure operators. It said internet-accessible control system assets were increasing and being targeted, and that the security of legacy control systems had not kept pace. Many of these systems were designed decades ago for closed, trusted environments, long before anyone imagined connecting them to the internet.

The alert did not come out of nowhere. In February 2020, CISA had described a ransomware attack on a natural gas compression facility. The attacker used a spearphishing link to get into the company’s IT network, then moved into its OT network and deployed ransomware on both. The operator had not planned for that kind of event, and it shut down the pipeline’s operations for about two days. The July alert generalized the pattern.

It also pointed to history. It cited the 2015 attack on Ukraine’s power grid as the reason operators should plan for control systems that act against safe operation, as well as for systems that stop. It referred to reported attempts against Israeli water systems in the spring of 2020 as evidence that attackers were interested in water.

It described attacker techniques ranging from spearphishing and ransomware to direct manipulation of exposed devices. Spearphishing is a targeted email crafted to trick a specific person into clicking a link or opening an attachment. Ransomware is malicious software that locks up an organization’s computers and demands payment to unlock them. Direct manipulation is the most alarming: reaching a control device over the internet and changing what it does.

The alert was specific about how that happened. Attackers had connected to internet-accessible programmable logic controllers that required no authentication at all. They used commonly used ports and standard protocols to talk to the controllers, used vendor engineering software to download modified control logic, and changed both logic and parameters. In some cases ransomware encrypted systems on the OT network as well as the IT network.

It listed the impacts plainly. Operators lost availability of systems, lost part of their view of the process on their screens, lost productivity and revenue, and in the worst cases saw an attacker manipulate control of a physical process. It noted that search engines for internet-connected devices, such as Shodan, made exposed controllers easy to find. Widely available penetration testing frameworks also included modules aimed at control systems.

Recommendations included building an accurate map of control networks as actually operated, reducing internet exposure, having a resilience plan for operating without control systems, and exercising incident response. They sound modest. For many organizations, the first of them, simply knowing what was connected to what, turned out to be the hardest.

The detail under each heading was practical. Operators were told to keep “gold copy” backups of firmware, software, control logic, licenses, keys and configurations, stored off the network with at least one copy in a locked, tamperproof place. Tabletop exercises were to include executives, public affairs and legal staff, and to identify key decision points and who held authority over each. The network map was to be validated with passive tools and physical walkdowns. Monitoring was to include checking controller logic against a known-good baseline, and limiting set points so an intruder could do less harm.

Operational technology is usually seen as a concern for utilities and manufacturers. Banks, health systems and public agencies run it too: municipal water and wastewater treatment, hospital building automation, medical gas and clinical engineering systems, and the power and cooling behind every data center. If you own or rent a building, you almost certainly depend on operational technology, whether or not anyone in your organization calls it that.

The July 2020 alert set out a sequence of work that applies to all of them. First, map what is actually connected. Second, decide what should be connected and protect it. Third, be able to operate when the control systems cannot be trusted. Each step depends on the one before, and the first is where most operators were weakest.

The alert was addressed to defense and critical infrastructure operators, and its advice is general. How it lands differs by type of organization. A water utility is a regulated critical infrastructure operator in its own right. A hospital is a critical infrastructure operator whose OT mostly keeps buildings and clinical spaces running. A bank or a federal agency is often a tenant, depending on OT that a landlord or provider runs. The three steps apply to all of them, but who does the work, and under which rules, varies a great deal.

Sensors, controllers and screens

Every control system has a few common parts. At the bottom are sensors and actuators: devices that measure things, such as temperature, pressure or flow, and devices that do things, such as opening a valve or starting a pump. Above them sit controllers, small rugged computers that read the sensors and decide what the actuators should do. The most common kind is called a programmable logic controller, or PLC. A PLC follows rules such as these: if the tank level drops below a line, start the pump; if the pressure rises too high, open the relief valve.

Above the controllers are the screens people use to watch and adjust the process, called human-machine interfaces, or HMIs. An operator at a water plant or a hospital facilities engineer looks at an HMI to see what is happening and change settings. Around all of this sits a network that connects the pieces, and increasingly, links that connect that network to the outside world: to the corporate network, to vendors who provide remote support, and sometimes, directly to the internet.

Two features make these systems different from office IT. First, they run for a very long time. A controller installed fifteen or twenty years ago may still be running the same process today, with software, called firmware, that was never designed to resist attack. Second, they control physical things. If an email server fails, people cannot send email. If a chiller controller fails in a data center or a hospital, equipment overheats, procedures are cancelled or, in the worst case, people are harmed. That is why security teams cannot simply apply office IT habits to OT, and why the people who run the equipment are often wary of security changes.

How attackers actually reach a controller

The alert described two routes, and they need different defenses. The first starts in IT. An attacker sends a spearphishing email, gains a foothold on an office computer and then moves across the network until it finds a path into the control environment. That path is often an engineering workstation, the computer that runs the vendor’s programming software and can rewrite controller logic. Engineering workstations frequently connect to both networks, which makes them the bridge an attacker wants.

The second route skips IT entirely. An attacker searches the internet for exposed controllers or HMIs, connects and issues commands. This works because many industrial protocols were designed without authentication. Modbus, one of the oldest and most widely used, accepts commands from anyone who can reach the device. The protocol has no concept of a password. Security for these devices depended on the network around them, and when that network was opened to the internet, nothing was left.

A third pattern produces physical effects without touching a controller. Ransomware that encrypts the HMIs and servers on an OT network leaves the controllers running but blinds the operators. The alert called this a loss of view. A plant whose operators cannot see its process will often shut down deliberately, because running blind is unsafe. The February 2020 gas compression case worked that way.

The layered model, and where it breaks

Engineers often describe industrial networks in layers. Field devices sit at the bottom, then controllers, then supervisory systems such as HMIs and historians, then a boundary zone, then the business network and the internet. A historian is a database that records process readings over time, used for trending, reporting and compliance. Each layer is meant to talk only to its neighbors, through controlled points.

The model describes intent, and real networks drift from it. Vendors add direct links for support. Business teams ask for live data and someone opens a path from the office network to a controller. Wireless and cellular links bypass the layers altogether. The map exists to show where the actual network departs from the layered design, which is exactly where attackers find their way in.

Map what is actually connected

The alert asked operators to build an accurate map of control system networks as operated, and the distinction matters. Many organizations hold design drawings or vendor documents showing how systems were meant to be connected. Fewer know how they are connected today, after years of maintenance, remote support arrangements, temporary links and integrations feeding data to business systems. It is the difference between the architect’s original plans for a house and the house as it stands after twenty years of renovations, some of them done without permits.

The gaps take predictable forms. At a municipal water utility, it might be a remote access tool a contractor installed years ago. At a hospital, it might be a building management system reachable from the corporate network, or a vendor’s cellular modem attached to a chiller. A cellular modem is a small device that connects equipment to the internet over the mobile phone network, completely bypassing the organization’s own network and its security controls. For a bank, it might be the facilities systems at a colocation data center, run by the provider under a contract that says little about security. A colocation data center is a building where many companies rent space for their computers, while the provider runs the power, cooling and physical security.

Building the map takes a joint walk-through by IT and operations staff, cabinet by cabinet. Neither group can do it alone: IT staff know networks but not the equipment, and operations staff know the equipment but often not how it connects to everything else.

Passive monitoring tools designed for control system protocols can then confirm what talks to what without disturbing fragile devices. A protocol is the language devices use to talk to one another, and control systems use their own specialized ones. Passive monitoring means listening to that conversation without joining it. Active scanning, by contrast, sends probes to each device to see what responds. Active scanning can crash older controllers, so it needs operations sign-off and a maintenance window.

Listening to the network

Passive monitoring usually starts with a copy of network traffic. Many industrial switches can mirror traffic from selected ports to a monitoring port, often called a SPAN port. Where switches cannot, a network tap, a small device inserted into a cable, copies traffic without affecting it. The monitoring tool decodes industrial protocols and builds a picture of which devices exist, what they are and who talks to whom.

Passive tools have limits. They see only devices that talk during the observation period, and only on the network segments they are connected to. A controller that communicates once a week, or hangs off a serial link that never reaches an Ethernet switch, may not appear. For that reason the alert paired passive mapping with physical walkdowns. Vendor tools, which can query their own devices safely, fill some gaps.

The output should record each device, its firmware version, its connections, who maintains it and how vendors reach it. A map is only useful if it stays current, so changes to control networks need to pass through a change process that updates it. Otherwise the map falls out of date as soon as the next connection is added.

What goes in the inventory

The alert was specific about the inventory. It asked for devices with network addresses, software and firmware versions, process logic and programs, removable media, and standby and spare equipment. Each of those has a reason. Firmware versions tell you which vendor advisories apply. Process logic is what you will need to restore. Removable media, such as the USB drives technicians carry, is a route around any network control. Spare equipment matters because a spare controller loaded with old firmware becomes a vulnerability the day it is swapped in.

It also asked operators to catalog every external connection, including business, vendor and remote access links, and to review the service contracts behind third-party connections. That turns the map into a record of obligations as well as wiring. A connection with no contract behind it, or a contract with no named connection, is a finding in its own right.

How to tell whether the map is working

A few simple measures show whether a mapping effort is real. One is the share of assets with a named owner, a firmware version and a recorded way in for vendors. Another is the number of unknown devices or connections found in each review, which should fall over time. A third is the date of the last physical walkdown for each site. A map that has not been checked on the ground in two years is a historical document.

Decide what should be connected, and protect it

With a map in hand, the next step is reducing exposure. Control devices reachable directly from the internet should come off it. Remote access should go through one controlled, monitored path with strong authentication, rather than through tools each vendor installs for its own convenience. Connections between control and business networks should be limited to what is needed, with data flowing out through defined points rather than in both directions.

These choices involve trade-offs. Remote access lets small utilities and hospital facilities teams cover nights and weekends with few staff. Data from building systems supports energy management and regulatory reporting. Disconnecting everything is rarely realistic. Deciding deliberately is: record each connection’s purpose, owner and controls, and remove the ones nobody can justify.

Remote access, done properly

The alert’s guidance on remote access was concrete. It called for a virtual private network with strong encryption and multi-factor authentication for every remote connection. A virtual private network, or VPN, creates an encrypted tunnel between the remote user and the organization’s network. It advised allowing connections only from known addresses, using geographic blocking where appropriate, confirming the business need for each access path, and capturing and reviewing access logs.

For a small water utility, the realistic design is often a single remote access service in a boundary zone between IT and OT. Every vendor and every on-call operator connects through it. Accounts are individual, sessions are logged and vendor accounts are enabled only when a job is scheduled. That last control, turning vendor access on for a window and off afterwards, removes much of the risk of a forgotten account. It costs some convenience, because an operator must approve each session, and that cost is worth stating openly when the policy is agreed.

Protection for what remains connected follows from the map. Segmenting control networks into zones by process, monitoring traffic at the boundaries, and alerting on new connections or unexpected commands give operators a chance to see an intrusion before it reaches equipment. With segmentation, a breach in one zone stays there rather than spreading everywhere.

One-way flows and the boundary zone

Business systems often need data from OT: energy use for billing, water quality readings for regulators, equipment hours for maintenance planning. The safer pattern pushes that data out to a historian or file server in a boundary zone, where business users can read it, rather than letting business systems reach into OT. Where the risk justifies it, a data diode enforces the one-way flow in hardware. A data diode is a device that physically allows traffic in one direction only, so nothing from the business side can travel back into the control network.

When a device cannot be patched

Patching OT is harder than patching laptops. A patch may need vendor certification, a maintenance window and a plant shutdown, and some devices will never receive one. The alert asked operators to check their inventory against vendor advisories, CISA advisories and the national vulnerability database, and to apply mitigations where possible. Where a patch is not possible, compensating controls carry the weight: removing internet exposure, restricting which hosts can reach the device, disabling unneeded services and monitoring for changes. The record should show, for each known vulnerability, whether it was patched, mitigated or accepted, and by whom.

Watching for unauthorized change

The alert’s monitoring advice was aimed at the attacks it described. It recommended checking controller logic against a known-good baseline, so that a modified program would be detected. It advised against leaving controllers in remote program mode during operation. Many controllers have a physical key switch that determines whether their logic can be changed over the network, and leaving it in the remote position is a common convenience. It also suggested locking or limiting set points, so that an intruder could not push a chemical dose or a temperature outside safe bounds. These controls are cheap relative to their value, and they work even where the network itself cannot be fully trusted.

Default passwords on controllers and human-machine interfaces should be changed, a basic step the later water utility incidents showed was still often missed. A default password is the one set at the factory, often published in the device manual and easy for attackers to find.

Vendor contracts deserve attention here. Many control systems are maintained under agreements that predate any security expectations. Renewals are the moment to require named vendor accounts, multi-factor authentication, session logging and notice before changes. Named accounts mean each vendor engineer logs in as themselves rather than through a shared login. Multi-factor authentication requires something beyond a password, such as a code from a phone, so that a stolen password alone is not enough. Where a vendor cannot meet those terms, the remote path can be wrapped in controls the operator owns, such as a jump host that records every session. A jump host is a single, closely watched computer that every remote connection must pass through, so that visitors are checked and their actions recorded.

Be able to operate without trusted control systems

The alert asked operators to plan for running when control systems are unavailable or untrustworthy. That includes manual operation and restoring devices from known-good configurations. A known-good configuration is a saved copy of a controller’s program and settings from a time when it was working correctly, which can be reloaded if the device is tampered with or fails.

Many organizations are weakest here. Staff who once ran plants or building systems manually have retired. Procedures have not been rehearsed, and backups of controller logic and configurations may be missing or out of date.

Planning for a malfunction as well as a failure

The alert drew a distinction that many continuity plans miss. A failed system stops. A malfunctioning system keeps running but does the wrong thing, and may show operators a false picture while it does. Plans written for failure assume operators know something is wrong. Plans written for malfunction need ways to verify the process independently of the screens. Local gauges, hand-held instruments, laboratory samples and physical checks by staff on the floor all serve that purpose.

Safety systems matter here. Many industrial processes have a safety instrumented system, a separate controller designed to bring the process to a safe state if conditions go out of bounds. It is meant to be independent of the main control system. The 2017 Triton malware, found at a petrochemical facility in Saudi Arabia, targeted exactly such a safety system. Operators should know whether their safety layer shares networks, workstations or accounts with the systems it is meant to protect.

Resilience at this step is as much about people as technology. It needs documented manual procedures, staff trained to follow them and exercises that test them under time pressure. It also needs current backups of controller programs and firmware, stored offline and tested by restoring to spare hardware. Offline matters because ransomware that reaches a network can encrypt or delete backups stored on it. Testing matters because a backup that has never been restored is a hope, not a plan.

The alert’s list of what belongs in a gold copy is broader than most backup programs. It includes firmware, software, ladder logic, service contracts, licenses, keys and configurations. Ladder logic is the most common programming language for PLCs, named for its diagrams that resemble the rungs of a ladder. Licenses and keys are easy to forget. A restored engineering workstation is of little use if the vendor software on it will not run without a license server that was lost in the same incident.

Dependencies are the other gap. A hospital’s chiller plant depends on power, on the building management system and on the controls of the cooling towers. A data center’s cooling depends on the same building systems as its fire suppression. The alert asked operators to identify system and operational dependencies, because restoring one component in isolation rarely restores the service.

Above all, it needs clarity about when to shut a process down deliberately rather than run it on untrusted controls, and who has authority to decide. A water utility, a hospital and a data center will answer differently, which is why the decision has to be made in advance. In the middle of an incident, with alarms sounding and incomplete information, is the wrong moment to discover that nobody knows who can make that call.

Three surprises behind one hospital’s chiller

Take a county hospital whose cybersecurity team is strong on email and laptops. Its facilities team runs heating, ventilation and cooling through a building management system installed by a vendor years ago.

A joint walk-through finds three things nobody had on paper. The building management system’s HMI is reachable from the corporate network, so anyone who compromised a staff laptop could reach it. A chiller has a vendor’s cellular modem attached, installed so the vendor could monitor it remotely. And the controllers still use the passwords they shipped with.

All three are common, and none required bad intent; each was a sensible shortcut at the time. Fixing them is equally unglamorous: a firewall rule separating the building systems from the corporate network, the modem replaced by vendor access through the hospital’s own jump host with multi-factor authentication, passwords changed and recorded, and offline backups taken of each controller’s program. The facilities team writes down how to run the chiller plant manually, and the hospital agrees who can order a deliberate shutdown. The vendor contract, up for renewal, adds named accounts and session logging.

The fixes had side effects, as fixes do. Removing the cellular modem cut off the vendor’s remote condition monitoring, which had given early warning of compressor faults. The hospital restored it by sending readings out through its own boundary zone. The new firewall rule broke an energy dashboard that the sustainability team relied on, which led to a short argument and then a historian copy in the boundary zone. The vendor quoted an additional fee to support the jump host. Each issue was solvable, and each would have stalled the work if nobody had the authority to settle it.

The changes also needed operations sign-off and scheduling. Changing a controller password in a running plant can interrupt the HMI’s connection to it, so the facilities team did the work in a planned window with the vendor on call. Treating a security change with the same discipline as any other plant change is what keeps the operations staff on side.

Physical stakes, and nobody in charge

The stakes of operational technology are unusually direct. An incident can stop a water plant, cancel surgeries or take a data center offline. The costs include emergency response, lost revenue, regulatory scrutiny and, in public services, a loss of community trust that is slow to rebuild. Because the equipment is physical, the worst outcomes involve safety as well as data.

Across all three steps, ownership is the common gap. IT security teams often lack visibility of control systems and the expertise to work on them safely. Operations and facilities teams own the equipment but not its security. Vendors maintain much of it under thin contracts. Naming one owner for control system security, with authority across IT, operations, facilities and vendor management, is what allows the map, the exposure decisions and the resilience plan to happen at all.

For banks and public agencies the exposure is often indirect, which makes it easy to overlook. A bank may never touch a controller itself, yet its trading floor, call centers and data halls depend on power and cooling run by a landlord or a colocation provider. A county government may own a water plant run by a handful of staff and a contractor. In both cases, the risk sits in a contract and a relationship as much as in a device, and leaders need to know who is accountable for it.

Which rules reach which operators

The legal backdrop varies by sector, which affects both obligations and budgets. The America’s Water Infrastructure Act of 2018 already required community water systems serving more than 3,300 people to complete risk and resilience assessments and emergency response plans. They also had to certify them to the EPA. Electric utilities that operate parts of the bulk power system fall under the NERC Critical Infrastructure Protection standards, which are mandatory and audited.

Hospitals had no OT-specific federal security rule. The CMS emergency preparedness requirements for providers that participate in Medicare and Medicaid do require an all-hazards risk assessment, emergency plans, communication plans, and training and testing. A loss of building controls is a hazard those plans should cover. Federal agencies treat OT they own as federal information systems under FISMA, with NIST Special Publication 800-82 as the main guide to securing industrial control systems.

Banks rarely run OT themselves, but examiners look at the dependency. The FFIEC’s business continuity management booklet, issued in 2019, expects institutions to manage risks from facilities and from third-party service providers. For a bank in a colocation facility, the realistic controls are contractual. They include the right to review the provider’s security practices for building systems, notice of incidents affecting power and cooling, and evidence of tested resilience. Counsel should confirm which obligations apply to a given organization.

Where the owner should report

There are three common models for ownership, each with trade-offs. Placing OT security under the CISO brings security discipline and visibility, but risks a team that does not understand the plant. Placing it under the operations or facilities director keeps it close to the equipment, but often starves it of security skills and budget. A dedicated OT security lead with a dual reporting line to both is common in larger utilities and health systems. What matters most is that the owner can make a decision stick across both sides, including the decision to delay a change because the plant cannot tolerate it.

The case against this approach

Three objections come up often. The first is that critical control systems should simply be air-gapped, physically disconnected from every other network. True air gaps are rare once vendors, data needs and remote support are counted, and the Stuxnet attack on Iranian centrifuges, revealed in 2010, showed that removable media can cross a gap. An air gap is a design goal to verify, which brings the work back to the map.

The second objection is cost. Commercial OT monitoring platforms are expensive, and a small water utility or rural hospital cannot buy one. The three steps in the alert do not require one. A walkdown, a spreadsheet inventory, firewall rules, changed passwords, a single remote access path and offline backups are within reach of small operators. CISA offers free vulnerability scanning to public and private critical infrastructure operators, which can show what is exposed to the internet.

The third objection is that the threat is overstated, since the best-known US water incident of the period was later attributed to employee error. The 2023 compromises of internet-exposed controllers at US water utilities were real, and the February 2020 gas compression case showed that ransomware alone could stop physical operations. The probability of a sophisticated attack on any one operator may be low. The probability of ransomware or opportunistic access reaching a poorly mapped control network is not.

Practicing the bad day

The alert’s final recommendation, exercising incident response, is the one that ties the others together. An incident response plan is the written playbook for what happens when something goes wrong: who is called, who decides, how the organization communicates and how it recovers. Exercising it means rehearsing before it is needed.

The most common format is a tabletop exercise. Leaders and staff from IT, operations, facilities, communications and legal sit around a table, and a facilitator walks them through a realistic scenario, step by step. At the county hospital it might begin with a facilities engineer noticing that chiller set points have changed overnight with no record of anyone changing them. The facilitator then asks who is called first; whether the hospital disconnects the building systems from the network, and what that does to temperature control in the operating rooms; who decides whether to switch to manual operation; and who tells clinical leaders, and what they tell patients and staff.

Tabletop exercises are cheap and revealing. They expose gaps that no document review will find: phone numbers that are out of date, a vendor nobody can reach on a weekend, a manual procedure that assumes a retired engineer is still on staff, or two executives who each believe the shutdown decision is theirs. The aim is to find those gaps while the stakes are low, and passing matters far less.

Exercises also connect the three steps. A scenario quickly shows whether the map is good enough to tell responders what a compromised device is connected to, whether exposure has been reduced enough to contain the problem, and whether the organization can actually keep operating without trusted controls. Running one or two a year, and fixing what they reveal, turns a plan on paper into a capability.

Who belongs in the room

The alert was explicit that exercises should include executives, public affairs and legal staff as well as IT and OT. Each brings a decision the others cannot make. Executives decide whether to accept the cost of a shutdown. Public affairs decides what to tell customers, patients or residents, and when. Legal advises on notification obligations to regulators and law enforcement. The alert also asked participants to identify the key decision points and who holds authority over each, under which circumstances.

It recommended a scenario in which the control system acts against safe operation, using the techniques it described. Such a scenario is harder than one in which systems simply go down, because it forces the question of how operators would know. It also advised arranging third-party support in advance and reviewing service contracts and government emergency response services. An incident is a poor time to discover that the vendor’s contract covers business hours only, or that nobody has the number for the state’s emergency management office.

Measuring the exercise

A useful exercise ends with a short list of findings, each with an owner and a date. Over successive exercises, the measure of progress is whether the same findings recur. A plan that produces the same gaps every year has not been improved, however well the exercise was run.

Signs of a control system program that works

A working program is easy to recognize. There is a single named owner for control system security with authority across IT, operations, facilities and vendor management. An as-operated map exists, built on the ground and kept current through change control. Every connection has a recorded purpose and owner, nothing is directly exposed to the internet, and remote access runs through one path the organization controls. Default passwords are gone. Offline backups exist and have been restored. Manual procedures are written and practiced, and the shutdown decision has a named owner.

The failures follow from the ownership gap:

• Treating OT as someone else’s problem. IT assumes facilities has it covered; facilities assumes IT does.
• Trusting the drawings. Design documents describe how systems were meant to be connected, not how they are.
• Letting each vendor choose its own way in. Every convenience tool is another side door.
• Scanning without sign-off. Office-style scanning can crash fragile controllers and destroy trust between teams.
• Untested backups. A backup that has never been restored may not work when it matters.
• Improvising the shutdown decision. Deciding who can stop a process in the middle of an incident wastes the most valuable minutes.

Building control systems need an owner, a current map and tested backups

1. Which of our buildings, plants or data centers depend on control systems, and who owns their security?
2. Do we have a map of those systems as they are actually connected today, and when was it last checked on the ground?
3. Are any of our control devices reachable from the internet, and do all vendors connect for remote support through one path we control and record?
4. Do we hold offline backups of controller programs, and when did we last prove we could restore one?
5. Could our staff run critical processes manually, and who has authority to shut a process down rather than run it on controls we no longer trust?

With hindsight

Control system security moved up the agenda quickly. In February 2021, officials in Oldsmar, Florida, reported that someone had used remote access software to change chemical settings at the city’s water treatment plant. In 2023 the former city manager said it had been employee error, and the FBI said its investigation could not confirm that the incident was initiated by a cyber intrusion.

In May 2021, a ransomware attack on Colonial Pipeline’s business systems led the company to shut down pipeline operations as a precaution. That case is worth pausing on: the attack hit business systems, not the pipeline’s controls, yet the physical operation stopped as a precaution. The Transportation Security Administration issued security directives for pipeline operators in May and July 2021, making incident reporting and specific security measures mandatory for designated pipelines. In late 2023, attackers linked to Iran compromised internet-exposed controllers at several US water utilities.

Federal agencies issued repeated warnings to water systems. In March 2023, the Environmental Protection Agency issued an interpretive memorandum asking states to assess cybersecurity in the sanitary surveys they carry out on public water systems. It withdrew the memorandum in October 2023, citing litigation, and moved to voluntary support and technical assistance. The three steps in the 2020 alert, map, reduce exposure and plan to operate manually, remained the core of the guidance that followed.

The threat picture grew more serious. In February 2024, CISA, the NSA and the FBI warned that a Chinese state-sponsored group known as Volt Typhoon had pre-positioned itself on IT networks in the communications, energy, transportation and water sectors. The agencies said the aim was to be able to move into OT and disrupt critical functions in a crisis. They reported that the group had tested access to OT systems using default vendor credentials, the same basic weakness the 2020 alert had flagged.

Elsewhere, the security firm Dragos reported in July 2024 on malware it called FrostyGoop. It had been used in January 2024 against a municipal energy company in Lviv, Ukraine, sending Modbus commands that cut heating to hundreds of apartment buildings in winter. Modbus is the unauthenticated protocol described earlier in this article.

US water systems kept appearing in the news. In September 2024, Arkansas City, Kansas, switched its water treatment plant to manual operations after a cybersecurity incident. Its officials said the water supply was not affected. In October 2024, American Water, the largest regulated US water and wastewater utility, disclosed unauthorized activity in its IT systems and paused customer billing. It said its water and wastewater facilities were not affected. Both cases fit the pattern the alert described: the ability to run manually, and a clear line between business systems and control systems, determined how much damage an incident did.

Guidance kept maturing. NIST published a third revision of its guide to OT security, Special Publication 800-82, in September 2023, widening its scope from industrial control systems to OT generally. Its core advice on inventories, segmentation and recovery tracks the 2020 alert closely.

Seen together, these events make the alert’s priorities look well chosen. The Oldsmar episode, whatever its cause, showed how much attention a remote access tool on a control system could attract and how hard it can be to establish what happened. Colonial Pipeline showed that the decision to shut down operations deliberately is real and consequential, and best made with a plan in hand. The 2023 water utility compromises involved exactly the internet-exposed controllers the alert had warned about three years earlier.

The message from these events is that operational technology security does not begin with sophisticated tools. It begins with knowing what you have, deciding what should be connected and being able to run without it. Those are management disciplines as much as technical ones, and they depend on one person being accountable across IT, operations and vendors.