When vendors set facial recognition policy before agencies did

The week

In the space of four days in June 2020, three of the world’s largest technology companies changed the rules for American policing, and none of them was a government.

On June 8, IBM told members of Congress in a letter that it would no longer offer general-purpose facial recognition or analysis software, citing concerns about mass surveillance and racial profiling. The letter, from chief executive Arvind Krishna, also called for a national dialogue on whether and how police should use the technology. On June 10, Amazon announced a one-year moratorium on police use of its Rekognition facial recognition service. A moratorium is a temporary pause: Amazon was not leaving the business, but it would stop police from using the product for a year. On June 11, Microsoft said it would not sell facial recognition to US police departments until there was a national law grounded in human rights.

What each company actually committed to

The three announcements were not equivalent, and the differences mattered to agencies. IBM’s was an exit from a general-purpose product line, by a company that was not a major supplier of policing face search. Amazon’s was a time-limited pause, and it carved out organizations that used Rekognition to help rescue trafficking victims and find missing children. It said it hoped Congress would use the year to put appropriate rules in place. Microsoft’s was a conditional refusal of new sales to US police, tied to legislation that did not yet exist.

None of the three covered the smaller, specialist vendors that supplied much of the face search actually used by police. None changed the state and regional systems that searched driver’s license and booking photos. For many agencies, the immediate operational impact was small. The signal was large.

How facial recognition works

Facial recognition is software that compares a picture of a face against a collection of other pictures and suggests which ones might be the same person. It does not “recognize” anyone the way a friend would. It measures features of a face, turns them into a set of numbers, and ranks the closest matches in a database, often with a score. A police investigator might feed in a still from a store’s security camera and get back a list of possible names from a database of booking photos or driver’s license images.

The protests and the policy response

The announcements came during nationwide protests following the killing of George Floyd in Minneapolis on May 25. Public attention had turned sharply to how police use their powers, and to whether new tools made old problems worse. On June 8, the same day as IBM’s letter, members of Congress introduced the Justice in Policing Act, a broad police reform bill. The House passed it later that month. It did not become law.

Cities had already begun to act on their own. San Francisco had barred its city departments from using facial recognition in May 2019, and Somerville and Oakland followed that year. In March 2020, Washington State enacted a law setting conditions for government use, including accountability reports, testing and meaningful human review of decisions. Later in June 2020, Boston’s city council voted to bar city use of the technology.

The evidence on accuracy

There was already evidence on whether the tools made old problems worse. In December 2019, a NIST evaluation had found demographic differences in the accuracy of many facial recognition algorithms, with large variation between algorithms. NIST, the National Institute of Standards and Technology, is the federal agency that sets measurement standards and tests technology independently. It tested 189 algorithms from 99 developers. For many of them, false matches were more likely for some groups of people than others. In one-to-many searches against a database of mugshots, false positives were higher for African American women. NIST also found that the most equitable algorithms were among the most accurate, so the choice of product mattered.

Academic work pointed the same way. A 2018 study by researchers at MIT and Microsoft had found much higher error rates for darker-skinned women in commercial systems that classified gender from faces. The ACLU had run its own test of Rekognition that year and reported that it falsely matched members of Congress to mugshots, a test Amazon disputed on the grounds that the ACLU had used a lower confidence setting than it recommended for police.

Within two weeks of the announcements, the first widely reported case of a wrongful arrest based on a facial recognition match came to light. Detroit police had arrested a man in January 2020, at his home and in front of his family, for a theft he did not commit, after a face search produced him as a candidate. Prosecutors dropped the charges.

The discovery for agencies

For police chiefs, city managers and state officials, the week delivered an uncomfortable discovery. A decision about what tools their officers could use had just been made in corporate boardrooms. Many agencies had no policy of their own to measure it against.

The June 2020 announcements were made by vendors, under public pressure, in a matter of days. Police departments and other agencies that used or planned to use facial recognition found that a supply decision was now a policy decision. Many had no formal policy of their own to set against it.

Three controls were missing: an inventory of which tools were in use, rules for how they could be used and on what evidence of accuracy, and contract terms that made those rules enforceable. The same three apply to any AI system that supports decisions about people, whether in a police department, a benefits agency, a bank or a hospital.

The analysis starts with how the technology works, because each control answers a specific technical weakness. It then takes the three controls in turn: the inventory, the rules and the contract. It shows them working together on a single search, then carries them into other AI that supports decisions about people. It closes with the evidence a well-governed organization can produce, and the questions that test whether it can.

Probe, gallery and a ranked list

A facial recognition system has three main ingredients. The first is a probe image: the picture of the unknown person, such as a frame from surveillance video. The second is a gallery: the collection of known images to search, such as booking photos. The third is the algorithm: the software that converts each face into a numerical summary and compares the probe with every face in the gallery.

What comes out is a ranked list of candidates, each with a similarity score, rather than an answer. The top candidate is often right and sometimes wildly wrong, and the software has no idea which. A human has to look.

Several things affect how good the list is. The quality of the algorithm matters enormously; NIST’s testing showed that the best performed far better than others. The quality of the probe image matters just as much. A clear, well-lit, front-facing photo is very different from a grainy, angled, low-resolution frame captured at night. And the size and makeup of the gallery matter, because the more faces you search, the more chances there are for a stranger to look similar.

Two kinds of error are possible. A false negative means the system misses the right person. A false positive means it suggests the wrong person. In policing, a false positive is the more dangerous error, because it points investigators at someone who did nothing.

The demographic finding adds a layer. If an algorithm makes more false positives for some groups than others, the burden of its mistakes falls unevenly. For a public agency, that raises questions of fairness, civil rights and public trust as well as technical performance.

Checking a claim versus searching a crowd

There are two quite different uses, and they carry different risks. One-to-one matching, or verification, checks whether a person is who they claim to be, by comparing a live photo with one stored image. Unlocking a phone or boarding a plane works this way. One-to-many matching, or identification, searches an unknown face against a whole gallery. Investigative searches by police work this way.

The difference matters because false positives grow with the size of the gallery. A tiny error rate per comparison becomes a real chance of a wrong candidate when the probe is compared against millions of faces. That is why an algorithm that performs well at verification can still produce unreliable lead lists in identification, and why NIST tested the two uses separately.

Two settings shape the output, and someone at the agency or the vendor chooses them. The threshold is the similarity score below which the system returns nothing. The candidate list length is how many names come back. A low threshold and a long list produce more leads and more wrong ones. Many agencies did not know who had chosen these settings or what they were.

What goes into the gallery

Galleries are not neutral. Booking photo collections reflect who has been arrested, which reflects where and how policing has been done. A group that has been arrested more often is more heavily represented, and so has more chances to appear as a wrong candidate. Driver’s license galleries are different again: they include most adults who drive, almost none of whom has any connection to a crime. A 2016 study by Georgetown Law’s Center on Privacy and Technology estimated that about half of American adults were in a law enforcement face recognition network, largely through license photos.

Gallery quality also varies. Old photos, duplicate records and images taken under poor conditions all degrade results. An agency that searches another jurisdiction’s gallery usually has no visibility into any of this.

Gallery membership is a policy question too. A booking photo may stay in a searchable collection long after the charge was dropped, unless someone has a process to remove it. State laws on sealing and expungement differ, and face search systems were often built without any link to them. An agency that owns a gallery should know whose images it holds, on what authority and for how long, and should be able to show that removals required by law actually happen.

What goes into the probe

The probe is where some of the worst practices were found. A 2019 Georgetown report documented police submitting forensic sketches, digitally altered images and, in one New York case, a photo of a celebrity the suspect was said to resemble. Editing a probe, for example by pasting in eyes or a mouth from another face to replace an obscured feature, changes what the algorithm is comparing. Its scores then mean far less than they appear to.

Each of these weaknesses points to a control. Gallery and settings belong in the inventory. Probe quality and the meaning of a candidate list belong in the use rules. Disclosure of the algorithm and its test results belongs in the contract.

An inventory of tools in use

Facial recognition entered many agencies by routes that bypassed central governance. Free trials were offered directly to investigators. Features arrived embedded in other products, such as evidence management or video analytics. Detectives searched databases run by other jurisdictions, and some used services marketed to individual officers. Some agencies could not say whether their staff used facial recognition, which systems were involved, or which image collections were searched.

This is a familiar pattern to anyone who has watched software spread through an organization. A team signs up for a free tool because it solves a real problem. A vendor adds a new feature to a product the organization already owns. Nobody decided, at the top, to adopt the capability, and so nobody is accountable for how it is used.

The federal government was no better placed. A year after the vendor announcements, the Government Accountability Office reported that 14 federal agencies used facial recognition in criminal investigations and all of them used systems owned by others. Only one knew which of those non-federal systems its employees were using. GAO recommended that agencies track that use and assess its privacy and accuracy risks.

An inventory fixes that. For each tool it records the vendor, the algorithm and version, the image sources searched, the units and people with access, the purposes approved and the legal authority relied on. It includes capabilities embedded in other products and services reached through partners, such as a state or regional system. The same algorithm can be appropriate for one purpose and not another, so purpose belongs in the record.

An inventory is also the precondition for answering legislators, auditors, defense counsel and the public, all of whom began asking in 2020. When a city council asks “do we use facial recognition?”, the honest answer cannot be “we are not sure.”

Keeping the inventory current needs a rule with teeth. No algorithmic tool may be used on agency cases unless it is registered and covered by an agreement. Trials and embedded features count. Without that rule, the inventory decays as soon as it is published.

How to find what is already in use

Asking department heads is a start, but it misses the most common routes. A thorough inventory uses several sources at once. Procurement and purchasing-card records show paid subscriptions, including small ones below formal bidding thresholds. Network and web proxy logs show traffic to known face search services. Software asset records show which installed products include face-matching features, and whether they are switched on. Agreements with state, regional and federal partners show which shared systems staff can query.

A staff survey helps, especially if it comes with a short amnesty: report a tool now and it will be reviewed rather than treated as misconduct. Investigators who used a free trial in good faith are more likely to say so if they are not punished for it. The aim is a complete list, and people who fear discipline will not provide one.

Different agencies, different blind spots

The blind spots vary by type of agency. A large city police department may have a formal unit for face searches and still have detectives using outside services. A rural sheriff’s office may have no formal capability and rely entirely on requests to the state. A state police agency often runs the shared system that local agencies query, which makes it a provider as well as a user. Its inventory has to cover who searches its system and for what.

Gallery owners are a separate category. A state motor vehicle agency that lets police search license photos is making a policy decision about millions of residents, even though it uses no AI itself. Its inventory should record which agencies may search, under what agreement and with what logging. Prosecutors’ offices are another blind spot. They rarely run searches, but they need to know when cases rely on them, because that affects what they must disclose.

One sheriff’s office, three surprises

Suppose a county sheriff’s office believes it does not use facial recognition. An inventory exercise finds three things: a detective who has been using a free trial of an online service for six months, a video analytics package bought for jail cameras that includes a face-matching feature switched on by default, and routine requests to a state system that searches driver’s license photos. Not one of the three had been approved for any specific purpose. Each now goes into the register, with an owner, a purpose and a decision about whether to keep it. The free trial is stopped. The embedded feature is switched off pending review. The state system is kept, under a written use policy.

Rules for use and evidence of accuracy

Use policies answer practical questions: for which offenses or purposes a search is allowed; whether a match can be the sole basis for action; who reviews candidate matches, with what training, and how the review is documented; and when the public, or a defendant, is told that facial recognition was used.

Accuracy evidence belongs in the policy. The NIST evaluation showed that accuracy varied by algorithm and by demographic group, and that the best algorithms performed far better than others. Vendor claims could not substitute for evidence about the specific algorithm, on the image quality the agency actually used, for the specific purpose. Low-quality probe images, such as frames from surveillance video, produce much less reliable results than the test conditions vendors cite.

Vendor accuracy figures work like a car’s advertised fuel economy: measured under controlled conditions, and worse on icy hills with a full trunk. An agency needs to know how the tool performs on its own roads.

The cases that later drew most attention involved people wrongly identified and arrested, where a facial recognition result had been treated as more than an investigative lead. A rule requiring independent corroboration before any action against a person addresses exactly that failure. Corroboration means separate evidence, gathered by other means, that points to the same person. So does training that teaches investigators that a ranked list of candidates is not an identification. Auditing a sample of searches each quarter shows whether the rules are followed in practice.

What the rules should specify

A usable policy is concrete. It lists the purposes for which a search is allowed, such as violent felonies, identifying a deceased person or finding a missing person, and those for which it is not, such as identifying people at a lawful protest. It sets a minimum probe image standard and bans edited images, sketches and lookalike photos as probes. It fixes the threshold and the candidate list length, or requires that any change be approved and recorded.

It also sets who may run and review searches. Many agencies concentrate searches in a small group of trained examiners, rather than letting any investigator search. The Facial Identification Scientific Working Group, a body of practitioners and scientists, publishes guidance on training for facial comparison. A policy can require examiners to meet a recognized standard and to be retested periodically.

Finally, the rules decide what happens to the records. Search logs, probe images, candidate lists and examiner notes are evidence. A retention period tied to records law and to the time a conviction can be challenged keeps them available when they are needed.

Disclosure and due process

The hardest rule concerns disclosure. If a face search set an investigation in motion, the defense may argue it is entitled to know: which system was used, what the probe looked like, how many candidates were returned and why the others were ruled out. Prosecutors have constitutional duties to disclose evidence favorable to the defense, and an unreliable search that pointed at the defendant may fall within them. Agencies should check with counsel and prosecutors on how those duties apply in their jurisdiction.

A policy that records the use of facial recognition in every case file makes disclosure possible. One that treats the search as an informal tip, never written down, makes it impossible, and risks later convictions being challenged.

Who writes the rules

Governance of the rules matters as much as their content. Policies written by the unit that uses the tool tend to favor convenience. Agencies that involved legal counsel, a privacy officer and, in some jurisdictions, a civilian oversight body produced rules that held up better under public scrutiny.

Some cities had already built a process for this. Surveillance technology ordinances, adopted in places such as Seattle and Oakland before 2020, required council approval before an agency acquired a surveillance technology and annual reports on its use. Where they existed, those ordinances gave facial recognition a defined route to approval, or refusal, instead of leaving it to individual units.

Publishing the policy, along with annual counts of searches and of cases where a match led to an arrest, gave legislators and residents a basis for trust. Those disclosures also forced agencies to collect the data needed to know whether the tool was helping at all. An agency that cannot say how often a tool produced a useful lead cannot say whether it is worth the cost and risk.

The case for keeping the tool

The strongest objection to tight rules is that facial recognition works, often. Investigators have used it to identify victims of child exploitation, find missing people and generate leads in serious crimes that had no others. Amazon’s own moratorium carved out organizations doing this kind of work. Heavy restrictions, the argument runs, cost cases and protect offenders.

A second objection is practical. Where a city bans the tool, its detectives may simply ask a neighboring agency or a state system to run the search, with less oversight than before. A ban can move the activity out of sight rather than stop it.

Both objections strengthen the case for the three controls. If the tool is valuable, an agency needs evidence of when it helps, which only logging and outcome tracking provide. If bans push searches elsewhere, the inventory and the rules have to cover requests to partners as well as the agency’s own systems. Governance is what allows legitimate use to continue under public scrutiny.

Contract terms that make the rules enforceable

Procurement is where policy becomes enforceable. Contracts can require disclosure of the algorithm and its independent test results, logging of every search, audit rights, limits on the vendor’s use of agency data, and notice before material changes to the model. Without such terms, an agency cannot verify that a tool meets its policy. The vendor can also change or withdraw the product without warning, as the June announcements showed.

Each of those terms has a plain purpose. Algorithm disclosure tells you what you are actually using, so you can match it to independent test results. Search logging records who searched, when, with what image and for what case, which is the raw material for any audit. Limits on the vendor’s use of agency data stop images collected for public safety from being reused to train or sell other products. Change notice matters because software is updated constantly; a model that passed testing last year may behave differently after an update.

Clauses worth specifying

• Identification of the algorithm. The vendor names the algorithm and version in production and the independent evaluation it relates to, and says when either changes.
• Performance on agency-like data. The vendor supports testing on images of the quality the agency actually uses, before go-live and after major updates.
• Logs the agency owns. Every search is logged with user, time, probe, settings, candidates and case number, exportable in a usable format and retained for the agency’s schedule.
• Data use limits. Agency images are not used to train the vendor’s models or shared with other customers without written approval.
• Exit and transition. On termination, for any reason, the vendor returns agency data, preserves logs and provides transition help for a set period.
• Public records. The contract anticipates that policies, logs and test results may be subject to public records requests, and limits the vendor’s confidentiality claims accordingly.

Contracts need exit terms as well. When a supplier withdraws, the agency needs its data back and its search logs preserved for oversight and litigation. It also needs a plan for investigative work that depended on the tool. Few contracts signed before 2020 contemplated a vendor leaving the market for policy reasons. Retention of logs matters most, because past searches may be challenged in court years later. A defendant’s lawyer may ask exactly how a match was produced, and “the vendor is gone and so are the records” is not an answer any agency wants to give.

Trials, pilots and shared contracts

Free trials are where procurement control most often fails. A trial has no contract, so it has none of the terms above. A simple rule closes the gap: any trial of a tool that processes agency data needs a short written agreement, an owner, an end date and a decision at the end. Small agencies often buy through state cooperative contracts or rely on regional systems. They should check that the master agreement contains the terms above, because they will have little bargaining power to add them later.

Bargaining power is uneven. A large state agency can insist on these terms; a small town police department buying a single subscription cannot. Joint purchasing helps. Agencies that buy together through a state contract or a regional consortium can set common terms once, share the cost of independent testing and audit the vendor collectively. A published model clause set, adopted by a state procurement office, gives every local buyer the same protections without each negotiating alone.

The trade-off is speed. Agencies that require full terms for every tool will adopt fewer tools, and more slowly. For tools that touch decisions about people’s liberty or benefits, that is the right trade. For low-risk tools, a lighter template can do.

What a well-governed search looks like in practice

Here are the three controls working together on an ordinary day, at the same sheriff’s office a year after its inventory exercise.

A store reports a theft and hands over a clip from its security camera. The assigned detective wants to know who the person in the clip might be. Under the county’s policy, theft above a set threshold is an approved purpose for a search, so the detective submits a request through the one registered system, the state service, rather than any tool found online.

The request goes to a small team of trained examiners rather than being run by the detective directly. An examiner first judges whether the probe image is good enough to search at all. The policy sets a minimum standard; a blurry side view is rejected, and the rejection is logged. If the image passes, the examiner runs the search and reviews the candidate list, comparing features by eye. The examiner records which candidates were considered, which were ruled out and why.

What goes back to the detective is clearly labeled as an investigative lead, not an identification. The form says, in plain words, that the result cannot on its own justify an arrest, a search warrant or a public statement. The detective then has to find separate evidence: a witness, a receipt, phone records, a vehicle sighting. Only if that corroboration exists does the case move forward, and the case file notes that facial recognition was used.

Every step leaves a trace. The vendor’s system logs the search under the contract. The examiner’s notes are filed with the case. Each quarter, a supervisor outside the unit pulls a sample of searches and checks them against the policy: was the purpose approved, did the image meet the standard, was corroboration obtained before action? Each year, the county publishes how many searches were run and how many led to an arrest.

Nothing here is exotic. It is the same discipline organizations already apply to cash handling or evidence storage: clear authority, separation of duties, records and independent checks. The difference is that many agencies had not yet recognized that an AI tool needed it.

What it costs

The process is slower than a detective running a search at a desk. It needs trained examiners, which a small agency may not have; many will route requests through a state or regional center instead. It needs someone outside the unit to audit, and someone to compile the annual figures. Those are real costs in staff time. Against them are set the cost of a wrongful arrest, a suppressed case or a lost lawsuit, any one of which can exceed years of oversight effort.

Fraud scores, credit models and triage tools

Facial recognition was the most visible case, but the three controls apply to any AI that supports decisions about people. Benefits fraud scoring, identity verification and eligibility screening carry similar risks of error falling unevenly on different groups. A fraud score that wrongly flags a family can delay the benefits they depend on. An identity check that works poorly for some faces can lock people out of services. An eligibility screen that encodes old patterns can deny help to those it was meant to reach.

Public agencies had already seen what happens without these controls. Michigan’s automated unemployment fraud system, introduced in 2013, wrongly accused many thousands of claimants of fraud before the state reviewed it, and litigation over the failures ran for years. The pattern was the one described above: an automated output treated as a finding, with no human check and no audit until the damage was done.

Banks already had the template

The same is true in the private sector. Banks use models to approve credit and spot fraud. Insurers use them to price policies and triage claims. Hospitals use them to prioritize patients. Employers use them to screen applicants. In each case, an organization that cannot list its AI tools, state the rules for using them and enforce those rules through contracts has handed its policy to its suppliers.

Banks were better placed than most, because supervisors had required something similar for a decade. The Federal Reserve and OCC’s 2011 guidance on model risk management expects banks to keep a model inventory, validate models independently, and govern vendor models as well as their own. Fair lending law adds rules on outcomes. Under the Equal Credit Opportunity Act, a lender that denies credit must give the applicant specific reasons, whatever model it used. A bank that has applied that discipline to credit models already has most of the three controls. The gap is usually in newer tools, such as identity verification with face matching, that arrived through operations or fraud teams rather than through model risk management.

Health systems and agencies start further back

Hospitals and health systems had fewer explicit requirements in 2020. Clinical decision support tools, many embedded in the electronic health record, could be switched on by a vendor update without a formal review of how they performed on the hospital’s own patients. The controls translate directly: list the tools, set rules for how clinicians use their outputs, test performance locally across patient groups, and write contracts that require disclosure and notice of changes.

Benefits agencies, motor vehicle departments and tax authorities face the identity question most directly. Remote identity verification that compares a selfie with a document photo is a one-to-one match, with lower risk than investigative search. It still fails some people more than others, and a failure can block access to money a household needs. An agency that adopts it needs an alternative route that does not depend on the face match, and data on who ends up using it.

Why it matters to the business

The business stakes are real. Errors that harm people produce lawsuits, settlements and regulatory attention. A vendor’s sudden withdrawal can disrupt operations. Public trust, once lost over a single wrongful outcome, takes years to rebuild. And elected officials, boards and regulators increasingly expect leaders to answer basic questions about the AI their organizations rely on.

Agencies that build the inventory, rules and contract terms once can scale them to each use by risk. A tool that suggests which potholes to fix first needs light controls. A tool that helps decide who gets investigated, arrested or denied benefits needs the full set. Waiting for a vendor announcement or a statute leaves the policy to someone else.

Answers without scrambling

A well-governed agency or company can do the following without scrambling:

• Produce a current list of every AI tool that touches decisions about people, including trials, embedded features and partner systems.
• Show the written rules for each high-risk tool, who approved them and when they were last reviewed.
• Point to evidence of accuracy for the actual tool, on its own kind of data, for the stated purpose.
• Show audit results that test whether staff follow the rules.
• Produce the contract clauses that guarantee logging, disclosure, change notice and an orderly exit.

The failures are predictable. Relying on vendor accuracy claims. Treating a free trial as harmless. Letting the team that wants the tool write the rules alone. Treating a candidate list as an answer. Signing contracts that say nothing about logs, data use or what happens if the supplier leaves. And assuming that because nobody approved a capability, nobody is using it.

Measures a council or board can follow

A few numbers show whether governance is real. The count of registered tools, and how many were found outside the register in the last review. The share of high-risk tools with current local accuracy evidence. The number of searches or automated decisions audited each quarter, and the rate of policy breaches found. The number of contracts that contain the required clauses. For facial recognition specifically, the number of searches run, the share rejected for poor image quality and the number that led to an arrest after corroboration.

Collecting them needs no advanced analytics. It needs an owner, a regular schedule and a willingness to publish results that may be unflattering. A falling count of tools found outside the register is one of the clearest signs that the inventory rule is working.

Who is writing your AI policy

1. Do we use facial recognition, or any AI that helps make decisions about people, and how do we know the list is complete?
2. What independent evidence do we have of its accuracy on our own kind of data, and for which groups of people?
3. Can a match or score ever be the only basis for action against a person, and what corroboration do we require?
4. Do our contracts give us search logs, algorithm disclosure, change notice and limits on how the vendor uses our data?
5. If the vendor withdrew tomorrow, would we keep our data and logs, and could our work continue?

If the honest answers are vague, the policy is being written somewhere else.

With hindsight

Amazon extended its moratorium on police use of Rekognition indefinitely in 2021. What began as a one-year pause became an open-ended one, confirming that agencies relying on a single supplier could not count on its product being available. Microsoft went further in June 2022, when it published a revised responsible AI standard. It restricted access to its face identification services and retired features that inferred emotional states and attributes such as gender and age from faces.

The inventory gap proved as wide as it had looked. In June 2021, GAO reported that only one of the federal agencies using non-federal face recognition systems knew which ones its employees used. It also found that several agencies had used the technology on images from the unrest that followed George Floyd’s death. GAO’s recommendations to track use and assess risk were largely taken up over the following years, though some remained open well into the 2020s.

Several cities and states passed their own restrictions on government use of facial recognition. Other agencies kept using services from smaller vendors, including a tool built on images scraped from the internet that drew regulatory action in several countries. “Scraped” means the images were collected automatically from websites and social media, without the knowledge of the people pictured. A 2022 settlement under Illinois’s biometric privacy law restricted that company from selling its database to most private businesses nationwide. The result was a patchwork: the same investigative technique could be banned in one city, restricted in another and unregulated in a third.

Wrongful arrests based on facial recognition matches, reported from 2020 onward, led to lawsuits and settlements that required changes to police practices. These were the failures the use rules described above were meant to prevent: a candidate list treated as an identification, without independent corroboration. Each case carried a human cost for the person arrested and a financial and reputational cost for the agency. In June 2024, Detroit settled the lawsuit over its January 2020 arrest. The settlement set safeguards on when police could use the technology, barred arrests based on a facial recognition result alone, required training and disclosure to the defense, and gave a federal court four years of oversight.

Regulators also turned to private-sector use. In December 2023, the Federal Trade Commission barred the pharmacy chain Rite Aid from using facial recognition for surveillance for five years. The agency alleged that the system had generated false matches against customers and that the company had failed to test it, monitor it or train staff. The order read like a list of the three missing controls.

Federal AI governance developed in stages. A December 2020 executive order on trustworthy AI in government asked agencies to inventory their AI uses. OMB, the White House Office of Management and Budget, issued government-wide requirements for agency AI use in March 2024, with added safeguards for uses affecting rights and safety. It replaced them with revised memoranda in April 2025 that kept inventories and risk management for high-impact uses. Under both versions, the inventory, use rules and contract terms agencies lacked in 2020 are the core obligations.

Sector rules followed the same pattern. In 2022 the Consumer Financial Protection Bureau said that lenders using complex algorithms must still give applicants specific reasons for adverse decisions. A 2024 HHS rule under Section 1557 of the Affordable Care Act required covered health programs to make reasonable efforts to identify and reduce the risk of discrimination from patient care decision support tools. In early 2022, the IRS moved away from requiring taxpayers to verify their identity through a third-party face match after public criticism. Outside the United States, the EU’s AI Act banned building facial recognition databases by untargeted scraping and tightly limited real-time remote identification by police in public places, with those prohibitions applying from February 2025.

The June 2020 episode is a durable warning in any sector. When you do not set the rules for the AI your organization uses, someone else will, whether a supplier responding to public pressure, a legislature responding to a scandal, a regulator responding to a complaint or a court responding to a lawsuit. Building the inventory, the rules and the contract terms is slower and less dramatic than any of those. It is also the only route that leaves the decisions with the people accountable for them.