NIST’s bias study made face recognition a buyer’s problem

NIST’s December 2019 study found that demographic error rates depended on the algorithm, the task, the threshold and the photos. That moved the question from whether face recognition is biased to whether an agency can specify, test and control the system it buys.

Looking Back revisits past weeks one at a time, up to the most recent, and asks what it meant for the banks, health systems and public agencies we serve. The series was written in 2026, with the benefit of hindsight, and each article says which week it looks back on.

Two-minute summaryThe dek and the first sentence of each section, quoted from the article

NIST’s December 2019 study found that demographic error rates depended on the algorithm, the task, the threshold and the photos. That moved the question from whether face recognition is biased to whether an agency can specify, test and control the system it buys.

  1. The week On December 19, 2019, the National Institute of Standards and Technology published Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects, numbered NISTIR 8280.
  2. How a match becomes an error A face recognition algorithm does not recognize anyone in the everyday sense.
  3. Specify: writing down the system the agency will actually run The first decision is the specification, and it is a step that can be skipped.
  4. Select: testing bidders on the agency’s own conditions NIST’s published results were the obvious place to start an evaluation.
  5. Sustain: keeping the tested system the system in service Face recognition vendors update their algorithms.
  6. Contain: use that never went through procurement The fourth decision concerns use the agency never bought.
  7. Where the argument is weakest The strongest objection is capacity.
  8. How it differs by kind of agency Federal agencies running large identification systems are the case the report most directly addressed, since its images came from State, DHS and the FBI.
  9. With hindsight NIST kept measuring.

The week

On December 19, 2019, the National Institute of Standards and Technology published Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects, numbered NISTIR 8280. NIST is the Commerce Department agency that sets measurement standards and runs independent technology tests. Its face recognition program had already reported on the accuracy of one-to-one and one-to-many algorithms. This third report asked a narrower question: do error rates differ by sex, age and race or country of birth, and by how much.

The scale was large. NIST ran 189 mostly commercial algorithms from 99 developers, which its release described as a majority of the industry. It used four sets of photographs from systems the government operates: domestic mugshots, application photos from people seeking immigration benefits, visa application photos and photos of travelers crossing the border. In all, 18.27 million images of 8.49 million people were processed. The images came from the State Department, the Department of Homeland Security and the FBI. None were scraped from social media or taken from video surveillance.

The datasets carried sex and age for each person. The mugshots also recorded race, while the other sets recorded only country of birth. For those, NIST limited its analysis to 24 countries in seven regions that had seen lower levels of long-distance immigration, treating country of birth as a reasonable proxy. The report itself ran to 81 pages. NIST supplemented it with more than 1,200 pages of charts in seventeen annexes, reporting results for each algorithm. It named its intended audience as policy makers, algorithm developers, systems integrators and managers of face recognition systems.

The report tested the two tasks separately. One-to-one verification checks whether two photos show the same person, as when a phone unlocks or a passport is checked. One-to-many identification searches a database for anyone who matches a probe photo, which is how an investigator or a fraud unit looks for a person of interest. NIST measured both kinds of error for each task: false positives, where two different people are wrongly matched, and false negatives, where the same person is missed.

The headline finding was that most algorithms showed demographic differentials, and that false positive differentials were much larger than false negative ones. Across demographic groups, false positive rates often varied by factors of 10 to beyond 100. False negative differences tended to be specific to the algorithm, and often varied by factors below three. False positives were higher in women than in men, and higher in the elderly and in children. Image quality drove many false negative effects. The border crossing photos, which do not meet capture standards, behaved differently from the mugshots, which were taken with a standardized photographic setup.

Two further findings pointed toward buyers rather than developers. Results varied widely between developers, and some supplied identification algorithms whose false positive differentials NIST could not detect. NIST’s release noted that the most equitable algorithms were among the most accurate. And the report’s section on implications said that because operational systems usually run a single algorithm, it is incumbent on the system owner to know their algorithm. It suggested measuring the operational algorithm on the operational image data, perhaps with a biometrics testing laboratory. It also warned that NIST tests algorithms submitted as prototypes, not the finished commercial products agencies actually buy.

The December 2019 report is easy to read as a verdict on a technology. On that reading, face recognition was biased, or it was not, or it was improving. For a public agency, that framing was the least useful one available. The report showed that the answer depended on which algorithm, which task, which threshold and which images. Those are not facts about face recognition in general. They are facts about a particular product deployed in a particular way, and they are fixed, or left open, in procurement.

Our argument is that NIST’s report moved responsibility for demographic performance from the research community to the buyer. An agency that runs face recognition owns the error rates of the system it bought, at the settings it chose, on the photos it feeds it. Those error rates are decided by four procurement decisions, in order: what the agency specifies, how it selects among bidders, what it keeps stable after award, and how it contains use the procurement never covered. We take each in turn after a short account of the mechanism. A sixth section tests the argument against the strongest objections, and the last looks at how it plays out for different kinds of agency.

The sector here is government because every image set in the report came from systems that government agencies operate. Banks and health systems that use face matching for customer onboarding face a version of the same question for one-to-one checks, and some of what follows applies to them. Agencies should check their specific obligations with counsel, especially where state law restricts face recognition.

How a match becomes an error

Scores, thresholds and the two kinds of mistake

A face recognition algorithm does not recognize anyone in the everyday sense. According to NIST’s report, it detects a face, converts it into a template of numbers, and compares two templates to produce a similarity score. The score is a vendor-defined number. It is compared with a threshold to decide whether the two images show the same person. Raise the threshold and false positives fall while false negatives rise. Lower it and the reverse happens.

That single dial explains why a general verdict on bias is hard to act on. A demographic differential is a difference in error rates between groups at a given threshold. Move the threshold and the rates move. NIST observed that most systems run at a fixed threshold, and that reports of accuracy rarely give both false negative and false positive rates for each demographic group at that threshold. Most reports, it said, dealt only with false negatives.

Why the task matters

The consequences of each error depend on the task. In one-to-one verification, NIST noted, a false negative is usually an inconvenience: the person tries again. A false positive is a security problem, because an impostor gets in. In one-to-many identification, a false positive puts the wrong person on a list of candidates for human review. NIST’s release pointed out that the consequences could include false accusations.

One-to-many systems return either a fixed number of the most similar candidates or only those above a threshold. That design choice matters too. A system that always returns, say, the top twenty candidates will return twenty names even when the person is not in the database. Every one of those names is, by definition, a false candidate. The human reviewer then carries the whole burden of avoiding an error.

Why image quality matters

NIST traced many false negative effects to the photos rather than the algorithms. The mugshots were taken with a setup standardized to produce good images across races. The border crossing images were not, and there the false negative pattern changed. An agency that buys an algorithm on the strength of results from good photos, and then runs it on poor ones, is buying a different system from the one it evaluated.

These three variables, threshold, task and image quality, are all set by the operator. That is why the report’s implications section addressed system owners directly. The developer controls the algorithm. Everything else that determines the error rates belongs to the agency.

What the report did not claim

The report was careful about its limits, and those limits matter to a buyer. It did not explore what caused the differentials, though NIST’s release suggested that training data was one area for research. It tested algorithms, not products. And it drew a sharp line between face recognition and other face analysis.

That last distinction is easy to lose. NIST noted that much public discussion of bias cited studies of gender classification algorithms, which estimate a person’s sex from a face. Those studies did not test recognition algorithms, which compare two faces. NIST recommended that any report of bias say which class of algorithm was evaluated. An agency buying a recognition system should not rely on classification results, favorable or unfavorable. It needs evidence about the class of system it is buying.

The report also stated that demographic effects can be advantageous or disadvantageous depending on the application. A higher false negative rate for one group is a nuisance at a phone lock and a barrier at a benefits portal. A higher false positive rate is a security gap in access control and a risk of wrongful suspicion in an investigation. The same measured differential can call for different responses. That judgment depends on the use, and only the agency knows the use.

Specify: writing down the system the agency will actually run

Name the task and the population

The first decision is the specification, and it is a step that can be skipped. A requirement that says “facial recognition capability” tells a bidder almost nothing. The NIST findings suggest at least four things a specification should state. The task, one-to-one or one-to-many. The size and type of the gallery, if there is one. The kind of images that will be submitted, with examples of their quality. And the population the system will serve.

The population point is easy to miss. NIST found that false positive rates varied with country of birth, sex and age, and that the elderly and children had elevated false positives. A benefits agency serving older residents, a school safety program and a border program serve different populations. A specification that ignores this invites a bidder to quote performance on the population of its own test set.

Specify error rates per group, at the operating threshold

The second element is the performance requirement itself. NIST’s complaint about incomplete reporting gives the template. Ask for false positive and false negative rates, for each relevant demographic group, at the threshold the agency intends to use. A single overall accuracy figure hides exactly what the report measured. So does a false negative rate on its own.

Agencies may not know in advance which threshold they want. In that case the specification can ask bidders to report performance across a range of thresholds. The agency then picks its operating point later, with evidence, and records why. That record becomes important when someone later asks why the system flagged a particular person.

Decide what the human does

A one-to-many specification should also set the human review process, because the algorithm’s output is a candidate list and nothing more. Who reviews it, with what training, and what may be done with a candidate before independent corroboration are policy questions. They still belong in the procurement file, because they change what the agency is buying. A system bought to generate investigative leads needs different safeguards from one bought to make automated eligibility decisions.

A useful discipline is to write the specification so that a skeptical reviewer could reconstruct the operating conditions from it alone. If the document does not say what images go in, what threshold applies and who reviews the output, then nobody can later show that the system performed as intended. The specification is the agency’s first piece of evidence.

Choosing the operating threshold

Most of the specification describes conditions. The threshold is a choice, and it is the one with the most direct effect on demographic outcomes. The report’s findings imply a way to make it. Start from the consequence of each error in the agency’s use. Then pick the point at which the more harmful error is acceptably rare for every group measured, and on average, and accept the cost in the other error.

For one-to-many searches there is a further consideration. Each search compares the probe with every enrolled template, or with a large share of them. The larger the gallery, the more comparisons there are with people who are not the person sought. A threshold that produces few false matches in a small gallery may produce more in a large one. An agency planning to grow its gallery should test at the size it expects to reach.

The threshold decision should be recorded with its reasons and an owner. It should be revisited when the gallery, the images or the use change. A threshold set once at installation and never examined is a decision nobody made.

A route for people the system fails

Every face recognition system fails some people, and the NIST results showed that failure is not spread evenly. A specification should therefore require a fallback for anyone the system cannot verify or wrongly flags. For a benefits or licensing agency, that means an alternative way to prove identity that is reasonably quick and does not penalize the person for the machine’s error. For an investigative use, it means a rule that a candidate cannot be the sole basis for action.

The fallback has costs, in staff and in fraud exposure, and those costs belong in the business case from the start. An agency that budgets for the biometric path and not the fallback will find the fallback underfunded. People in the groups with higher error rates will then wait longest.

Who writes it

A specification this detailed cannot come from the contracting office alone. We would expect it to need at least five contributors. The program owner defines the task and the human review step. The privacy officer sets limits on the images and on retention. A civil rights or equal opportunity office defines which groups to measure and what differential is acceptable. The chief information officer’s team covers integration, logging and security. Counsel reviews the use against any state or local restrictions.

The sequencing matters. If the specification is drafted after a vendor has been chosen informally, it risks describing that vendor’s product. Drafting it first, with these roles signed off, gives the agency a neutral yardstick. It also produces the documented decisions an inspector general or a court would later look for. A short decision memo for each choice, such as threshold, gallery and review rule, is enough. It costs little and saves a great deal when the system is questioned.

Select: testing bidders on the agency’s own conditions

Public results are a filter, not a verdict

NIST’s published results were the obvious place to start an evaluation. They covered most of the industry, and they used government images. But the report itself warned against treating them as the last word. NIST tests algorithms submitted as prototypes. It does not test the finished products that agencies buy, which wrap an algorithm in software, workflow and default settings. The algorithm in a vendor’s product may also be a different version from the one NIST tested.

The sensible use of the public results is as a screening step. An agency can require bidders to identify which submitted algorithm their product uses and to show where it ranked. That narrows the field to developers with demonstrated accuracy. NIST’s observation that the most equitable algorithms were also among the most accurate suggests the screen does double duty.

Test on operational data, with care

The second step is the one the report recommended: measure the operational algorithm on operational image data. In procurement terms, that means a test during evaluation, before award. An agency can run candidate products against a sample of its own images, under its own threshold settings, and measure both kinds of error by group. Where it lacks the skill, the report suggested a biometrics testing laboratory.

Testing on real images raises its own obligations. The images are personal data, and may be sensitive. The test needs a privacy review, a data handling agreement with each bidder and a plan for deleting the data afterwards. Ground truth is also needed: the agency must know which image pairs are genuinely the same person. Building that set is the slowest part of the work, and it should start before the solicitation is released.

Score what was measured

The evaluation criteria should give weight to measured differentials as well as to overall accuracy and price. Otherwise the test produces evidence that the award ignores. A simple approach sets a ceiling on the ratio between the highest and lowest group false positive rates at the operating threshold. Bidders above the ceiling are not technically acceptable. Among the rest, overall accuracy and cost decide.

This is also where an agency learns what it cannot measure. Small subgroups may not have enough images for a stable estimate. The honest record says so, and states what the agency will monitor after deployment to fill the gap. A test that claims more precision than its sample supports gives false comfort, and it will not survive an audit.

What the evaluation file should hold

At the end of selection, the agency should have a file that would let an outsider check the award. At a minimum it should contain the following:

  • The specification, with the sign-off of each contributor.
  • The NIST results each bidder relied on, with the algorithm each product actually uses.
  • The test design: images used, how ground truth was established, thresholds tested and groups measured.
  • The results for each bidder, with both kinds of error by group and the uncertainty for small groups.
  • The scoring, including any bidder excluded for exceeding the differential ceiling.
  • The privacy review and evidence that test data was deleted or retained under agreement.

These documents form the ordinary record of any technical evaluation. What the NIST report changed was the content: demographic performance moved from a footnote to a scored criterion with evidence behind it.

Sustain: keeping the tested system the system in service

Algorithms change under contract

Face recognition vendors update their algorithms. Each update can change error rates, and therefore demographic differentials. NIST’s program page says its evaluation runs continuously, and developers may submit a new version as often as every four calendar months. Improvement is welcome. It also means that the system tested at award may not be the system running two years later.

The contract should therefore define the tested configuration and treat changes to it as controlled. That includes the algorithm version, the threshold, the candidate list length and any image quality filters. A change to any of them should trigger notice to the agency and, for material changes, a retest on the agency’s reference set before it goes live. The contract should say who pays for the retest.

Evidence the agency can keep

Agencies also need the evidence to survive the vendor relationship. The reference test set, the results at award and the results of each retest belong in the agency’s own records as well as the vendor’s. So do the system’s settings over time. When a court, an inspector general or a legislature asks how the system performed, the agency should be able to answer without asking the vendor for help.

Logging is part of this. For one-to-many searches, the agency should be able to reconstruct who ran a search, on which image, at what threshold, what candidates came back and what the reviewer decided. Those records allow the agency to measure performance in operation, which is the only place real demographic effects show up. They also allow it to find misuse.

Owning the vendor relationship

These terms only work if someone in the agency enforces them. Change notices must be read, retests scheduled and results compared with the baseline. If the contracting officer’s representative is a program manager with other duties, those tasks slip. Face recognition contracts need a named owner with the technical support to interpret test results, and a budget line for retests. Without that, the change control clause becomes a filing exercise.

Exit and portability

A further contract issue is portability. NIST describes the similarity score as a vendor-defined number, and templates are produced by each vendor’s own extractor. If they cannot be carried across, a gallery would have to be re-enrolled from the original images to move. If the agency has not kept those images and the rights to use them, switching to a more accurate or more equitable algorithm becomes expensive. The contract should preserve the agency’s ability to re-enroll elsewhere.

Without that clause, the procurement that was meant to choose the best algorithm locks the agency into it. NIST’s results made clear that algorithms improved quickly. An agency that cannot move cannot benefit from that improvement, and its differentials stay where they were at award.

Monitoring in operation

Pre-award testing estimates performance. Operation reveals it. An agency should decide in advance what it will measure once the system is live, and who will look at the numbers. Some measures are straightforward. For one-to-many searches, the agency can track how often reviewers reject the top candidate, how often a candidate is later confirmed or excluded, and how many complaints and corrections arise.

Demographic monitoring is harder, because the agency may not know the demographics of a probe image. Where the gallery holds that information, as mugshot records may, results can be broken down by the candidate’s recorded attributes. Where it does not, periodic retesting on the reference set is the fallback. Either way, the measure needs an owner outside the team that runs the system. A program that measures itself risks finding what it hoped.

The monitoring cadence should follow the risk. A high-volume investigative system deserves quarterly review. A low-volume access control system may need only an annual check. The schedule should be written into the program’s governance documents, so that it survives staff turnover and changes of leadership.

Contain: use that never went through procurement

Outside services and other agencies’ systems

The fourth decision concerns use the agency never bought. Face recognition can be reached outside procurement, through other agencies’ systems and commercial services. An investigator who runs a search on a service the agency has no contract with bypasses every control in the three previous sections. There is no specification, no test, no version control and possibly no log the agency can see.

Containment starts with policy: staff may use only systems the agency has approved, and approval requires the steps above. It continues with an inventory of the systems actually in use, which means asking staff and checking expense records as well as the contract register. It ends with consequences for use outside the approved list.

Shared systems need shared terms

Some agencies, especially smaller ones, search a state or federal partner’s system rather than running their own. That arrangement can be sensible. It moves the testing burden to a better-resourced partner. But it should be written down. The agreement should state the partner’s algorithm and settings and the testing it has done. It should also state the logs the requesting agency can see and the review required before a candidate is acted on.

The same logic applies in the other direction. An agency that lets others search its gallery is responsible for how its system is used. It should know who searched, why, and with what result. A memorandum of understanding that says nothing about thresholds or human review leaves both sides exposed.

Retiring systems that cannot be evidenced

Some agencies will find, on inventory, systems for which no test exists and none can now be run. The choice is then between testing them retroactively, restricting their use, or retiring them. Any of the three is defensible if it is recorded. Continuing to use an untested system because it is already in place is the option that cannot be defended after an error.

Signs that use is escaping

Some signals suggest that use has moved outside approved systems. Investigative reports may cite face matches with no corresponding log entry. Small recurring charges for online services may appear on purchase cards. Staff may ask for help exporting images in bulk. Partners may mention searches the agency cannot see. No single signal proves misuse. Each is a reason to ask, and the agency’s policy should make clear who asks and what happens next.

Where the argument is weakest

Agencies may not be able to test

The strongest objection is capacity. A county sheriff’s office or a small benefits agency cannot build a ground-truth test set, run a structured evaluation and maintain version control over a vendor’s algorithm. Telling them to do so is telling them not to use the technology at all.

That objection is partly right. The full approach suits a large agency running its own system. For smaller ones, the answer is to buy through, or search through, a partner that has done the work, and to insist on seeing it. Containment then becomes the main task. A small agency can still specify the task, require the vendor to disclose the algorithm and settings, and keep logs. Those steps cost little and close the largest gaps.

The vendor will not allow it

A second objection is that vendors resist testing on agency data, resist disclosing algorithm versions and resist change control. Some may. But an agency has the most bargaining power it will ever have before award, with competitors waiting. The NIST report gave agencies a public, neutral basis for asking. A vendor that will not say which algorithm its product uses, or allow it to be tested on the buyer’s images, has told the buyer something important.

The question is policy, not procurement

A third objection is that the true decision is whether to use face recognition at all, and that procurement framing launders a political question. Procurement cannot settle whether a use is legitimate, and we do not suggest it can.

But where an agency has decided to use the technology, or is required to, the error rates are still decided in procurement. A policy that permits use without saying how systems are specified, tested and controlled leaves those decisions to the vendor by default. The procurement framing does not replace the policy question. It answers the one that remains after the policy question is settled.

Better algorithms will make this moot

A fourth objection is that algorithms improve so fast that differentials will disappear, making the testing apparatus unnecessary. NIST did find that some identification algorithms had undetectable false positive differentials. Improvement is real. But the report also showed that image quality, thresholds and task design produce effects no algorithm removes. Poor photos stay poor. A badly chosen threshold stays badly chosen. Those remain the operator’s responsibility however good the algorithm becomes.

It costs too much

A fifth objection is cost. Building a reference set, running a structured test and retesting after updates take staff time and, possibly, outside help. Change control may raise the contract price, since vendors bear the cost of retests and slower releases. For a small program, these costs may approach the price of the software itself.

The answer depends on what the system does. For a low-stakes, one-to-one convenience feature with a non-biometric alternative, a lighter approach is reasonable. Public results, a vendor attestation and monitoring may suffice. For one-to-many searches that can put a person under suspicion, the cost of testing should be weighed against the cost of a wrongful identification. That cost falls on the person first, and then on the agency through complaints, litigation and lost public trust. The more a system can harm, the more evidence the agency should hold before using it.

Taken together, the objections narrow the argument without overturning it. They show that the full approach suits large programs and high-stakes uses, and that smaller agencies need shared testing and simpler controls. They do not show that anyone other than the operator can own the error rates. Every objection still leaves the threshold, the images and the use in the agency’s hands.

How it differs by kind of agency

Federal agencies with large galleries

Federal agencies running large identification systems are the case the report most directly addressed, since its images came from State, DHS and the FBI. They have the data to run operational tests and the scale to justify them. Their weaker point can be the edges: components and task forces using outside services, and partners searching their galleries under old agreements. For them, containment and shared-system terms deserve as much attention as selection.

Large agencies may also face a sequencing problem. A system bought before the report may run under a contract that says little about versions or retesting. Rebidding is slow. The practical path is to run the operational test on the system in service, record the result, and add change control at the next modification or option exercise. That gives the agency evidence now and contract terms later, instead of waiting years for a new award.

Federal agencies are also the natural source of shared testing. An agency with a large, well-labeled gallery and a testing team can measure products on behalf of others, within the limits of privacy law and its own policies. Doing so would spread the cost the capacity objection raises. It would also give smaller agencies the evidence they cannot produce alone.

State identity and benefits agencies

State motor vehicle agencies use one-to-many searches to detect people applying for licenses under more than one identity, a use the NIST report described. Benefits agencies may use one-to-one checks to verify applicants remotely. Both serve the whole adult population, including the older residents among whom NIST found elevated false positives. For them, the specification’s population clause and the threshold record matter most, along with a fallback route for anyone the system fails.

Local law enforcement

For local agencies that rely on a partner’s system or a commercial service and have no testing staff, the priorities are an approved-systems list, a written agreement with any partner, logging, and a rule that a candidate is a lead and not an identification. Those controls depend on discipline rather than budget.

Agencies buying through shared contracts

Some agencies buy through statewide or cooperative contracts negotiated by someone else. Those vehicles save time, but they may fix terms that the four decisions need, or leave them out. An agency ordering from such a contract can still add its own specification and acceptance test at the order level. Where the vehicle prevents that, the agency should raise it with the contract holder before relying on it. The contract holder may be best placed to run the test once, for every agency that orders.

Banks and health systems

Outside government, banks and health systems that use face matching for remote account opening or patient identity checks are running one-to-one verification. Their exposure differs. A false negative locks out a legitimate customer or patient. A false positive lets an impostor through. Differentials in either direction affect fair treatment, and lenders should ask counsel whether fair lending rules are engaged. The same four decisions apply. The emphasis falls on specifying error rates by group and on keeping a non-biometric fallback that does not disadvantage those the system fails.

Across all of these, the test of a procurement is whether the agency can answer three things from its own records. The first is which algorithm is running today, and at what settings. The second is how it performed on the agency’s images, by group, at the last test. The third is who used it outside the approved path. An agency that can answer them has taken on the responsibility the report assigned. One that cannot has left it with the vendor.

With hindsight

NIST kept measuring. In July 2022 it published NISTIR 8429, which summarized demographic differentials across algorithms. It later split and renamed the program: FRVT became the Face Recognition Technology Evaluation (FRTE) and the Face Analysis Technology Evaluation (FATE). Its demographics page continued to publish summary tables, updated as recently as March 2025. Those tables fix each algorithm’s threshold at a stated false match rate. The page adds that false negatives depend strongly on image quality, citing under-exposure of dark-skinned faces. It says false positive differences can arise when a group is under-represented in the images used to train an algorithm. The operator-side variables the 2019 report emphasized remained part of NIST’s own explanation.

Oversight confirmed the containment problem. In June 2021, GAO reported on 42 federal agencies that employ law enforcement officers. Twenty owned face recognition systems or used systems owned by others. All fourteen that used it for criminal investigations used systems owned by non-federal entities. Only one knew which non-federal systems its employees used. GAO made two recommendations to each of 13 agencies: track the non-federal systems employees use, then assess their privacy and accuracy risks.

Progress was uneven. GAO’s recommendation page, in an Internet Archive copy captured in October 2026, records that the FBI issued a policy directive in August 2022 covering face recognition procurement, tracking and evaluation. Personnel had to coordinate procurement with its Science and Technology Branch. A December 2023 Justice Department interim policy required components to implement tracking. Yet the same page records that, as of March 2026, the ATF and the DEA had not assessed the risks of the non-federal systems they used. Of the 26 recommendations, 17 were closed as implemented, three were open and two were open but partially addressed. GAO closed the other four, risk assessments it had recommended to the Fish and Wildlife Service, the U.S. Park Police, the IRS Criminal Investigation Division and the U.S. Capitol Police, as “no longer valid,” recording in each case that the agency was not using, or had no plans to use, non-federal face recognition technology.

Broader guidance moved toward the buyer as well. In 2024 the National Academies published a consensus study finding that the technology had outpaced laws and regulations. It recommended ways for agencies deploying it to mitigate harm. In April 2025, OMB memorandum M-25-21 presumed several uses to be high-impact AI. They included biometric identification in law enforcement and one-to-many identification in publicly accessible spaces. For high-impact uses it required pre-deployment testing, an impact assessment and ongoing monitoring. Where an agency lacks access to a vendor’s model or data, the memo required alternative methods, such as giving the vendor evaluation data and obtaining the results.

Its companion, M-25-22, turned that into acquisition practice. Agencies must, to the greatest extent practicable, test proposed AI solutions when evaluating proposals. They should consider test environments on their own networks, and should guard against vendor lock-in through data and model portability terms. OMB said it would develop playbooks for specific kinds of AI, naming AI-based biometrics among them. M-25-21 rescinded and replaced the earlier M-24-10, and both April 2025 memos are cited here as issued.

What held up was the 2019 report’s core point: performance belongs to a specific system in a specific setting, and the owner must know it. What took longest was the unglamorous part. Inventories, tracking and risk assessments of systems staff used outside procurement were still incomplete at some agencies in March 2026.

Spotted an error? Tell us and we will check it and correct the article where needed.

  • face recognition
  • NIST
  • procurement
  • biometrics
  • AI governance