A weekend of public red-teaming showed agencies that AI testing never finishes

The week

Over the weekend of August 10 to 13, 2023, the DEF CON security conference in Las Vegas hosted a public red-teaming exercise against large language models from several leading AI developers. In a convention hall filled with rows of laptops, each participant sat with a timer and a simple brief: get the chatbot in front of you to do something it should not. Make it give dangerous advice, state a falsehood with confidence, say something biased, or slip past the safety rules its makers built in.

A large language model is the technology behind chatbots such as ChatGPT: software trained on vast amounts of text to predict and generate fluent language in response to a prompt. “Red-teaming” is a term borrowed from the military and from security work. A red team plays the adversary, deliberately attacking a system to find its weaknesses before real adversaries do.

The exercise was organized by the conference’s AI Village with partners, and the White House had announced in May that leading AI developers would submit their models to it. Participants had timed sessions to make the models produce harmful, false or biased output, or to break their safeguards, with results collected for later analysis. The aim was to see what happens when many different people, with different backgrounds and ideas, push on the same systems at once, rather than to embarrass any one company.

The May announcement, a White House fact sheet issued on May 4, named the firms that had agreed to take part: Anthropic, Google, Hugging Face, Microsoft, NVIDIA, OpenAI and Stability AI. Testing would run on an evaluation platform built by Scale AI, consistent with responsible disclosure principles. The fact sheet said the exercise would show how the models aligned with two earlier federal documents. One was the White House’s Blueprint for an AI Bill of Rights. The other was NIST’s AI Risk Management Framework, a voluntary guide published that January.

The same fact sheet carried two other items that mattered for agencies. The National Science Foundation would fund seven new national AI research institutes. And the Office of Management and Budget would release draft guidance on federal agencies’ own use of AI for public comment that summer. That guidance did not arrive until November, after the President’s AI executive order. In August 2023, agencies buying AI had no government-wide rule on how to test it.

It was a busy week for AI and security in Las Vegas. Earlier that week, at the Black Hat conference, DARPA, the Defense Department’s research agency, announced the AI Cyber Challenge, a multi-year competition to build AI systems that find and fix software vulnerabilities, the flaws in code that attackers exploit to break in.

DARPA announced the challenge on August 9. It described a two-year competition, with a semifinal at DEF CON in 2024 and a final there in 2025, and cash prizes at each stage. Up to seven small businesses could receive DARPA funding to compete, while other teams could enter an open track at their own expense. Anthropic, Google, Microsoft and OpenAI agreed to make their AI technology and expertise available to competitors. The Open Source Security Foundation, a Linux Foundation project, served as an advisor, because so much of the code that critical infrastructure runs on is open source.

Both events followed the voluntary commitments the White House secured from major AI companies in July, which included internal and external security testing of models before release. Testing by outsiders was moving from a nice idea to an expectation.

Those July 21 commitments came from seven companies: Amazon, Anthropic, Google, Inflection, Meta, Microsoft and OpenAI. They committed to test their AI systems internally and externally before release, with part of the testing done by independent experts. They also committed to make it easier for third parties to discover and report vulnerabilities in their systems after release, and to share information on attempts to get around safeguards. The DEF CON weekend was, in effect, a large public trial of that second commitment.

The security community was writing its own rules at the same time. That summer, the OWASP open security project published its first list of the most serious weaknesses in applications built on large language models. Prompt injection, the trick of slipping new instructions into what a model reads, came first. The vocabulary that agencies would later need for contracts and test plans was being drafted in public.

For government agencies starting to buy and use these tools, the weekend raised a practical question: if outsiders can find new problems in an afternoon, who in the agency keeps looking, and for how long?

The DEF CON exercise showed something useful about large language models. Ordinary people, given an hour and some prompts, could make leading models misbehave in ways their developers had not anticipated. These were not obscure research systems. They came from the companies whose products agencies were beginning to consider.

The exercise had limits, and they matter for reading it fairly. Sessions were short. Participants worked on general-purpose models, not on any agency’s application, data or users. Findings went to the developers under responsible disclosure, and no model was certified or failed. What the weekend demonstrated was a method and a scale of discovery, not a verdict on any product.

For agencies, the more practical message was about workload. A model that can be broken in new ways by new people cannot be certified once and filed. Testing is a recurring activity, and every recurring activity in government needs budget, staff and authority. That workload falls into three stages, which together cover an AI system’s life in an agency: before award, before deployment, and in operation. Behind all three sits the question of why these systems are so hard to test in the first place.

Why AI chatbots are hard to test

Traditional software is predictable. If a benefits calculator is given the same inputs, it produces the same answer every time, and a tester can check that answer against the rules.

A large language model is different. It does not look up answers. It generates them, word by word, based on patterns learned from enormous amounts of text. The same question, phrased slightly differently, can produce a different answer. A question it handles well in English may go wrong in Spanish. And because users can type anything at all, the range of possible inputs is effectively unlimited. Testing a chatbot is more like interviewing a new employee: you can ask a hundred questions and still be surprised by the hundred-and-first.

Developers add safeguards, often called guardrails: extra instructions and filters meant to stop the model from producing harmful content. But guardrails can be talked around. A user might ask the model to role-play, to write a story in which a character explains something dangerous, or to ignore its previous instructions. This kind of manipulation is often called a “jailbreak.” The people best at finding jailbreaks are often not engineers. They are curious, persistent users who think about language and human behavior, which is exactly why a public exercise with many participants found things that small internal teams had missed.

When the model reads things, the attack surface grows

The DEF CON models mostly answered questions typed by the person in front of them. Agency uses rarely stay that simple. A system that summarizes case files, reads uploaded documents or searches the agency’s own website is also taking in text the agency did not write. Anyone who can place text in that stream can try to give the model instructions. A document with a hidden line telling the model to ignore its rules is a prompt injection, and the model has no reliable way to tell data from commands.

The risk grows with each connection. A chatbot that can only talk can mislead. One that can look up a case record can disclose it. One that can send a letter or update a field can act on a manipulated instruction. Testing has to cover what the system can reach and do, as well as what it says. A sensible design keeps the model’s permissions narrow, so that a successful attack has little to work with.

Models also state falsehoods fluently, a failure often called hallucination. A chatbot may cite a regulation that does not exist or quote a deadline from the wrong year, in the same confident tone it uses for correct answers. A common mitigation is to have the system retrieve passages from the agency’s own approved documents and answer from them. That narrows the problem without ending it, because the model can still misread or misquote what it retrieved. Testers have to check answers against the source, not against their own sense of what sounds right.

Measurement is hard too

Because the same prompt can produce different answers, a single test run proves little. Testers need to run the same case several times and look at the spread of results. They also need a way to grade answers. For factual questions, a reviewer can compare the answer with the correct one. For tone, fairness or helpfulness, grading takes judgment. Some teams use a second model to grade the first, which is faster but adds its own errors and needs spot checks by people.

Public benchmarks, the standard test sets researchers use to compare models, help with general capability. They say little about whether a model gives correct answers about one state’s eligibility rules. An agency’s own test set, built from its own cases, is the only measure of fitness for its own use.

There is one more complication. AI products change underneath their users. A vendor may swap in a newer model, adjust its filters or add new features without the interface looking any different. A system that passed every test in March can behave differently in June. That is the core reason testing has to be continuous.

What red-teaming looks like in practice

Inside an organization, red-teaming an AI system is less theatrical than a hall full of laptops, but the idea is the same. A small group is given time, a target system and a clear brief: find the ways this system could cause harm in our setting. They are rewarded for finding problems, not for confirming that everything works.

A typical session starts with a list of harm categories the organization cares about. For a public agency, that list might include wrong information about benefits or deadlines, advice that could hurt someone’s health or safety, unequal treatment of different groups, disclosure of personal information, and responses that could be used to defraud the program. Testers then try many approaches against each category: plain questions, tricky phrasing, role-play, other languages, long conversations that slowly drift off course. Every failure is written down with the exact prompt that caused it, so engineers can reproduce it and check later that the fix holds.

The best red teams mix backgrounds. A security specialist knows technical tricks. A caseworker knows how real applicants phrase questions. A lawyer spots answers that sound like legal advice. Someone fluent in another language tests whether safeguards hold outside English. The DEF CON exercise made this point at scale: diversity of testers is itself a testing method.

Rules of engagement

An internal exercise needs written ground rules before it starts. They set out which system and which version is in scope, which environment testers may use, and what data they may see. They say whether testers may try to reach connected systems or only the chat interface. They name who receives findings and how fast serious ones must be reported. Without these rules, a tester who finds a way to pull personal data has created an incident in the act of discovering it.

Findings are sensitive in their own right. A working jailbreak is a recipe others can reuse. Logs of test prompts may contain realistic personal details. The findings register should be protected like a vulnerability report, with access limited to those who need it.

Red-teaming of model behavior also differs from the penetration testing agencies already buy. A penetration test checks whether an attacker can break into the infrastructure: servers, networks, accounts. Behavioral red-teaming checks whether the system says or does harmful things through its normal interface. An AI system needs both. A clean penetration test says nothing about whether the chatbot invents eligibility rules.

Insiders, outsiders and the public

Internal testers know the program and can test continuously. They also grow used to the system and stop seeing its oddities. Outside red teams bring fresh tricks and no assumptions, at a price and for a limited time. The public is the largest test team of all, and it will find problems whether or not the agency asks.

That last group can be put to use. Federal civilian agencies were already required by a 2020 CISA directive to publish vulnerability disclosure policies, which tell outsiders how to report security flaws safely. Extending that channel to cover harmful AI behavior gives the agency a route for what the public finds. It also needs someone to triage the reports, which brings the question back to staffing.

The output is a list of findings ranked by severity, rather than a pass or fail grade, and it feeds decisions about fixes, limits on use, or whether to proceed at all.

Before award

Procurement is where an agency has the most influence over an AI vendor and usually the least testing capacity. Evaluation panels typically score written proposals and scripted demonstrations. Neither reveals how a model behaves when users push it, misunderstand it or try to abuse it. A scripted demonstration is the vendor’s best day, chosen by the vendor.

An agency can do better without building a laboratory. Solicitations can require vendors to describe their own red-teaming, including who performed it, what categories of harm it covered and what was fixed as a result. Evaluations can include a hands-on test period in which agency staff, including frontline users, try the system against realistic tasks and known failure patterns. Contracts can require notice of model changes and access for agency testing after award.

Who is responsible for which layer

Many products sold to agencies are built on a model from another company. The vendor writes the application, the instructions and the filters; the underlying model comes from a developer the agency never contracts with. Responsibility for testing is split the same way. The solicitation should ask the vendor which model it uses, which version, how it learns of changes from the model developer, and what testing it does on its own layer. A vendor that cannot answer has not thought about the problem.

Existing approvals do not fill this gap. A federal cloud service may hold a FedRAMP authorization, which assesses its security controls. That authorization says nothing about whether the model’s answers are accurate, fair or safe for a given program. Agencies sometimes treat an authorized platform as an approved tool. For AI, those are different questions, and the second one remains with the agency.

Each of these needs people who understand both the mission and the technology. A contracting officer cannot design a test of a benefits chatbot alone. A program office cannot judge a vendor’s red-team report without someone who knows what good testing looks like. Agencies that lack that person end up accepting vendor assurances by default.

There is a market effect too. When agencies ask the same testing questions in every solicitation, vendors learn to answer them well and to invest in the testing behind the answers. When each agency asks differently, or not at all, vendors optimize for demonstrations. Shared solicitation language across agencies, maintained by a central team, is cheaper than each agency writing its own and gives buyers a common baseline for comparing bids.

Contract terms that make testing possible

Testing after award depends on terms written before it. A short list covers most needs. The vendor gives advance notice, in a set number of days, before changing the underlying model, its safety filters or its system instructions. The agency gets a non-production environment where it can run its test set at no extra charge. The agency can see conversation logs and the vendor’s own monitoring data for its deployment. The vendor may not use agency data to train models without written permission.

Further terms deal with failure. The vendor reports serious AI failures within a defined time, as it would a security incident. It cooperates with testers the agency appoints, internal or external. And the agency can suspend or end the contract if a model change materially degrades performance against the agreed test set. Each term costs the vendor something, so the evaluation should give credit to bidders who accept them.

Different buyers, different starting points

A large federal department may have an AI office, data scientists and a chief AI officer to design tests. A county or a small state agency usually has none of these. It often buys through state contracts or cooperative purchasing agreements, where the terms were set by someone else. For those buyers, the practical options are to add testing terms at the order level and to borrow test plans from a larger agency or a state CIO’s office. They can also join with peers that run the same product.

Suppose a county human services department is buying a chatbot to answer questions about food assistance. Instead of scoring three polished demos, it gives each shortlisted vendor two weeks of supervised access. Caseworkers ask the questions applicants actually ask, including confused, angry and misspelled ones. A fraud investigator tries to get the system to explain how to hide income. A bilingual staff member tests the Spanish answers. One vendor’s system invents an eligibility rule that does not exist. That finding costs the agency two weeks of staff time. Discovering it after launch would have cost far more.

Before deployment

A model tested by its developer has been tested in general. An agency deploys it in a specific context: particular users, particular data, particular consequences of error. A chatbot that answers tax questions, a tool that summarizes case files and a system that drafts eligibility letters each fail in different ways. A wrong answer about office hours is an annoyance. A wrong answer about eligibility can cost a family its benefits.

Pre-deployment testing should therefore be built around the agency’s own use. That means test sets drawn from real questions and documents, cases representing groups the system might treat unfairly, and adversarial attempts by people who know how the program is gamed. A test set is simply a collection of example inputs with known good answers, which can be run against the system again and again. Fraud investigators, caseworkers and call center staff often find failures that technologists miss, because they know what real users and bad actors do.

Obligations the test plan should reflect

Some testing duties come from rules that predate AI. Federal agencies must make their electronic and information technology accessible to people with disabilities under Section 508 of the Rehabilitation Act. A chatbot that cannot be used with a screen reader fails that test regardless of how good its answers are. Agencies and recipients of federal funds also have obligations to people with limited English proficiency under Title VI of the Civil Rights Act and related guidance. Testing in the languages residents actually use is therefore part of meeting existing duties. Agencies should check the specifics with counsel.

A test set also needs to be big enough to mean something. Ten questions show whether the system works at all. A few hundred, covering the common questions and the risky edge cases, show how often it fails and where. The set should be versioned and kept, because it becomes the baseline for every later regression test.

Testing also needs a decision attached. Before testing starts, the agency should decide what results would stop deployment, what would require mitigation, and who signs off. Without that, test findings become a report that nobody is obliged to act on. The authorizing official for the system, or an equivalent senior owner, should accept residual risk in writing, as already happens for security authorizations. Residual risk is whatever risk remains after all the fixes; someone senior has to look at it and say, on the record, that it is acceptable.

A staged launch reduces the cost of being wrong. Some agencies run a new system in shadow mode first, where it drafts answers that staff review before anything reaches the public. Others open it to a small group of users or one office before going wider. Each stage should have exit criteria agreed in advance: an error rate, a complaint level, a set of test cases that must pass. Moving to the next stage is then a decision with evidence behind it.

Mitigation is often a design choice rather than a model fix. An agency can limit the chatbot to approved topics, require it to cite the agency’s own pages, route any eligibility question to a human, or show a clear notice that answers are not decisions. Each limit reduces usefulness a little and risk a lot. The test results tell the agency which limits it needs.

Test data raises its own issues. Realistic tests need realistic records, and those records often contain personal information. Agencies should decide early whether testing can use synthetic or de-identified data, and where it cannot, run tests inside the same controlled environment as production. Synthetic data is invented data that looks like the real thing; de-identified data is real data with names and identifying details removed. The privacy office should sign off on the test plan as well as the deployment.

In operation

The DEF CON results arrived after the exercise, and they described models that would soon be updated. That is the normal state of AI products. Vendors change underlying models, adjust safety filters and add features, sometimes monthly. Each change can fix one failure and introduce another. User behavior also changes once the public learns what a system will and will not do.

Operational testing has three parts. Monitoring watches outputs in production for known failure signs, such as refusals, complaints, escalations to human staff and unusual patterns of use. Regression testing reruns the agency’s test set whenever the vendor announces a model change, to check that what used to work still works; the name comes from checking that the system has not slipped backward. Periodic red-teaming brings in fresh testers, internal or external, to look for new failure modes. Findings need a route back to the vendor, with contract terms that require a response, and a route to the program owner, who decides whether to pause use.

What to measure, and what to keep

Monitoring needs a few numbers tracked every week. Useful ones include the share of conversations handed to a human, the share the system refused, complaint volume, and the error rate in a weekly sample of conversations reviewed by staff. Each number needs a threshold that triggers review. A rise in handoffs may mean the system is failing, or that a policy change has confused users. Either way, someone should look.

Conversation logs make this possible and raise their own questions. They may contain personal information, so access and retention need rules. They may also count as records under federal or state records laws, with retention duties of their own. Agencies should settle these points with their records officer and counsel before launch, not after the first public records request.

Version control is a trade-off. Some vendors let a customer stay on a fixed model version for a period. That protects the agency from surprise changes, but fixed versions are eventually retired, and staying back can mean missing safety improvements. The contract should state how much notice the vendor gives before a version changes or retires, so the agency has time to retest.

Sharing the load

Few agencies can staff all of this alone, and many are buying the same kinds of tools. That makes shared capacity worth building. Agencies running the same product can pool test sets and findings, since a jailbreak that works on one deployment will often work on another. A central team, at a federal shared-service provider or a state CIO’s office, can maintain test sets for common uses and run red-team exercises for agencies that lack staff. Communities of practice let testers compare methods and failures.

Sharing has costs. Findings about one agency’s system can be sensitive, and agencies need agreements on what is shared and with whom. Common test sets drift away from any one agency’s program over time. The shared layer works best for general risks, such as jailbreaks, injection and accessibility, while each agency keeps its own tests for its own rules and users.

All three require ongoing staff time, and that is where most agency plans in 2023 were thin. AI teams were small, and the people who understood both models and missions were in demand across government and industry. A realistic plan budgets testing as an operating cost for each deployed system, names the people who do it, and shares capacity across agencies where possible. Shared test sets for common uses, such as public-facing chatbots, reduce duplication. Without that capacity, testing requirements written into policy will be met on paper and missed in practice.

Over a year, that county’s experience might run like this. The food assistance chatbot launches in January after passing the agency’s tests. In April, the vendor announces a model upgrade; the agency reruns its test set over two days and finds that the new version answers a common question about reporting income changes less accurately than before. The vendor fixes it before the upgrade reaches the public. In July, monitoring shows a spike in conversations being handed to human staff; call center workers trace it to a new rumor about benefit changes circulating online. In October, a fresh group of testers from a neighboring county’s program spends a week trying to break the system and finds a role-play trick that gets around one safeguard. The events are undramatic, and every one needs someone whose job it is to notice.

Operational findings also need an incident path. When a public-facing system gives harmful advice, the agency needs to decide quickly whether to suspend it, correct it or warn users, and to know who has authority to do each. Treating serious AI failures as incidents, with the same logging, escalation and after-action review as a security incident, gives the testing program teeth and gives leaders evidence about where to invest.

Every incident should also leave something behind. The prompt or document that caused the failure becomes a new case in the test set, so the same failure is checked on every future update. Over time the test set turns into a record of everything the system has got wrong in this agency’s setting. That record is worth more than any vendor benchmark, and it moves with the agency if it changes vendors.

Trust, fairness, cost and people

For an agency head, a chief operating officer or a general counsel, AI testing can sound like a technical detail best left to specialists. It is better understood as a question of public trust, legal exposure and money.

Public trust. A government chatbot that gives wrong or offensive answers embarrasses more than the vendor: it speaks with the agency’s voice. Residents rarely distinguish between the tool and the institution behind it. They also have no alternative provider. A bank customer who dislikes a chatbot can change banks; a benefits applicant cannot change agencies.

Rights and fairness. Systems that help decide or explain eligibility, enforcement or access to services can treat groups differently in ways that are hard to see without deliberate testing. Those failures can lead to complaints, oversight inquiries and litigation. Testing that compares results across groups, languages and reading levels is the main way an agency finds these problems before a complainant does.

Cost. Testing costs staff time every month, in identifiable pieces. They include maintaining the test set as rules change, reviewing a weekly sample of conversations, rerunning tests after each vendor update, and an outside red-team exercise at intervals. An agency can estimate each piece in hours and put it in the system’s operating budget. Not testing shifts that cost onto the public and, eventually, onto the agency in rework, appeals and lost confidence. The cheapest moment to find a failure is before award; the most expensive is on the evening news.

Workforce. The binding constraint is people. An agency can write the best testing policy in government and still fail to carry it out if no one has the time and skills to do the work. Training existing staff is often more realistic than hiring. Caseworkers and investigators can learn structured testing in days; teaching an engineer how a benefits program is gamed takes much longer.

Put together, these points turn AI testing from a technical chore into a leadership decision. Someone has to decide how much risk the agency will accept from a given system, how much it will spend each year to keep that risk visible, and who answers for the result. Those are the same kinds of choices leaders already make about physical safety, financial controls and cybersecurity. AI simply adds a new, fast-changing system to the list.

Treating each system as a living service

An agency in good shape treats each AI system as a living service rather than a finished purchase. It asks consistent testing questions in solicitations, gives frontline staff hands-on time before award, builds test sets from real cases, decides in advance what would stop deployment, and names a senior owner who accepts residual risk. Once live, it monitors outputs, reruns tests on every model change, brings in fresh red-teamers periodically and handles serious failures as incidents. And it budgets for all of this as an ongoing cost.

Roles make the program real. A typical setup names a system owner who accepts risk and a test lead who maintains the test set and schedules regression runs. A rota of frontline staff gives a few hours a month to testing. Privacy, legal and accessibility reviewers join at defined points rather than on request. The chief AI officer or CIO sees a summary across all systems each quarter. In a small agency, these roles may need no new hires, but each needs a name and time set aside.

The case for testing less

A reasonable objection is that this is too much. Agencies have many low-risk uses, such as drafting internal memos or summarizing public documents, where a mistake is caught by the person using the tool. Imposing a full testing program on each would slow adoption, and slower adoption has costs too: backlogs, wait times and staff doing work a tool could help with.

The objection is right, and the answer is proportion. Testing effort should follow the consequences of error. A tool that drafts text for a staff member to check needs light testing and good training. A public-facing system that explains eligibility, or any system that informs a decision about a person, needs the full program. The work is to sort systems into tiers honestly and to move a system up a tier when its use expands. Federal policy later drew a line of this kind.

A second objection is that vendors test better than agencies ever could, with larger teams and better tools. The point holds for general behavior. It fails for the agency’s own rules, users and data, which the vendor does not know. The vendor’s testing is an input to the agency’s decision, not a substitute for it.

Agencies tend to stumble in predictable ways:

• Treating testing as a one-time gate. A pass at launch says little about the system six months and three model updates later.
• Trusting the demo. Scripted demonstrations show what the vendor wants to show.
• Leaving frontline staff out. The people who know how programs are misused are often the best testers available.
• Testing without a decision rule. Findings with no agreed consequence tend to stay in a report.
• Writing policy without staffing it. Requirements without people produce compliance on paper only.
• Testing what the system says but not what it can do. A model connected to records or actions needs testing of those connections as well.

Who keeps looking

These questions are for the official who owns an AI system, and for the chief AI officer or CIO who oversees the portfolio. Each should have an answer on file, with a name attached.

• For each AI system we use or plan to buy, who tests it, how often, and how many hours a month does that take?
• Did our last solicitation ask vendors to describe their own red-teaming, and did our staff get hands-on time before award?
• Do our contracts require the vendor to tell us before changing the underlying model, and do we rerun our tests when they do?
• Who decided in advance what test results would stop deployment, and who signed off on the remaining risk?
• If our public-facing system gave harmful advice tonight, who could suspend it, and how fast?

An agency that can answer all five has turned testing into a job. One that cannot has a policy and a hope.

With hindsight

Red-teaming moved into federal policy quickly after that weekend in Las Vegas. The October 2023 executive order on AI directed NIST, the National Institute of Standards and Technology, to develop red-teaming guidance. NIST’s generative AI profile for its AI Risk Management Framework, published in July 2024, built testing into its recommended practices. What had looked like a public demonstration became part of the expected toolkit for organizations using these systems.

The same order used the Defense Production Act to require developers of the most powerful models to report the results of their red-team testing to the government. NIST set up a U.S. AI Safety Institute that November to build evaluation methods. Those arrangements were aimed at frontier developers, not at agencies buying products. They still signaled that testing by people other than the builder was becoming a public expectation.

OMB, the White House office that sets management rules for federal agencies, followed with binding direction. Its March 2024 guidance required pre-deployment testing and ongoing monitoring for rights- and safety-impacting uses, meaning AI systems whose errors could affect people’s rights or physical safety. The replacement guidance issued in April 2025 kept pre-deployment testing and monitoring for high-impact AI even as other parts of federal AI policy changed. The April 2025 memo also kept the requirement for each agency to have a chief AI officer, which gave testing programs a senior owner, at least on paper. Across two administrations with different priorities, the idea that important AI systems must be tested before launch and watched afterward survived.

The policy shifts around that core were large. The 2023 executive order was revoked in January 2025. The AI Safety Institute was renamed the Center for AI Standards and Innovation that June, with a stronger focus on standards and national security. OMB’s acquisition guidance, first issued in late 2024 and replaced in April 2025 alongside the use memo, kept attention on what agencies should ask of AI vendors. The tiering this article argued for, with heavier testing for systems that affect people’s rights or safety, matched what both administrations’ memos required.

Buying also became easier. In 2025 the General Services Administration negotiated government-wide deals that offered agencies leading AI products at very low prices, some at a nominal dollar per agency for the first year. Lower prices removed a barrier to adoption. They did nothing to reduce the testing work each deployment needs, and arguably increased it, because more systems reached more staff faster.

DARPA’s AI Cyber Challenge, announced the same week as the DEF CON exercise, ran its final round at DEF CON in August 2025, bringing the multi-year competition full circle. The idea behind it, that AI could help defenders find and fix flaws faster, sits alongside the recognition that AI systems themselves need constant scrutiny.

The methods themselves matured. Prompt injection, barely discussed in procurement documents in 2023, became a standard item in security reviews of AI systems as models gained access to documents, email and tools. Developers published more detailed accounts of their own testing. The gap between general testing by the developer and contextual testing by the buyer, the gap this article focused on, remained.

Capacity is the open question. Both OMB memos set testing and monitoring requirements without funding the people to carry them out, so the staffing problem described here remains for each agency to solve. That leaves leaders with a practical choice: treat testing as a line in a policy document, or as a job with named people and a budget. The DEF CON weekend showed how quickly fresh eyes find new problems. The policy that followed made finding them an obligation. Whether agencies meet it depends on whether someone is paid to keep looking.