A hallucination rate is a sales claim, and Texas enforced it like one

The week

On September 18, 2024, the Texas Attorney General put an AI accuracy claim on trial, so to speak. The office announced a settlement with Pieces Technologies, a Dallas company whose generative AI product summarized patient information and drafted clinical documentation for hospital staff.

Tools like it were spreading. A nurse handing over a ward full of patients at the end of a twelve-hour shift, or a hospitalist trying to make sense of a chart that runs to hundreds of pages, was the target customer. In 2024 a growing number of hospitals were buying software that read the patient record and drafted a summary or a clinical note in seconds, and the pitch often came with a reassuring number about how rarely the tool got things wrong.

“Generative AI” means software that writes new text, rather than looking things up, by predicting what words should come next based on patterns learned from vast amounts of writing. It is fluent and fast, and it can also produce statements that sound right but are not supported by the source material. The industry calls these errors “hallucinations.”

The Attorney General’s office alleged that Pieces had made false and misleading statements about its product’s accuracy, including advertised metrics for its rate of severe hallucinations. It said those statements violated the Texas Deceptive Trade Practices Act, a consumer protection law aimed at misleading business practices. The office described the case as a first-of-its-kind settlement involving generative AI in healthcare.

Pieces denied wrongdoing. Under the agreement, the company committed to two things. First, to disclose clearly how any accuracy metric it advertised was defined and calculated. Second, to inform customers of known or reasonably knowable harmful or potentially harmful uses of its products. The settlement carried no financial penalty.

The timing mattered too. A week later, the Federal Trade Commission, the federal agency that polices unfair and deceptive business practices, announced “Operation AI Comply,” a set of actions against deceptive AI claims. Within a few days, a state and a federal enforcer had both signaled the same thing: a number in an AI sales deck is a promise, and promises about products are something regulators already know how to police.

The settlement created no new AI rule, and that is the most important thing about it. Texas did not need a fresh law about algorithms or a new agency with technical staff. It took a statute written for misleading advertising and applied it to a number that describes how often a generative model gets things wrong.

Once a state enforcer treats that number as a representation that must be accurate and explained, any health system that relied on one in a purchasing decision has a reason to look at it again. Most states have similar consumer protection laws, so the theory travels. A hospital in any of those states that bought a documentation tool on the strength of a headline accuracy figure now has good reason to ask what that figure really means.

A clinical accuracy claim has three components, and each can fail independently. The first is the definition: what counts as an error. The second is the measurement: on which records, by whom, under what conditions. The third is maintenance: whether the number still describes the product after it changes. A claim is only as reliable as the weakest of the three, and a buyer who examines all three has examined the whole claim. First, though, what these tools actually do.

How a documentation tool writes a note

A generative AI documentation tool is built on what is called a large language model. It has been trained on enormous quantities of text, and from that training it has learned the patterns of language so well that, given a prompt, it can produce paragraphs that read as if a knowledgeable person wrote them.

In a hospital, the tool is connected to the electronic health record, the digital chart that holds a patient’s history, medications, lab results, notes and orders. When a clinician asks for a summary, the software pulls the relevant parts of the record, feeds them to the model with instructions (“summarize this patient’s hospital stay for the discharging physician”), and returns a draft. The clinician reads it, edits it and signs it.

The appeal is easy to see. Clinicians spend a large share of their day on documentation, much of it after hours. A tool that turns a sprawling chart into a crisp page could give time back to patient care and reduce burnout, which in turn affects staffing costs and retention.

The catch is built into how the technology works. A language model does not look facts up the way a database does. It generates the most plausible next words. Most of the time, plausible and true coincide. Sometimes they do not, and the model writes a medication the patient never received, a lab value that was never measured, or a date that is slightly off. Because the output is fluent and confident, these mistakes do not announce themselves. Hence the industry’s word “hallucination,” and the weight carried by any number describing how often it happens.

The tool behaves rather like a very fast, very articulate new resident who has read every textbook but sometimes fills gaps in a patient’s story with what usually happens rather than what did happen. You would not ban that resident from writing notes. You would want to know how often it happens, what kind of gaps get filled, and who checks the work.

The definition: what counts as an error

A hallucination rate sounds precise until someone asks what a hallucination is. A summary that omits a medication, one that invents a lab value and one that places an event on the wrong day are different failures with different clinical consequences. So is an omission compared with a fabrication. A vendor can report a very low rate by counting only the most severe category, or by excluding omissions.

Suppose a vendor tells a regional health system that its tool has a severe hallucination rate below one in a thousand. That sounds excellent. But suppose “severe” means only invented facts that would plausibly change treatment, and omissions are not counted at all. A summary that leaves out a patient’s penicillin allergy would not register. Neither would one that lists the right drug at the wrong dose, if the scorers judged that “moderate.” The headline number could be perfectly honest and still tell the buyer very little about the risks they care about.

Buyers should ask for the error taxonomy in writing. A taxonomy is the agreed list of error types and how they are graded. The request should cover the categories, the severity scale, the rules for scoring ambiguous cases, and examples of each, with rates for every category as well as the headline one.

They should also ask who did the scoring. Clinicians independent of the product team give a different answer from the engineers who built it. This is less a question of honesty than of perspective. A nurse knows which missing detail would cause trouble at 3 a.m.; an engineer may not.

The definition must also fit the intended use. A rate measured on discharge summaries says little about nursing handoff notes, prior authorization letters or patient-facing messages, where the cost of a given error differs. A small slip in an internal summary that a physician reviews is one thing. The same slip in a message sent directly to a patient, or in a letter to an insurer that determines whether a procedure is paid for, carries different consequences for safety, revenue and reputation.

The measurement: where and on whom

Even a well-defined error rate is only as good as the test that produced it. An error rate measured on one hospital’s records, in one specialty and one documentation style may not hold elsewhere. Patient complexity, note templates, local abbreviations, interpreters and languages, and the configuration of the electronic health record all affect performance. An AI accuracy number measured at an academic medical center with tidy templates may not survive contact with a community hospital whose physicians use their own shorthand.

The next set of points concerns sampling, meaning how the test cases were chosen. A buyer needs to know how many cases were reviewed and whether they were drawn at random or chosen; whether the sample included complex patients with many conditions, pediatric cases and patients with limited English proficiency; and whether the evaluation was repeated after model updates, with results from each round available. A vendor that cannot answer has a marketing figure rather than a measurement.

The measurement also depends on the reference standard, the “answer key” the drafts are compared against. If reviewers compared the draft only with the source note, errors already present in that note go uncounted. If they compared it with the full chart, the rate will be higher and more honest. Neither is wrong, but the buyer needs to know which one produced the number, and whether the comparison was blinded to which drafts came from the tool. Blinding means reviewers do not know whether a given draft came from the software or a human, which removes the temptation, conscious or not, to grade one more generously.

Testing the claim at home

A health system can close part of this gap itself with a structured local pilot. A practical design is a fixed sample of local encounters, drafted by the tool and reviewed by clinicians against the source record using the vendor’s own taxonomy. That produces a local rate the organization can compare with the advertised one.

It also produces something more useful: a list of the error types the tool makes in this environment, which shapes training and review policy. If the pilot at a community hospital shows that the tool tends to garble medication changes made late in a stay, the hospital knows exactly what to train clinicians to check, and it has specific evidence to bring back to the vendor.

The maintenance: keeping the number true

Most software changes in visible steps: version 3 becomes version 4, and someone sends release notes. Generative AI products change more often and less visibly. Vendors update underlying models, prompts and retrieval components, sometimes without visible version changes. The “prompt” is the set of standing instructions the software gives the model; “retrieval” is the part that decides which pieces of the chart to send the model in the first place. Change any of them and behavior can shift. A number measured when the contract was signed may not describe the product six months later.

The settlement’s second commitment, disclosure of known harmful uses, implies an ongoing duty, and buyers can mirror it in contracts. Useful terms include notice of material model changes, re-measurement of advertised metrics after significant updates, a defined channel for reporting errors with response times, and disclosure of harmful uses or incidents identified at other customers. Renewal is the moment to enforce these terms, because the buyer then has the most bargaining power.

Internally, maintenance means a low-friction way for clinicians to flag errors from inside the workflow, and a named owner who reviews the reports, spots patterns and escalates to the vendor. If reporting an error means leaving the chart, logging into a separate system and filling in a form, busy clinicians will not do it, and the organization will be blind.

It also means monitoring how much clinicians edit drafts. A sudden fall in edits may mean the tool improved, or that reviewers stopped reading carefully. Only sampling against the source record tells the two apart. The pattern is familiar enough to have a name, automation complacency: when a system is right most of the time, people gradually stop checking it. The more often a tool is right, the less closely its drafts tend to be reviewed.

Safety, contracts and the health system’s own claims

Enforcement against the vendor leaves the health system’s own exposure in place on five fronts.

Patient safety and liability. Vendor enforcement does not shift the provider’s own responsibility. A health system that deploys a documentation tool remains responsible for the accuracy of the medical record and for patient safety, whatever the vendor advertised. If a fabricated detail in a signed note contributes to harm, the hospital and the clinician who signed it are the ones most directly exposed.

Contracts and risk allocation. Counsel can help allocate risk through warranties and indemnities. A warranty is a contractual promise that something is true; an indemnity is a promise to cover certain losses if it is not. Both are worth negotiating, but no contract term replaces knowing what the number means. Recovering money after an incident does not undo the incident.

Regulatory exposure. The case shows that existing consumer protection law reaches AI claims. That cuts both ways. Health systems that buy tools should scrutinize vendor numbers, and health systems that themselves advertise AI-enabled services to patients should apply the same discipline to their own statements.

Reputation and trust. Patients and clinicians are still forming their views about AI in care. An organization that can explain clearly how it tested a tool, what errors it found and how it guards against them is in a far stronger position than one that can only repeat a vendor’s headline figure.

Return on investment. If the tool saves ten minutes per note but requires careful review to catch subtle errors, the real time saved may be smaller than projected. Knowing the actual error profile lets finance teams build honest business cases rather than optimistic ones.

How buyers end up repeating the vendor’s number

A well-run health system approaching an AI documentation tool treats the vendor’s accuracy number as the start of a conversation rather than the end of one. It asks for the definitions and the testing method in writing before signing, runs its own pilot on its own patients, and writes change notification and re-measurement into the contract. It names an owner for ongoing monitoring and gives clinicians an easy way to report problems. And it keeps the final responsibility for the record where it belongs: with the clinicians who sign and the organization that employs them.

The failures are the mirror images of those habits.

• Buying the headline. Accepting a single accuracy figure without asking what it counts or excludes.
• Assuming portability. Assuming a number measured elsewhere will hold in your hospitals, with your patients, templates and abbreviations.
• Treating go-live as the finish line. Testing once at purchase, then never again, even as the vendor updates the product behind the scenes.
• Reading falling edits as good news. Celebrating that clinicians edit less, without checking whether they are still reading closely.
• Outsourcing responsibility. Believing that a strong indemnity clause means the vendor now owns the risk.
• Ignoring use creep. Approving a tool for one purpose, such as internal summaries, and then allowing it to drift into others, such as patient messages, without fresh evaluation.

What an accuracy figure has to survive

Governing a language model well does not require knowing how one is trained. It requires questions that expose whether anyone has done the work.

1. Which AI tools are currently drafting or summarizing anything that ends up in a patient’s record or in front of a patient, and do we have the written definitions behind each vendor’s accuracy claims?
2. Do we know the rates for every error category, including omissions, and who scored them?
3. Have we tested the tool on our own patients, including complex, pediatric and limited English proficiency cases?
4. Do our contracts require the vendor to tell us when the model or prompts change, and to re-measure afterward?
5. Are we sampling drafts against the source record, so we know what a change in edit rates means?

Two further checks belong alongside them. Each tool’s use may have expanded since it was approved, and every new use deserves its own evaluation. And a health system that describes AI-enabled services to patients or the public should test whether its own claims would survive the scrutiny Texas applied to Pieces.

The questions need no technical background, though they reward one. Taken together, the answers tell a leader whether an advertised accuracy number is something the organization understands or something it merely repeats.

With hindsight

The Pieces settlement became a reference point for later state activity on AI in healthcare. It showed that state attorneys general did not need to wait for AI-specific legislation to act, and that a vendor’s advertised accuracy figure could be treated as a claim with consequences.

Texas itself went further in 2025, passing two relevant laws. SB 1188, effective September 2025, requires practitioners who use AI for diagnostic purposes to review the records it creates and to disclose that use to patients. The Texas Responsible AI Governance Act (HB 149), effective January 2026, requires providers to disclose AI use in treatment to patients. Where the 2024 settlement focused on what vendors told buyers, these laws turned to what clinicians do with the output and what patients are told.

Federal enforcement priorities on AI shifted after January 2025, which made state law relatively more important. For a health system operating in several states, the practical rulebook for AI is increasingly a patchwork of state consumer protection and health statutes rather than a single federal standard. Keeping track of it is now a standing compliance task rather than an occasional one.

Meanwhile, the technology did not wait. Ambient documentation tools, which listen to a clinical conversation and draft the note from it, spread quickly through 2025 and 2026. The review burden on AI-drafted notes fell, as the settlement implied it would, on clinicians and their organizations. Every one of those drafts still needs someone to read it with care, and every organization still owns what ends up in the chart.

An accuracy number is useful only if you know what it counts, where it was measured, and whether it still holds after the product changes. The organizations best placed today are the ones that built that habit early: asking for definitions, testing locally, contracting for change notice and watching for complacency. Those habits apply equally to the next generation of tools, whatever they are called.