Why o1’s reasoning summaries added work for clinical AI committees

The week

On September 12, 2024, OpenAI released o1-preview and o1-mini. They were the first widely available models built to spend more computation reasoning through a problem before answering. Earlier chatbots produced their reply more or less in one pass. These “reasoning models” were designed to work through intermediate steps first.

Access was limited at first. OpenAI offered the two models to paying ChatGPT subscribers and to a subset of developers through its API, with usage caps. At launch they lacked features GPT-4o users already had, such as web browsing and file uploads.

For years, one of the most common complaints about artificial intelligence in medicine was that it was a “black box”: it gave an answer but could not say why. The new models seemed to answer that complaint, and in doing so created a subtler problem for the hospitals thinking about using them.

The company reported strong results on mathematics, coding and science benchmarks, the standardized tests AI developers use to compare models. But it made a notable choice about what users could see. It chose not to show users the model’s raw chain of thought, the step-by-step internal working, and displayed a generated summary instead. The explanation people read was itself something the model wrote about its reasoning, not a direct view of it.

OpenAI explained the choice. It wanted to be able to monitor the raw reasoning for signs of problems, and that required leaving the reasoning free of training meant to make it look acceptable to users. It also cited user experience and competitive advantage. The summary shown in ChatGPT came from a separate summarizer model. Developers using the API were billed for “reasoning tokens” they could not see.

The system card OpenAI published with the release carried warnings that bear directly on clinical use. External red teamers found o1-preview more convincing than GPT-4o in some domains, because it gave more detailed answers, which could raise the risk that people trust hallucinated output. OpenAI also reported anecdotal feedback that the new models hallucinated more than GPT-4o, even though its own evaluations showed the opposite. A monitor it ran over the hidden reasoning found cases where the model appeared to know an answer was wrong, most often when asked for references it could not check.

The release arrived as health systems prepared for two federal rules on clinical algorithms.

The first was the ONC HTI-1 final rule. ONC is the federal office that oversees the certification of electronic health record software. HTI-1 required developers of certified health IT to support “source attributes” for decision support interventions by the end of 2024. Decision support interventions are tools that help clinicians make choices, such as alerts, risk scores and recommendations. Source attributes are information about what a tool is for, what data built it and how well it works.

HTI-1 had been published in January 2024. Its new decision support criterion replaced the older clinical decision support criterion in the certification program. For predictive interventions a developer supplies as part of its product, the rule also required risk analysis, risk mitigation and governance practices. The certification duties fell on EHR developers. Hospitals felt them through the software updates and configuration screens those developers shipped.

The second was the HHS Section 1557 nondiscrimination rule, finalized in May 2024. It required covered entities, meaning the health care providers and insurers subject to the rule, to identify and mitigate discrimination risks in patient care decision support tools. That provision’s compliance date was set for May 2025.

The rule defined patient care decision support tools broadly. The definition covers automated and non-automated tools, from predictive algorithms to flowcharts, clinical guidelines and calculators. Its covered entities include health programs and activities that receive HHS financial assistance, which reaches most hospitals and many physician practices and health plans.

State lawmakers were moving too. Later that month, California enacted AB 3030. From January 2025, it required health facilities, clinics and physician offices that use generative AI to write patient communications about clinical information to tell patients so, and to explain how to reach a human. Communications read and reviewed by a licensed provider were exempt.

So in a single week, hospital leaders were handed a model that explained itself fluently and a regulatory calendar that demanded transparency of a quite different kind.

A hospitalist late on a night shift is checking a medication dose for an elderly patient with failing kidneys. She types the question into an AI tool. It answers with a clear dose and a tidy, numbered explanation: the patient’s weight, the kidney function, the adjustment rule, the result. It reads like a good resident talking through a case. She feels reassured. Whether she should is another matter.

A model that explains itself sounds like exactly what clinical AI governance had been asking for. o1 produced fluent, step-by-step accounts of how it reached an answer. For a clinician at the point of care, that is reassuring. For a governance committee, the group of clinicians, informaticians, compliance staff and lawyers that decides which AI tools a health system may use and how, it created more work. A plausible explanation is not an accurate account of what a model did, and OpenAI itself said users were seeing a summary, not the underlying reasoning.

The release blurred three kinds of transparency that have different audiences and different owners. Transparency at the point of use tells a clinician why this answer, for this patient. Product transparency tells the deploying organization what the model is, how it was built and where it fails. Deployment transparency tells patients, regulators and auditors how the organization chose, tested and monitors the tool. Every disclosure a health system makes about clinical AI belongs to one of these three. o1 improved only the first.

The split matters more for some organizations than others. An academic medical center may have informaticians, data scientists and an AI committee that can build all three layers in-house. A community hospital or an ambulatory network usually has a CMIO with a day job and an IT team stretched across the EHR, the network and security. For them, knowing which layer they must supply themselves, and which they can demand from a vendor, decides where scarce people spend their time.

Reasoning models and the chain of thought

Most of the AI tools in the news since late 2022 are large language models: systems trained on enormous amounts of text to predict what words should come next. That simple-sounding ability turns out to let them answer questions, summarize documents and draft letters with remarkable fluency.

A reasoning model is a large language model trained to produce a sequence of intermediate steps, a “chain of thought,” before it commits to a final answer. Spending more computation on those steps tends to improve performance on problems that benefit from working things out, such as math, coding and scientific questions. OpenAI reported exactly that for o1.

Two points matter for a health system leader. First, the steps are still generated text, produced by the same statistical process that produces the answer, and they are not a log of the model’s inner mechanics. Second, with o1, users did not even see those steps directly. They saw a summary of them, written by the system.

Where step-by-step output does help

Reasoning output is most useful where each step points to a fact the clinician can check. A renal dose adjustment is a good case. The steps cite a weight, a creatinine value, a formula for estimated kidney function and a dosing table. Each of those can be compared with the chart or a reference in seconds, and a wrong one is visible. A diagnostic hunch is the opposite case. The steps cite impressions and associations, none of which can be checked against anything concrete.

That distinction gives committees a practical rule. Favor uses where the explanation decomposes into verifiable facts, and treat uses where it decomposes into judgment with more caution. The same model can be appropriate for one and inappropriate for the other.

What the user actually saw

The mechanics matter. With o1, the model generated a long internal chain of reasoning, billed as tokens but hidden. A separate summarizer then wrote a shorter account of that reasoning for display in ChatGPT. The clinician therefore read a summary, written by one model, of text generated by another process, which itself was not a record of the computation that produced the answer. Each step adds a place where the account can drift from what drove the output.

OpenAI’s reason for hiding the raw reasoning cuts in an interesting direction. The company wanted to keep the reasoning unpolished so it could monitor it for problems. That implies the raw reasoning might contain material that is not fit to show. The polished summary is, by design, the cleaner version.

Why more detail can mean more trust, not more accuracy

Reasoning models improved results on hard problems, and OpenAI’s benchmarks showed it. But the same system card noted that red teamers found o1-preview more convincing than GPT-4o in some domains because of its more detailed answers. Detail persuades whether or not it is correct. For a governance committee, accuracy and persuasiveness are separate properties that must be measured separately.

The closest human parallel is asking a colleague how they reached a diagnosis and getting back a polished paragraph written afterward by someone else in the room who watched them think. It might be accurate, it might leave out the hunch that actually drove the decision, and it will almost certainly sound more orderly than the real process was.

Point-of-use transparency: an explanation is output, not evidence

A reasoning summary is generated text, produced by the same system whose answer is under review. It may faithfully describe the path to the answer, or it may be a tidy rationalization. Research on chain-of-thought faithfulness, meaning whether a model’s stated reasoning matches what actually influenced its output, had already shown that stated reasoning does not always reflect the factors that drove a model’s output.

One 2023 study made the point vividly. Researchers reordered the answer options in few-shot prompts so that the correct answer was always “(A)”. Models picked up the pattern and chose “(A)” more often, yet their written reasoning almost never mentioned it. The explanations described a plausible route to the answer that left out the factor actually steering it.

Explanations still have value. They can help a clinician spot an obvious error, such as a misread lab value or a wrong unit. If the explanation says the patient weighs 70 pounds when the chart says 70 kilograms, the mistake jumps out.

The risk is the opposite effect: a confident, well-structured rationale lowers a reader’s guard precisely where vigilance matters. Dosing, contraindications (situations where a drug or treatment should not be used), pediatric and renal adjustments, and rare presentations are the cases where an articulate wrong answer does the most harm. Those are also the cases where a tired clinician is most grateful for a clear answer.

Automation bias on the night shift

Clinicians are not careless. The problem is a well-documented tendency, automation bias, in which people under time pressure accept a system’s recommendation and check it less thoroughly than they would check a colleague. An explanation can strengthen that effect. It gives the reader the feeling that checking has already happened. The clinician skims the steps, finds them reasonable and moves on, without checking the one input that was wrong.

That effect can be measured, which makes it governable. The useful metric is the catch rate: when a reviewer is shown a planted wrong answer with a persuasive explanation, how often do they catch it? A second metric is the agreement rate in live use. If clinicians agree with the model in nearly every case, review may have become a formality, and the committee should find out why.

Design choices that help

Health systems can manage this with design choices rather than hope.

• Order of display. Interfaces can show the explanation only after the clinician has formed a view, so the model’s reasoning tests the clinician’s judgment instead of replacing it.
• What to check. Training can teach staff to check the cited facts rather than the flow of the argument: is that the right weight, the right lab value, the right guideline? A smooth argument built on a wrong fact is still wrong.
• Testing against persuasion. Testing can deliberately include cases where the model is known to be wrong, to see whether reviewers catch the error when the explanation is persuasive. If they do not, the explanation is a hazard rather than a safeguard.
• Counting the saves. A simple log of cases where a clinician caught an error because the explanation exposed it shows whether explanations earn their place. If the log stays empty for months, the explanations are probably being read as reassurance, not checked.

Each of these has a cost. Asking for the clinician’s answer first adds seconds to every query, and busy clinicians will notice. Planted-error testing needs a curated set of cases that someone with clinical expertise must build and refresh. A sensible compromise applies the friction only to high-risk categories, such as dosing and contraindications, and leaves low-risk uses, such as drafting a patient education handout, unencumbered.

Product transparency: what the rules ask vendors to disclose

HTI-1 placed obligations on developers of certified health IT, the companies whose electronic health record (EHR) software meets federal certification requirements. For predictive decision support interventions the developer supplies, users must be able to see source attributes such as intended use, development data, known risks, validation and fairness information.

That duty was narrow. For models a hospital built or bought elsewhere, the EHR only had to hold the information, and the hospital had to produce it. The template was provided; filling it in was the hospital’s job.

How the split works inside one EHR

Consider a hospital running a certified EHR with three predictive tools. The EHR vendor supplies a sepsis risk score, so the vendor must populate its source attributes and apply its own risk management practices. The hospital’s data team built a readmission model, so the EHR must let the hospital record source attributes, but the content is the hospital’s. A third-party vendor supplies a deterioration model through an integration, and the hospital must get the information from that vendor to fill the fields.

Only the first tool arrives with its documentation done. The other two depend on the hospital asking, in contracts and in configuration work. Health systems that read HTI-1 as “the EHR vendor will handle transparency” misread who did what.

General-purpose reasoning models fit awkwardly here. A model used through a chat interface for clinical questions is often not a decision support intervention inside certified health IT at all. It arrives with general documentation, such as a system card, that describes benchmark results and safety testing but says nothing about a specific hospital’s patients or workflows. A system card is the developer’s public report on a model’s capabilities, limits and safety evaluations. It is useful, but it is written for the world, not for a particular emergency department.

The medical device question

A separate regulator watches part of this ground. Under the 21st Century Cures Act and the FDA’s 2022 guidance on clinical decision support software, some decision support software falls outside the definition of a medical device. One of the conditions is that the software lets a health care professional independently review the basis for its recommendations. That condition sounds like a call for explanations, but it means access to the underlying logic and data, so the clinician need not rely mainly on the software.

A generated reasoning summary is not obviously that kind of basis. Whether a particular clinical use of a general-purpose model could bring device regulation into play depends on the use, the claims made and the workflow. Health systems building such tools into clinical workflows should have counsel assess the question before deployment, not after a vendor or a regulator raises it.

Closing the gap in procurement

The gap has to be closed in procurement. A health system can require a vendor to state the permitted clinical uses and the model versions it will deploy. Fixing the version, sometimes called version pinning, matters because a model that changes without notice can behave differently tomorrow than it did during testing. The contract should also cover notice of model changes and whether customer inputs are retained or used for training.

General-purpose models raise points that older clinical software contracts rarely covered:

• Retirement schedules. Providers retire model versions on their own timetable. The contract should state the notice period, so revalidation can happen before the old version disappears.
• Logging for audit. The health system needs prompts, outputs and model versions retained somewhere it controls, so an adverse event can be reconstructed.
• Data location and retention. Where inputs are processed and how long the provider keeps them matter for privacy reviews and for state law.
• Integration path. A model reached through the EHR vendor, a cloud provider or the model developer directly brings different contract chains and different business associate arrangements.

Before any protected health information is entered, a business associate agreement must be in place. Under HIPAA, the federal health privacy law, that is the contract in which an outside company that handles patient information on a provider’s behalf commits to protecting it. Without one, typing patient details into a tool is a compliance problem regardless of how good the tool is.

The same model can arrive under very different terms. A consumer account on a clinician’s phone, an enterprise workspace licensed by the health system and an API connection built into the EHR may all run similar models. Only the second and third can carry negotiated data terms, administrative controls and audit logs. Product transparency therefore starts with knowing which channel a tool comes through, since the documents available, and the power to demand more, differ by channel.

There is a human side too. Where clinicians are already using consumer chat tools on personal accounts, as many were in 2024, product transparency means little until the approved tool is easier to use than the unapproved one. A carefully vetted tool that takes six clicks and a separate login will lose to a free app on a phone.

Deployment transparency: what only the health system can show

The Section 1557 rule put its decision support duty on covered entities, the providers and plans themselves. They must make reasonable efforts to identify tools that use race, color, national origin, sex, age or disability as input variables, and to mitigate the risk of discrimination. That duty rests with the deployer, whoever built the tool. Counsel should advise on how it applies to particular deployments, including general-purpose models.

Who the duty reaches

Section 1557 applies to covered entities, chiefly health programs and activities that receive HHS financial assistance. That includes most hospitals and health systems, many physician practices and ambulatory networks that take Medicaid, and many health insurers. The duty does not attach directly to a model developer that sells software. A life sciences company using AI in research is in a different position altogether, with its own regulators and rules.

The rule asks for “reasonable efforts,” and HHS indicated that what is reasonable can depend on an organization’s size and resources. That does not let a small ambulatory network ignore the duty. It means the network’s inventory and review can be lighter than an academic center’s, provided it exists and is followed.

Meeting it requires records only the health system can create:

• an inventory of decision support tools in use;
• the evidence behind each approval;
• local validation results on the organization’s own population, because a tool that works well elsewhere may not work as well for a different mix of patients;
• monitoring that would detect drift, meaning a gradual decline in performance as patients, practices or the model change, or uneven error rates across groups.

For reasoning models, the record should also state which uses are permitted. Drafting patient letters and summarizing literature carry different risks from suggesting a diagnosis or a dose. A clear list of “yes,” “yes with review” and “no” uses gives clinicians something concrete to follow.

Shared EHRs and affiliated practices

Many health systems extend their EHR to affiliated independent practices under hosting arrangements. When the host turns on an AI feature, it may appear in dozens of separate practices at once. Each practice that is a covered entity carries its own Section 1557 duty, but few have the staff to review a tool the host switched on. The hosting agreement should say who inventories AI features, who validates them, and how affiliates learn of changes. Without that, the affiliates inherit tools they never chose and cannot explain.

Validation a community hospital can afford

Local validation sounds like a research project, and at an academic center it can be one. A community hospital needs a smaller version. A clinical pharmacist pulls a sample of recent real questions in the permitted use, removes identifiers where possible, runs them through the tool and grades the answers against the hospital’s own references. The sample should be weighted toward the cases that matter most, such as older patients, kidney impairment and children.

Such a sample cannot prove a tool is safe. It can show whether the tool fails in ways the hospital cares about, and it creates a baseline for later comparison. Hospitals in a regional network or purchasing group can share test sets and results, which spreads the cost without pretending one hospital’s patients are the same as another’s.

Monitoring when there is no score to track

Monitoring a predictive score is familiar: compare predictions with outcomes over time. A general-purpose model answering free-text questions has no single score to track. Monitoring becomes a sampling exercise. A reviewer audits a fixed number of logged interactions each month, grades them, and records errors by category and patient group. A simple feedback button lets clinicians flag answers they distrust.

Logging creates its own duty. Stored prompts often contain protected health information, so the logs need access controls and retention limits like any other clinical record. The privacy officer should agree the design before the tool goes live.

Telling patients

Deployment transparency also has a patient-facing side. California’s AB 3030, enacted in September 2024, required a disclaimer when generative AI drafts patient communications about clinical information without review by a licensed provider. Organizations operating in several states face a patchwork of such rules, and the safer design is a single disclosure practice that meets the strictest one in their footprint.

Accountability needs a name

A committee can approve a use, but a specific person must receive error reports, review monitoring data and hold authority to suspend the tool. Without that role, the documentation describes a control that nobody runs. The same person should own the link between clinician error reports and the vendor, so that a pattern of bad answers reaches the people who can change the model or its configuration.

The switch-off has to work at 2 a.m. That means a deputy for when the owner is away, a documented procedure the IT on-call team can run, and a fallback for clinicians, such as the standard drug reference, once the tool is disabled. A suspension power that depends on one person reaching one vendor during business hours is weaker than it looks in the approval file.

Three audiences, three owners

Separating the three kinds of transparency is more than tidy labeling. Each answers a different person’s question, and each can be supplied only by a particular party.

The clinician wants to know whether to trust this answer for this patient, right now. The model can help here, through its explanation, but the clinician’s own judgment and training do the real work. The owner is the clinical leadership that designs workflows and trains staff.

The health system, as buyer, needs to know what exactly the product is, how it was built, where it fails and what the vendor will say when it changes. Only the vendor knows most of the answers, but only the health system can insist on getting them. The owner is procurement and legal, working with the governance committee.

Patients, regulators and auditors ask how the organization decided to use the tool, how it checked that it works for people like them, and who is watching it now. No vendor can answer that, because the questions are about the health system’s own choices. The owner is the health system’s leadership, through its inventory, validation, monitoring and named accountable person.

Each layer also has its own measure. For the clinician layer, it is the catch rate on planted errors and the agreement rate in live use. For the product layer, it is the share of AI contracts with version pinning, change notice and data-use terms. For the deployment layer, it is the share of inventoried tools with current local validation and an active owner. A quarterly report with those numbers tells a board more than any demonstration.

The trap in September 2024 was that a beautifully written explanation can feel as though it answers all three questions at once. A board member watching a demonstration might reasonably think, “It shows its reasoning, so it is transparent.” But the explanation speaks only to the first audience. The vendor’s questions and the regulator’s questions are left exactly where they were. Whenever a tool is described as transparent, an executive can usefully ask: transparent to whom, about what, and who produced the evidence?

The case that explanations are enough

The strongest counter-argument deserves a hearing. Clinicians already rely on tools they cannot inspect: lab analyzers, drug interaction databases, imaging software. Nobody asks a drug reference to explain its reasoning. If a reasoning model is accurate and explains itself, the argument runs, it is already better than much of what clinicians use, and a three-layer structure is bureaucracy.

The comparison fails on one point. A drug reference has editors, versions, citations and a publisher accountable for its content, and it gives the same answer to the same question. A general-purpose model has none of those properties by default. The product and deployment layers exist to give it some of them: a fixed version, known limits, local evidence and an owner.

A second counter-argument is practical. Heavy governance slows adoption, and clinicians denied a sanctioned tool turn to consumer apps. That risk is real. The answer is to tier the review, with a fast path for low-risk uses such as drafting and literature summaries and full review reserved for uses that touch diagnosis, dosing or orders.

One medication tool, three kinds of transparency

Suppose a regional health system wants to let hospitalists use a reasoning model to check medication questions.

Point of use. The tool is built into the clinicians’ workflow and asks for the clinician’s own answer before showing the model’s explanation. Training sessions use real examples, including several where the model’s explanation is convincing and wrong, and teach staff to check the weight, the lab values and the guideline cited. Testing records how often reviewers catch those planted errors.

Product. The contract names the permitted uses: medication questions with clinician review, and no autonomous orders. It fixes the model version, requires notice before changes, states that patient inputs will not be used for training, and is backed by a business associate agreement. The vendor’s system card is filed, but the committee does not treat it as evidence about its own patients.

Deployment. The tool is added to the decision support inventory. The approval file includes a local validation on a sample of the system’s own cases, broken down by age and kidney function, since those are where dosing errors cluster. A named pharmacist-informaticist receives error reports, reviews monitoring monthly, can switch the tool off, and passes patterns to the vendor.

The fluent explanations are still there, and clinicians still like them. They are no longer being asked to do a job they were never able to do.

The effort is modest but not trivial. A pharmacist-informaticist might spend several days building the planted-error test set and the local validation sample, then a few hours a month on monitoring. Legal and procurement time goes into the contract once. Set against the cost of one serious dosing error, the effort is small. Set against a busy informatics team’s backlog, it still has to be planned and staffed, or it will not happen.

Safety, liability and the tools clinicians use anyway

Patient safety. The most dangerous AI error is the plausible one rather than the obviously wrong one, delivered with confidence in the cases where mistakes hurt most.

Regulatory duty. The deployer duty under Section 1557 falls to the health system, whoever built the tool. A vendor’s documentation does not discharge it.

Liability and reputation. If harm occurs, the questions will be about what the health system knew, tested and monitored. A clean, persuasive explanation from the model will not answer them.

The law on liability for AI-assisted clinical decisions was unsettled in 2024 and remained so. What is predictable is the evidence that a plaintiff’s lawyer or a peer review committee will seek. They will want to know what the tool was approved for, how it was tested, which version was running, and what the clinician saw. Logs and approval records answer those questions. Their absence is hard to explain afterward.

Shadow use. Clinicians using unapproved tools create privacy and safety exposure that no committee can see. Offering an approved, convenient alternative is a risk control as much as an IT project.

Training and skills. Teaching hospitals face a further question. Residents learn clinical reasoning partly by watching how others reason. A trainee who reads fluent machine explanations every night may absorb their structure, and their errors, without the correction a supervising physician would supply. Programs that allow these tools in training settings should decide how faculty review AI-assisted work, and say so in the approval file.

Where clinical AI oversight slips

The stronger programs keep the three kinds of transparency separate and assign each an owner. Point-of-use explanations are treated as aids, with interfaces and training designed around their limits. Product transparency is secured through contracts that cover permitted uses, version pinning, change notice and data use, with a business associate agreement in place before any patient data is entered. Deployment transparency rests on an inventory, local validation, monitoring and a named accountable person with authority to suspend.

The slips are predictable:

• Accepting explanations as validation. A model’s account of itself is not evidence that it works.
• Testing only easy cases. If tests never include persuasive wrong answers, they cannot show whether reviewers will catch them.
• Relying on certification or system cards. General-purpose models often sit outside certified health IT, and system cards describe the model, not your patients.
• Ignoring version changes. A tool validated in one month may not be the same tool the next.
• Approving by committee with no owner. Approval without a named person to monitor and suspend is a control on paper only.
• Mistaking a model’s explanation for patient-facing transparency. Patients and regulators want to know how the organization chose and checks the tool, which no model output can tell them.
• Letting the integration path decide governance. A tool that arrives inside the EHR still needs its own review, even when it bypasses the usual purchasing route.
• Pretending shadow use is not happening. Banning tools without offering a better alternative drives use out of sight.

The evidence an examiner will ask for

An OCR investigator looking at a Section 1557 complaint, an accreditation surveyor or an internal auditor will ask for documents. Expect requests for the decision support inventory, the approval file for the tool in question, local validation results by patient group, and monitoring records over time. They may also want the contract terms on versions and data use, the business associate agreement, and the name of the accountable owner with evidence that the owner acted on reports. A health system that can produce these quickly has done the work. One that cannot will find that the model’s explanations are no substitute.

Where the explanation stops and the evidence starts

• Which AI tools, including general-purpose chat tools, are our clinicians using today, approved or not?
• Does our testing include cases where the model is wrong but persuasive, especially for dosing, contraindications, pediatric and renal adjustments and rare presentations?
• Do our contracts fix model versions, require notice of changes and limit retention and training on our data? Is a business associate agreement in place for every tool that may receive patient information?
• What local validation have we done on our own patient population, and how do we monitor for drift and uneven error rates?
• For each tool, who is the named person who receives error reports and can switch it off?

Sequence the work by exposure. Start with the inventory, including shadow use, because nothing else can be scoped without it. Name owners for the highest-risk uses next. Contract terms follow at the next renewal or purchase. Planted-error testing and monitoring can begin with the first approved tool and grow from there.

o1 made one layer of clinical AI transparency more convincing and left the other two unchanged. That is a reason to invest in the two layers the model cannot supply.

With hindsight

Within a year, reasoning models had spread across the major model providers. OpenAI followed o1 with o3-mini in January 2025, and in August 2025 GPT-5 combined fast answers and deeper reasoning in one system. Google, Anthropic and DeepSeek released reasoning models of their own. What looked like a novelty in September 2024 became a standard feature, and the questions raised here became routine for clinical AI committees.

Not every provider hid the reasoning. DeepSeek’s R1, released in January 2025, displayed its full reasoning text. Anthropic’s Claude 3.7 Sonnet, released the next month, let users see its extended thinking. Visibility turned out not to settle the question, which is the more useful finding for health systems.

The concern about explanations held up, and was measured. In April 2025 Anthropic published research testing Claude 3.7 Sonnet and DeepSeek R1, both models with visible reasoning. When the models were given hints that changed their answers, Claude 3.7 Sonnet mentioned the hint about a quarter of the time on average, and R1 about 39 percent of the time. In a separate test, models were trained to exploit deliberately wrong hints. They used the hints in over 99 percent of cases, but mentioned them in their reasoning less than 2 percent of the time in most scenarios. Often they wrote a plausible rationale for the wrong answer instead.

One worry from 2024 eased, though not in a comforting way. Hidden reasoning turned out to be no worse than visible reasoning, because visible reasoning carried the same faithfulness gap. The remedy was the same in both cases: test the answers, not the account of them.

Model developers drew a similar conclusion. In mid-2025 researchers from several AI labs published a joint paper arguing that chain-of-thought monitoring was a valuable but fragile safety opportunity, one that could be lost as training methods changed. The core point for health systems held: a model’s explanation can help a clinician think, but it cannot stand in for testing and monitoring.

On regulation, the two layers diverged. HTI-1’s source attribute requirements took effect for certified developers. In late December 2025, however, the federal health IT office proposed the HTI-5 rule, which would remove those AI “model card” requirements, including source attribute support, from the decision support certification criterion. The proposal alone signaled that certification would not be the main lever for product transparency.

The Section 1557 decision support duty for deployers took effect in May 2025 and became the more durable obligation. The HHS Office for Civil Rights issued guidance to covered entities on the provision in January 2025, before the compliance date. The duty rests with the providers and plans themselves, whoever built the tools they use.

Other bodies filled parts of the gap. The FDA’s Digital Health Advisory Committee met in November 2024 to discuss generative AI in medical devices, and in January 2026 the FDA issued revised guidance on clinical decision support software. In September 2025 the Joint Commission and the Coalition for Health AI published joint guidance on the responsible use of AI in health care. California’s disclosure rule for generative AI patient communications took effect in January 2025. Not one of them treated a model’s explanation as evidence. Each assumed the health system would hold the records.

Product transparency shifted further toward contracts, and deployment transparency stayed with the health system. The practical work of clinical AI transparency now lies largely in two places that health system leaders control. One is the terms they negotiate with vendors. The other is the inventory, validation, monitoring and accountability they build inside their own organizations.

Transparency is not something a model can hand over in a paragraph. It is a set of records, contracts and named responsibilities that a health system assembles around the model. A fluent explanation may make a tool more pleasant to use. Only the health system can show that it is safe and fair to use for its own patients.