A chatbot’s history sidebar is a records system with someone else’s retention policy

The week

In March 2023, staff in hospitals, clinics and health plans were already using ChatGPT to draft letters, summarize documents and tidy up notes. Most of them thought of it as a clever calculator. Then, for a few hours, some users saw something strange in their own accounts: the titles of conversations they had never had.

The tool was less than four months old. OpenAI had released ChatGPT to the public on November 30, 2022, and it spread faster than almost any consumer service before it. The weeks around this incident were among the busiest in its short history. OpenAI released GPT-4 on March 14. Microsoft announced Microsoft 365 Copilot on March 16. Google opened access to its Bard chatbot on March 21. OpenAI announced plugins for ChatGPT on March 23, letting the assistant reach outside services.

Some large employers had already moved to restrict staff use. Several big banks, JPMorgan Chase among them, were reported in February 2023 to have limited employee use of ChatGPT. Health systems were mostly further behind. Many were still digesting a December 2022 bulletin from the HHS Office for Civil Rights warning that tracking technologies on patient-facing websites could disclose protected health information to third parties.

On March 20, 2023, OpenAI took ChatGPT offline after users reported seeing the titles of other people’s conversations in their history sidebar. The history sidebar is the list down the side of the screen that lets a user reopen past chats, and each entry is labeled with a short title generated from what the conversation was about.

On March 24 the company explained that a bug in an open-source library used for caching had caused some users to see data belonging to others. The library was redis-py, a widely used client for the Redis data store. Under certain conditions, a request that was cancelled partway through could leave its response waiting on a shared connection, and the next user’s request could receive it. Open-source software is code that is freely shared and widely reused by many companies. Caching is a technique for speeding up a system by keeping recently used information close at hand. When a cache mixes up whose information is whose, one person can be handed another person’s data.

The exposure went beyond conversation titles. OpenAI said that during a window of about nine hours on March 20, from roughly 1 a.m. to 10 a.m. Pacific time, payment-related details of about 1.2 percent of ChatGPT Plus subscribers active at the time may have been visible to other users. Those details included name, email address, payment address, and the last four digits and expiry date of a card, though not full card numbers. OpenAI said it had fixed the bug and contacted affected users. It also said it had added checks so that data returned from the cache matched the user who requested it. It had reached out to the library’s maintainers with a fix.

A related policy change had taken effect three weeks earlier. The API is the interface developers use to build the model into their own software. From March 1, 2023, OpenAI said it would no longer use data submitted through it to train its models unless customers opted in. The consumer ChatGPT service was not covered by that change, and conversations there could still be used for training unless a user opted out.

The story did not stop there. On March 30 Italy’s data protection authority, the national regulator responsible for enforcing privacy law, ordered a temporary limit on ChatGPT’s processing of Italian users’ data, citing the incident among other concerns. The order also questioned the legal basis for using personal data to train the models, the accuracy of information the service produced about people, and the absence of age checks for children. OpenAI responded by blocking access to ChatGPT in Italy while it worked with the regulator.

For health organizations, the payment details mattered less than the fact that the sidebar existed at all. Every question a nurse, coder or physician typed into the tool was being kept, labeled and stored on someone else’s servers.

By the standards of 2023’s breaches, this one was minor. It was fixed quickly, the data involved was limited, and the vendor explained it within days. Its significance for health systems lay elsewhere.

Many staff treated a chatbot like a calculator: ask a question, get an answer, move on. The bug showed that each conversation was stored, indexed and displayed back. It was a record on a vendor’s servers. Treating AI conversations as a data store, rather than a passing exchange, raises three governance questions for any health organization: what goes in, how long it stays, and who can see it. Those are the same questions a health system already asks of its electronic health record, its email and its billing software. The challenge is recognizing that a chat window belongs on that list.

The answers differ by type of organization. A hospital or health system is a HIPAA covered entity. So is a health plan, which handles claims, appeals and member records. A billing company, transcription service or revenue-cycle vendor is usually a business associate. Its own staff using a chatbot can put its clients’ data at risk, and its contracts with its own vendors need subcontractor agreements. A life sciences company is generally not a covered entity. It still holds clinical trial data, trade secrets and, increasingly, consumer health data under state laws such as Washington’s My Health My Data Act, signed in April 2023.

The sections below start with what an AI assistant is and why HIPAA reaches it. They then take the three questions in order, run one ordinary task through all three, and end with what a hospital looks like after a year of work on them.

A dictation service with a memory

An AI assistant like ChatGPT is a service that runs on a company’s computers rather than the user’s own device. When a staff member types a question, the text travels over the internet to the vendor, is processed by a large language model, and the answer comes back. A large language model is software trained on enormous amounts of text to predict and generate fluent language. It can draft, summarize and rewrite with striking skill, which is exactly why busy staff find it so useful.

Three features matter for governance. First, the text leaves the organization: it is sent to a cloud service, meaning computing run by an outside company in its own data centers. Second, it is usually stored, so the user can come back to the conversation later. Third, the vendor may use it for its own purposes, such as improving its models or monitoring for abuse, depending on the product and the terms.

How a caching bug crosses users

The March 2023 failure shows why “stored” also means “shared infrastructure.”

Large online services keep frequently used data in a fast cache, such as Redis, so they do not have to rebuild it from the main database each time. Many users’ requests travel over a pool of shared connections to that cache. Each request is supposed to receive only its own answer.

The bug broke that pairing. If a request was cancelled at the wrong moment, its response stayed queued on the connection, and the next request on that connection could receive it. Nobody attacked anything. A small error in widely reused open-source code was enough to hand one person’s data to another. Health systems run the same kinds of shared components in their own patient portals and apps, which is why the failure mode should be familiar.

Consumer, API and enterprise products carry different terms

A vendor’s brand name says little about how data is handled. In March 2023, OpenAI’s consumer service and its API ran under different data rules. API data was no longer used for training by default, while consumer conversations could be. Enterprise products with negotiated terms came later. Other AI vendors had similar splits between their consumer apps, their developer platforms and their cloud offerings.

For a health system, the product is the unit of review. A contract covering a vendor’s cloud platform may not reach the chat app a clinician installs on a phone. Procurement and privacy teams need to know which product, under which account and which terms, staff are actually using.

The nearest comparison is a dictation service. If a physician phoned an outside company to transcribe a letter about a patient, nobody would doubt that the company now held patient information, and that a contract, security checks and retention rules were needed. An AI assistant is the same in substance. It just feels more like a private conversation with a piece of software.

In healthcare, the relevant law is HIPAA, the federal law that governs how health providers, health plans and their vendors handle patient information. Under HIPAA, protected health information, or PHI, is health information that can be linked to an individual. An outside company that handles PHI on a health organization’s behalf is a business associate. The two sign a business associate agreement, or BAA, a contract in which the vendor commits to protect the information and follow HIPAA’s rules.

HIPAA has a narrow exception for “conduits,” such as internet service providers and couriers that only transmit information and do not store it beyond what transmission requires. An AI assistant does not fit that exception. It processes the text, generates new text from it and keeps the conversation. Covered entities that rely on the conduit idea to justify chatbot use are on weak ground.

What goes in

The first question is classification: deciding which kinds of information are allowed into which tools. Clinicians and staff use AI assistants with whatever helps them work. That can mean a draft patient letter, a referral summary, a denial appeal, a spreadsheet of scheduling data or a block of clinical notes to tidy up.

The titles exposed by the bug were generated from conversation content. Even the metadata, the descriptive information about a conversation rather than the conversation itself, could reveal what someone was working on. In a hospital that can include a patient’s condition. A title like “appeal letter for chemo denial” says a lot, even without a name attached.

Whether a title alone is PHI depends on context. A title like that, held in an account linked to an identifiable nurse at a named hospital, is a clue rather than a complete record. Combined with dates and the rest of the conversation, it can point to a specific patient. Privacy officers should treat titles and other metadata as part of the record rather than as harmless labels.

Under HIPAA, entering protected health information into a consumer service with no business associate agreement is, in most cases, a disclosure the Privacy Rule does not permit. In early 2023, consumer chatbots offered no such agreement. Many staff did not see a chatbot as a cloud service at all. A policy forbidding PHI in unapproved cloud services did not obviously apply, in their minds, to a chat window.

Closing the gap

Closing that gap means naming AI assistants in acceptable use and data handling policies, the documents that tell staff what they may and may not do with organizational systems and data. Those policies should state which data classes may go into which approved tools. They should also give practical examples of what de-identification means, because “remove the name” falls well short of HIPAA’s standards. De-identification is the process of stripping information so that it can no longer reasonably be linked to a person. Dates, locations, rare conditions and record numbers can all identify someone even when the name is gone.

Why free text is hard to de-identify

HIPAA offers two routes to de-identification. The Safe Harbor method removes eighteen specified identifiers, including names, most dates other than the year, geographic units smaller than a state with limited exceptions, phone numbers, record numbers and full-face photographs. It also requires that the organization have no actual knowledge that what remains could identify someone. The Expert Determination method relies on a qualified expert’s judgment that the risk of re-identification is very small.

Clinical narrative defeats Safe Harbor easily. A referral letter mentions the date of an admission, the town the patient was transferred from and the name of the treating physician. A note about a rare disease in a small community can identify a patient by itself. Staff who have been told to “remove the name” will miss most of this, and automated scrubbing tools reduce the risk without eliminating it.

How the risk varies by organization

A large academic medical center has a security team and data loss prevention tools. Its research office already governs data sharing under data use agreements and institutional review boards. Its risk is volume: thousands of staff with many legitimate reasons to summarize text. A small ambulatory network may have no data loss prevention at all and rely on a managed service provider for IT. For it, browser controls, device management and a short approved-tools list are the realistic first steps.

Health plans face concentrated temptation in claims, appeals and member services, where much of the work is drafting letters from case files. Life sciences firms worry as much about trade secrets and unpublished trial results as about patient data. Their controls look similar, but the classification categories differ.

Matching data classes to tools

A workable policy ties data classes to tool tiers. Public and general internal content, such as a draft newsletter or a generic policy summary, can go into a wider range of tools. Confidential business information, such as contract terms or unreleased financials, belongs only in tools with enterprise terms. PHI belongs only in tools covered by a business associate agreement for that product.

Approval does not remove HIPAA’s other limits. The minimum necessary standard still applies to most uses of PHI, so staff should include only what a task requires. A summary of a long chart may need the clinical history but not the patient’s address or insurance number. Teaching that habit is cheaper than detecting its absence afterward.

Technical controls back the policy. Web filtering can steer users away from unapproved tools and toward approved ones. Data loss prevention, software that watches for sensitive information leaving the organization, can be tuned for medical record numbers and clinical content. Neither control is perfect, but together they make the safe path the easy one.

Training should be specific to roles. Revenue-cycle staff drafting appeals face different temptations from physicians summarizing charts. A billing specialist may be tempted to paste a denial letter and a patient’s history to get a strong appeal. A physician may want a long, messy note turned into a tidy summary. Each needs examples drawn from their own work, showing what is fine, what is not, and which approved tool to use instead.

A ban is the obvious alternative, and some organizations chose it in 2023. Bans have a place for tools that cannot be governed. On their own, they tend to push use onto personal phones and home computers, where the organization sees nothing. A ban paired with an approved alternative works better than a ban alone, because it gives staff a lawful way to get the same help.

How long it stays

The second question is retention: how long information is kept, and who decides. Consumer AI services kept conversation histories by default so users could return to them. At the time, OpenAI also used consumer conversations to improve its models unless users opted out. Information typed into a prompt could therefore persist in at least two forms: as a stored conversation and as training data. A prompt is the text a user types into the assistant.

The vendor’s own controls changed quickly. In April 2023, OpenAI let ChatGPT users turn off chat history. Conversations started with history off were not used for training, and the company said it would keep them for 30 days to monitor for abuse before deleting them. That gave individuals more control. It gave organizations none, because the setting belonged to each user, who could switch it back on at any time.

Health organizations face retention pressure in both directions. Privacy principles and state laws push toward deleting data when it is no longer needed. Medical records rules, litigation holds and HIPAA’s documentation requirements push toward keeping certain material. A litigation hold is a legal instruction to preserve relevant information because of a lawsuit or investigation.

The retention rules that apply

Several rules apply at once. HIPAA requires covered entities and business associates to keep their required documentation, such as policies, risk analyses and certain records of disclosures, for six years. Medical record retention periods are set mostly by state law and vary by state and by type of record. Under federal discovery law, parties must preserve relevant electronically stored information in their possession, custody or control once litigation is reasonably anticipated.

The last rule creates the sharpest problem. Whether a conversation in an employee’s personal AI account is within the organization’s control is an open and fact-specific question. If it is, the organization has a preservation duty it cannot meet. If it is not, the organization has lost track of work done on its behalf. Neither outcome is comfortable, and counsel should be involved in deciding how AI use is covered by the organization’s hold procedures.

A consumer tool controlled by the individual user and governed by the vendor’s terms meets neither need. The organization cannot delete what it does not control, and it cannot produce what it never captured. If a patient complaint or lawsuit later turns on what a staff member wrote with AI help, the organization may have no access to that conversation at all, while the vendor still holds it.

Enterprise agreements and the contract checklist

Enterprise agreements, which began to appear later in 2023, changed the picture. These are business versions of AI tools, sold to organizations rather than individuals, with contracts negotiated accordingly. They offered commitments on retention periods, exclusion from model training and, for some vendors and products, a business associate agreement.

Those are procurement questions, and they belong in the standard contract checklist for any AI tool. The checklist should cover:

• Where conversations are stored and for how long.
• Whether the customer can set retention.
• Whether conversations are excluded from model training.
• What logs the vendor keeps for abuse monitoring.
• Whether the vendor will sign a BAA for the specific product in use, as distinct from its platform generally.

That last point catches many buyers. A vendor may offer a BAA for one product and not another, and a contract that covers the platform in general terms may not cover the tool staff actually use.

Each item on the checklist has a trade-off. Short retention reduces exposure but can make it harder to investigate misuse or to reconstruct how a document was drafted. Excluding data from training is now standard in enterprise terms, but it does not end retention for abuse monitoring, which some vendors keep for a set period. Audit logs are valuable only if someone reviews them, and storing them adds cost. A buyer should decide what it wants each setting to achieve before negotiating.

Which outputs are records

There is also a records question inside the organization. If a clinician uses an approved assistant to draft part of a note or a patient message, some of that work may belong in the designated record set. Under HIPAA, the designated record set is the collection of records an organization uses to make decisions about a patient, and which the patient has a right to see.

Health information management leaders, the people responsible for the integrity of medical records, should decide which AI-assisted outputs are records. They should make sure the final versions land in the EHR, the electronic health record, rather than only in the chat history. A patient letter that exists only in a vendor’s chat log is a record the organization cannot reliably find, correct or produce.

Patient rights make this concrete. Patients may request copies of information in their designated record set and may ask for it to be amended. If an AI-drafted message forms part of a care decision, the patient can ask to see it. If it contains an error, the organization must be able to locate and correct the version that matters. A chat history held in a vendor account cannot serve either purpose.

Who can see it

The third question is access. The bug was a failure of the vendor’s own separation between users. Ordinary access risks matter more day to day. Shared logins, unmanaged personal accounts and conversations that persist after an employee leaves are common in consumer tools. An organization cannot revoke access to an account it does not know exists.

Finding the accounts

Discovery comes before control. An organization can find unmanaged AI use through web proxy and firewall logs, through cloud access security broker tools that classify software-as-a-service traffic, and through expense reports showing paid subscriptions. Sign-ups made with work email addresses are another trail. Some enterprise AI products let an organization verify its email domain and bring accounts created with that domain under central management.

No single method finds everything. Personal accounts on personal phones over home networks remain invisible. The aim is to find enough use to size the problem and move the bulk of it onto approved tools. The approved tool then has to be easy enough that staff have little reason to go elsewhere.

Suppose a scheduler at a community hospital sets up a personal ChatGPT account using her work email and uses it for two years to draft patient reminders and reorganize clinic schedules. When she leaves for another job, her badge is deactivated and her hospital accounts are closed. Her AI account, and two years of conversation history, remain under her control. The hospital has no record that it ever existed.

Offboarding procedures are built around accounts the organization issued. HIPAA’s Security Rule asks covered entities to have procedures for terminating access when employment ends, and most organizations meet that through their identity system. A self-created account using a work email address is outside that system unless someone has found it first.

Shared logins create a different problem. When a department shares one paid account to save on licenses, every member can read every other member’s conversations, and no audit trail can say who typed what. The Security Rule requires unique user identification for systems that hold electronic PHI. An AI tool used with patient data should meet the same standard as any clinical application, and a shared login fails it by design.

Familiar controls, applied to a new tool

The controls are familiar from other software:

• Require single sign-on so access follows employment and ends with it. Single sign-on means staff log into the tool with their organizational credentials, so when those are switched off, access ends too.
• Prefer tools that support audit logs the organization can review, showing who used the tool and when.
• Restrict sharing features that create public links to conversations.
• Treat the AI vendor as a business associate where PHI is involved, with the security review and incident notice terms that implies.

The vendor’s own isolation between customers also deserves questions. In an enterprise product, each customer organization is a tenant, sharing the vendor’s systems but supposedly walled off from the others. A caching bug that crossed user boundaries in a consumer product could, in principle, cross tenant boundaries in an enterprise one. Contracts should require prompt notice of any such failure.

Questions for the vendor about isolation

Buyers can ask for evidence rather than assurances. Useful items include independent audit reports such as SOC 2 Type II and a summary of recent penetration testing. Buyers can also ask for a description of how customer data is separated in storage and in caches, and a list of subprocessors that touch customer content. Buyers should also ask how the vendor tests for cross-customer exposure after code changes. A vendor that has never thought about that question has told the buyer something useful.

When a vendor bug becomes a HIPAA breach

The legal analysis depends on whether a business associate agreement exists. If a covered entity uses an AI tool under a BAA and the vendor suffers a bug that exposes PHI, the vendor must notify the covered entity. The HIPAA breach notification rule allows no more than 60 days after discovery, and many contracts set shorter periods. The covered entity then assesses whether a breach occurred, weighing the nature of the data, who saw it, whether it was actually viewed and how far the risk was mitigated.

Without a BAA, the problem starts earlier. The staff member’s act of pasting PHI into an unapproved tool may itself be an impermissible disclosure, whether or not the vendor ever suffers a bug. The organization may then have to assess that disclosure as a potential breach. A health system that discovers widespread unapproved use may face a large and awkward investigation, which is one more reason to provide an approved path early.

Regulators showed they would treat these stores as personal data in the full sense. Italy’s order a week later cited the incident alongside broader concerns about legal basis, transparency and age verification. For US health systems, the parallel authority was their own HIPAA program, applied to a new kind of system.

One appeal letter, three questions

A single, ordinary piece of work shows all three questions at once. Take a claims nurse at a regional health plan, the reviewer, with a stack of appeals to answer before the end of the week. One appeal concerns a patient whose cancer treatment was denied. She has the patient’s letter, the physician’s notes and the denial reason in front of her.

On the question of what goes in, the temptation is obvious. Pasting all three documents into a consumer chatbot and asking for a clear, sympathetic reply would save her an hour. It would also send identifiable health information to a vendor with no business associate agreement, which is the disclosure HIPAA is designed to prevent. If the plan has an approved tool, with a BAA, she can do much the same work there. If it does not, the realistic choices are to do the work by hand or to break the rules, and under deadline pressure many people choose the second.

The pressure is structural. Health plans handle appeals under regulatory timelines, and reviewers are measured on throughput and accuracy. An approved tool that takes ten minutes to open and requires a separate login will lose to a browser tab that is already open. The design of the approved path is a compliance control in its own right.

On the question of how long it stays, the consumer path leaves a copy of that patient’s story in the reviewer’s personal history. It may also end up in the vendor’s training data, for a period the plan cannot see or set. The approved path keeps it under a retention period the plan has chosen, with the final reply saved into the plan’s own systems where it belongs.

On the question of who can see it, the consumer path means the conversation is visible to anyone who can get into the reviewer’s personal account, and it stays with her if she leaves. The approved path ties access to her work login and leaves an audit trail.

The reviewer is no bad actor; she is trying to help a patient quickly. The approved path needs to be as quick and easy for her as the consumer one.

The strongest counter-argument

Some clinicians and managers argue that this framing overstates the risk. Vendors have strong incentives to protect data, the March 2023 bug was quickly fixed, and the productivity benefit of AI drafting is real. Heavy governance could slow adoption while staff burnout stays high.

The argument has merit on productivity and none on accountability. The risk at issue is not mainly that a vendor will be careless. It is that the organization cannot show where patient information went, how long it was kept or who could see it. Those are obligations HIPAA places on the covered entity, whatever the vendor’s record. Approved tools address both concerns at once, which is why the governance work and the productivity case point the same way.

The hospital, a year later

Suppose the same community hospital takes this seriously for a year. It now has an approved AI assistant, bought under an enterprise agreement with a BAA for that specific product. A BAA for one service does not cover every tool the vendor offers. The agreement also provides configurable retention and a commitment that its data will not be used for training. Staff log in through single sign-on, so personal AI accounts and their histories no longer leave with former staff. The acceptable use policy names AI tools explicitly, because generic rules about cloud services are rules staff do not connect to a chat window. It lists what may go into which tool and includes role-specific examples, including why removing the name falls short of de-identification when clinical details, dates and locations can still identify a patient.

Web filtering nudges staff from consumer chatbots to the approved one; an outright ban with no approved alternative tends to push use underground onto personal phones and accounts. Health information management has decided which AI-assisted documents are records and how they reach the EHR, so patient communications do not live only in chat histories. The compliance team can pull audit logs when it needs to.

The hospital has also folded AI into its HIPAA risk analysis, the documented assessment of risks to electronic PHI that the Security Rule requires. The approved assistant appears in its inventory of systems, with its data flows, its BAA, its retention settings and its access controls recorded. If the Office for Civil Rights asked how the hospital governs AI tools, that analysis would be the first document it produced.

It tracks a few measures. They include the share of AI use that runs through single sign-on and the number of data loss prevention alerts involving AI sites each month. They also include the count of business associate agreements covering AI products and the completion rate for role-specific training. The trend in those numbers shows whether the approved path is winning.

The work has a cost. Enterprise licenses are priced per user, so the hospital has had to decide which roles get access first. It started with roles where drafting is heavy and the benefit is clear, such as patient communications, appeals and clinical documentation support, and widened access as budgets allowed. Staff outside the first wave still have a channel to request access, which reduces the pull toward personal accounts.

Its incident response plan now names the AI vendor. The plan lists the vendor’s security contact, the notice period in the BAA and the internal owner who would assess a vendor-reported exposure. The hospital has rehearsed a scenario in which the vendor reports that some customers’ conversations were visible to others, which is the March 2023 incident restated for an enterprise tool.

Staff still use AI, now through a tool the hospital has approved, configured and can audit.

What the privacy officer needs to know about AI use

1. Which AI assistants are our staff actually using today, including personal accounts on work devices?
2. Does our acceptable use policy name AI tools explicitly, and does it say which kinds of information may go into which ones?
3. Do we have an approved AI tool covered by a business associate agreement for that specific product, with retention we can set and conversations excluded from training?
4. What happens to an employee’s AI conversation history when they leave?
5. If the vendor suffered a bug that exposed one customer’s data to another, how quickly would our contract require them to tell us?

With hindsight

The Italian episode had a long tail. Italy’s regulator lifted its restriction at the end of April 2023 after OpenAI made changes. In December 2024 it fined the company €15 million, and among its findings was a failure to notify the authority of the March 2023 breach. OpenAI challenged the decision, and in March 2026 a court in Rome upheld its challenge. Even a small bug produced years of regulatory and legal proceedings.

The market moved faster than the courts. In April 2023 OpenAI let users turn off chat history. In August 2023 it launched ChatGPT Enterprise, promising that business data would not be used for training and adding single sign-on and an admin console. Microsoft 365 Copilot became generally available to enterprise customers that November. Enterprise AI agreements soon routinely offered no-training commitments, configurable retention and single sign-on, and some vendors began offering business associate agreements for specific products.

The “what goes in” problem appeared well beyond healthcare. In spring 2023, reports that Samsung engineers had pasted confidential source code into ChatGPT led the company to restrict generative AI tools on its devices and networks. The episode became a standard example in corporate AI policies, because it showed the risk came from ordinary staff trying to get work done.

Health regulation began to address AI directly, though mostly in other areas. ONC’s HTI-1 rule, finalized in December 2023, set transparency requirements for decision support tools in certified health IT. HHS’s May 2024 rule under Section 1557 of the Affordable Care Act addressed discrimination through patient care decision support tools. California’s AB 3030 required providers to disclose generative AI use in certain patient communications from January 2025. In December 2024, HHS proposed a major update to the HIPAA Security Rule that would require a written inventory of technology assets, which would include AI tools that handle electronic PHI.

Then retention took an unexpected turn. In May 2025, in the New York Times copyright case, a federal court ordered OpenAI to preserve consumer chat logs it would otherwise have deleted. That obligation was largely lifted that autumn. The episode underlined that a vendor’s retention can be set by a court the customer has no part in. A deletion setting in a contract is a promise made under normal conditions; a court order in someone else’s lawsuit can override it. For health systems, that is a strong argument for keeping sensitive information out of tools the organization does not govern, and for knowing exactly what has gone into the tools it does. Retention is a setting a buyer chooses and also a risk the buyer inherits from everything else that happens to the vendor.

Sharing features caused trouble too. In mid-2025 OpenAI withdrew an opt-in option that let shared ChatGPT conversations be indexed by search engines, after reports that personal material shared that way was turning up in search results. Restricting public sharing links, one of the controls described above, turned out to be well founded.

The three questions now reach further, because assistants can read and act on information as well as respond to what is typed. They now extend to assistants connected to EHRs and email, where what goes in includes whatever the tool can reach. In July 2024 NIST published a profile of its AI Risk Management Framework specific to generative AI, covering risks such as data privacy and information integrity. Ambient documentation tools, which listen to a clinical visit and draft the note, spread rapidly through large health systems in 2024 and 2025. They raise the same three questions about audio recordings and transcripts as chatbots raise about typed text. Classification, retention and access remain the right frame. They apply to a much larger surface than a single chat window, and the governance work a health system does today should be sized accordingly.