Please listen to the article
This audio is automatically generated. Please let us know if you have any feedback.
At a House Subcommittee hearing Wednesday, lawmakers expressed concern about whether artificial intelligence is being used properly in healthcare and called for a stronger guardrail to oversee the rapidly evolving technology.
“As there is only an increase in the application of AI and machine learning, it is paramount that proper surveillance is applied, as all these innovative advances are being utilized throughout the US healthcare ecosystem.”
House Democrats and Republicans said applications ranging from mental health chatbots to advance approval reviews require monitoring of the technology, but have proposed few concrete plans for future regulations or guardrails.
The hearing, called “Exploring opportunities to use artificial intelligence technology to advance healthcare in America,” is at a critical time for healthcare AI. Most healthcare leaders say the technology is promising, but the majority of providers are not using it yet. It cites concerns about data privacy and reliability.
According to Michelle Mello, professor of law and health policy at Stanford University, the lack of federal oversight contributes to its “basic trust deficit.”
The Biden administration has taken several steps towards regulating AI in healthcare, including creating a task force to build regulations, but the Trump administration has halted those efforts. In July, the Trump administration announced an AI adoption plan, which is to support healthcare details and deregulation. This is an approach that is one step further than most witnesses' recommendations at Wednesday's hearing.
“The free space of this rule surprised hospitals, clinicians and patients are worried about the risks of AI, and that fear has been adopted,” Mello said.
Concerns increase beyond previous permits
Several lawmakers raised concerns about the role of AI in previous permitting, particularly for services covered by Medicare advantages.
Payers are facing increasing scrutiny to automate the claims review process. Last year, the country's three largest MA insurance companies discovered it. UnitedHealthCareHumana and CVS – utilized predictive intelligence to limit access to acute care and increase profits.
However, the federal government has recently proposed bringing AI into the claims review process. In July, CMS announced a program that pilots pre-authorized approvals in traditional Medicare for some services the Trump administration says are vulnerable to abuse.
The federal government said it would contract with a pilot program company to use AI for previous approvals. Stanford Mello told representatives that the pilot program would require humans to consider rejecting claims, but she is worried that it could be “prepared” by AI to accept denials.
Some lawmakers at the hearing expressed concern that contracted businesses would receive financial incentives to reduce care.
Rep. Greg Landsman, D-Ohio, called for the program to be “shut down” until more information is available about what Guardrails will be placed in technology companies, so as not to properly deny care to eliminate higher returns.
“If you can deny more and more claims, if you're that AI tech company, you'll get more money. That's going to lead to people getting hurt,” Landsman said.
Push to suppress treatment bots
Many of the hearings focused on regulating AI in mental health care, following media coverage of AI-induced “psychosis” and advice from the American Psychological Association in June that warned that protection is needed for adolescents using AI.
Rep. Raul Ruiz of D-Calif said it was actively harmful to those seeking care, including direct consumer chatbots like ChatGpt.
Lewis referenced the death of 16-year-old Adam Lane. Adam Lane's parents say they were encouraged to commit suicide by ChatGpt. Not only are Democrats worried that similar products may provide the baseline with correct information, such as how users look for local support resources, they can also indulge in user dark thoughts, boundaries that human experts will never intersect.
“Imagine accepting the idea of suicide. Clinicians will keep a poster board with a hotline number and continue a deeper conversation about how to actually complete the suicide,” Lewis said.
Part of the problem is that the market is full of unregulated chatbots making “deceptive and dangerous” claims, said Bail Wright, senior director of healthcare innovation at the American Psychological Association.
In one case, entertainment bots have established themselves as psychologists and engaged in millions of chats. In another case, Wright said the chatbot validated the users' violent thoughts on their families.
“This is unacceptable,” Wright said. “The APA has formally requested an investigation by the (Federal Trade Commission) and the Consumer Product Safety Commission to address these potentially harmful products in order to realize AI promises while protecting patients.”
The APA also called on Congress to enact laws prohibiting misrepresentation of AI as licensed professionals, requiring transparency and human oversight to clinical decisions. The organization also wants to see legislation that requires age-appropriate protection measures and limits access to harmful or inaccurate health content.
However, Wright acknowledged that due to the urgency of AI, more robust solutions could result in time-independent research.
“What we really need is independent research examining what the problem is and what the solution is. And they should be driven empirically, not just throwing spaghetti on the wall because it's meaning at the time,” she said.
More monitoring is required
Witnesses expressed support for several reforms to enhance AI surveillance.
Mello advocated modernizing the Food and Drug Administration's authority to regulate AI. She also called for post-relay monitoring of AI to be a Medicare funding requirement.
Andrew Ibrahim, chief clinical officer of health AI company Viz.ai, is seeking a review of the FDA policy.
“Now, (FDA) uses laws that don't really conform to decades-old laws or laws, authorities, or the way AI is used,” Ibrahim said.
Lawmakers have largely refused to support legislative proposals to enhance AI surveillance, but R-PA. Rep. John Joyce said he plans to introduce the Health Technology Investment Act in the House. The law was originally introduced in the Senate in April.
Some Democrats said they could not focus on modernizing the FDA while the HHS is in turmoil. Agents are fighting for restructuring and staff restructuring. Last week, Director Robert F. Kennedy Jr. fired the director of the Centers for Disease Control and Prevention a month after being confirmed by the Senate. Several other high-ranking CDC personnel remain in response.
“Rome is on fire,” said D-Colo MP Diana Degette. “Rome is on fire. What's more important is the danger that Trump administration's policies put my members and everyone on this panel.”
