AI in Healthcare: The Challenges of Monitoring, Accuracy, and Cost

The Promise of AI in Healthcare: Reducing Costs or Increasing Them?

Artificial intelligence (AI) is a hot topic in healthcare these days. Many believe AI will reduce healthcare costs and improve services. But that’s not the whole story. The AI systems used to improve healthcare and make it more accurate require a lot of trained people to operate and maintain them, which actually increases costs.

AI in Oncology: The Challenges of Accuracy and Timing

For example, oncologists (cancer specialists) are responsible for helping cancer patients make difficult decisions about their treatment, but they often struggle with this. At the University of Pennsylvania Health System, an AI algorithm prompts doctors to discuss end-of-life care priorities with patients—things like pain management, limits to medical treatment, and emotional support. This algorithm predicts the likelihood of death. However, a 2022 study revealed a major flaw in this AI: its accuracy dropped by 7% during the COVID-19 pandemic. Because of this inaccuracy, the AI-based tool failed hundreds of times to prompt doctors to have these crucial conversations with patients, potentially leading many patients to undergo unnecessary chemotherapy (treatment using drugs to destroy cancer cells). Sadly, many institutions don’t even regularly check their AI tools to make sure they are working correctly, further increasing the risk to patients.

The High Cost of Operating AI-Powered Healthcare Systems

Problems with algorithms are something computer scientists and doctors have known about for a long time. But now, it’s also worrying hospital executives and research scientists. To run AI-powered systems properly, constant monitoring and trained staff are essential. This means that running these new tools without errors requires spending much more money on both people and machines than originally anticipated. The initial idea was that AI would reduce healthcare costs and provide better treatment, but in reality, its use currently increases treatment costs by about 20%. Furthermore, many hospitals using AI for patient care don’t have enough money to thoroughly test these AI technologies.

The Growing Role of AI in Healthcare and Its Potential Benefits

AI has become deeply ingrained in the healthcare systems of many countries. AI algorithms are now used for crucial tasks such as accurately predicting the risk of patient death or decline in health, suggesting diagnoses, reducing doctors’ workloads by recording and summarizing appointments, and speeding up the approval of insurance claims.

Technology experts claim that using AI in healthcare can be very profitable, and they believe this technology will become ubiquitous and beneficial in the future. Evidence of this is that the investment firm Bessemer Venture Partners estimates about 20 AI startups working in healthcare will earn approximately 820 million rupees per startup annually.

The Risks of Unmonitored AI Systems in Healthcare

Checking whether these AI-based healthcare improvement products are working correctly is a challenge. And figuring out if they are consistently performing well, or if there’s a bug in their software, is even more difficult.

A recent study by Yale Medicine revealed significant variations in the performance of six “early warning systems” used in healthcare. These systems monitor patients’ conditions and alert doctors when their condition is at risk of sudden deterioration. A highly effective algorithm was used for this in-depth testing, and all the study’s data was analyzed by a supercomputer. The study found a significant difference in the performance of these six “early warning AI systems.”

Why Checking AI Performance in Healthcare is a Major Challenge

Hospitals and other healthcare providers face a major challenge: choosing the best AI algorithm for their specific needs. Not every hospital or doctor has a supercomputer, nor are there “consumer reports” for AI—meaning there’s no report that compares different AI algorithms to show which is best. But the most important thing is that today, we don’t have a standard way to figure out how to check the AI algorithms used in hospitals, how to maintain them, and how to ensure they’re working correctly when they are put into use. Because of this, hospitals are essentially shooting in the dark.

Ambient Documentation: A New AI Tool in Patient Care with Major Risks

A new technology called “ambient documentation” is being used in some doctors’ offices these days. This is an AI-powered assistant that creates a summary of the conversations between doctors and patients—this technology is developed with the goal of improving patient care. There’s significant investment in this technology—recently, investors at “Rock Health,” a major firm investing in digital health, put about 29 billion rupees into these companies. But a major challenge is that there’s no standard for checking the output of these devices. Just imagine, if these devices record patient information incorrectly, how badly it could affect a patient’s treatment. Researchers at Stanford University conducted a study to see if ChatGPT and other “large language models” could accurately summarize patients’ medical histories. They compared the summaries produced by these models with summaries written by doctors. The result was that even in the best cases, the information provided by the models had up to 35% errors—this is a very worrying finding.

The Accuracy of AI Models in Summarizing Medical Histories

Sometimes, there are clear reasons why AI algorithms fail. For example, suppose a hospital previously used tests from a lab called ‘A,’ and the AI algorithm was trained using the test results from that lab. Now, if the hospital starts using tests from a new lab called ‘B,’ the new lab’s test results might be different from lab ‘A’s. Because of this change, the AI algorithm won’t produce accurate results because it hasn’t been trained on the new type of data. But sometimes, problems arise even without any obvious reason.

Unreliable AI in Personalized Medicine: A Case Study

Similarly, a strange problem was found in the Personalized Medicine Program—a program where a patient’s treatment is determined based on their genetic makeup—at Mass General Brigham (a healthcare system in the US) in Boston. Here, an application was being tested to help genetic counselors find information related to DNA. But, surprisingly, when the same question was asked repeatedly, the application gave different answers each time! This clearly shows that this technology still needs a lot of improvement.

The Immense Challenge of Validating AI Models in Healthcare

At Stanford University, a surprising finding came to light: checking just two AI models, to see how reliable they are and give correct information, took eight to ten months and 115 human-hours of work. This shows how difficult it is to check AI systems.

Should AI Monitor AI? The Debate Over Automated Oversight

How to monitor AI is a big debate. Some experts believe AI should monitor AI, along with some data experts. This idea sounds quite appealing, but the reality is that doing so will cost a lot of money, and given the limited budgets of hospitals and the shortage of AI experts, this is a very difficult task. This is a paradox: more AI and experts are needed to monitor AI, while these are precisely the things (money and experts) that are scarce.

The Paradox of AI in Healthcare: More Experts and Funding Needed

Imagine a world where AI is the ‘boss’ of AI—one AI monitors another AI. While this sounds exciting, it’s also a big challenge. Are we really ready to accept this kind of control? And how many more trained people will we need to manage this AI-based monitoring system?

Discover an Ocean of Educational Resources! We provide a wide variety of learning materials that you can access through our internal links.

Nuutan.com is your gateway to a world of information and academic accomplishment. Books in e-book form, multiple-choice question-based online practice tests, practice sets, lecture notes, and essays on a wide range of topics, plus much more!

https://www.nuutan.com/