New study: Medical AI is less accurate in assessing treatment outcomes for new patients with schizophrenia

News on January 12, a new study finds that a computer algorithm used to assist doctors in treating schizophrenia patients does not adapt well to fresh data not seen in the previous development process. . As a result, this type of medical AI performs very poorly when it comes to evaluating treatment outcomes for patients it has never been exposed to.

These medical tools use artificial intelligence to discover features in large data sets and predict individual responses to specific treatments, which is the core of precision medicine. Healthcare professionals hope to use this tool to tailor treatment to each patient. In an article published in the journal Science, the researchers noted that the artificial intelligence model could predict treatment outcomes for the patients included in the training sample with a high degree of accuracy. However, when dealing with previously unseen patient data, the model's performance dropped significantly to only slightly better than random guessing.

To ensure the effectiveness of precision medicine, predictive models need to maintain stable accuracy under different circumstances and minimize the possibility of bias or random results.

"This is a big problem that people don't realize yet," said study co-author Adam Chekroud, a psychiatrist at Yale University in New Haven, Connecticut. "This study essentially demonstrates that algorithms still need to be tested on multiple samples."

Accuracy of Algorithms

Researchers evaluated an algorithm commonly used in predictive models of psychosis . They used data from five antipsychotic clinical trials involving 1,513 volunteers diagnosed with schizophrenia in North America, Asia, Europe and Africa. The trials, conducted between 2004 and 2009, measured volunteers' symptoms before and four weeks after taking one of three antipsychotic drugs.

The research team used the data set to train an algorithm to predict the degree of improvement in patients' symptoms after four weeks of antipsychotic medication. First, the researchers tested the accuracy of the algorithm in trials where it was developed, comparing the predictions with the actual effects recorded in the trials and found that the accuracy was high.

They then used a variety of methods to evaluate how accurately the AI ​​model analyzed new data. The researchers trained the model on a subset of data from one clinical trial and then applied it to another subset of data from the same trial. They also train the algorithm on all the data from a trial or set of trials, and then test model performance on other clinical trial data.

It was found that the artificial intelligence model performed poorly in these tests, and the predictions produced by the model appeared to be almost random when applied to untrained data sets. The research team repeated the experiment using different prediction algorithms, but obtained similar results.

Better testing

The study's authors said their findings highlight how clinical prediction models should be rigorously tested on large data sets to ensure their reliability. A systematic review of 308 clinical prediction models for psychiatric outcomes found that only about 20% of the models were validated on data sets other than the sample used for development.

“We should be thinking about model development more like we are developing drugs,” Chekrud said. He explained that many drugs perform well in early clinical trials but run into problems in later stages. "We must strictly abide by the principles of how to develop and test these algorithms. We cannot just do it once and think it is true." (Chenchen)

The above is the detailed content of New study: Medical AI is less accurate in assessing treatment outcomes for new patients with schizophrenia. For more information, please follow other related articles on the PHP Chinese website!