Developing a New Score: How Machine Learning Improves Risk Prediction

Disease risk scores are being furthered by machine learning algorithms, including a new risk score developed by IBM Research.

Composite risk scores have been used for decades to identify disease risk and health status in the general population. However, current approaches often fail to identify people who would benefit from intervention or recommend unnecessary intervention. Machine learning promises to improve accuracy, ensuring targeted treatment for patients that need it and reducing unnecessary intervention.

Framingham Risk Score, the gold standard for predicting the likelihood of heart disease, predicts hospitalizations with about 56% accuracy. It uses factors such as age, gender, smoking, cholesterol levels, and systolic blood pressure to calculate risk. It doesn't ask for family history, ethnicity, or physical activity level, all of which play a role in heart health.

A stroke-risk scoring system called CHADS2 (Cardiovascular Disease-Heart Attack-Diabetes) fares better, with an 82% accuracy rate, but can misclassify high-risk patients as moderate risk, according to a 2001 study.

Because machine learning algorithms analyze large data sets quickly, they predict disease risk with greater accuracy and can predict a patient's odds of hospital admission and readmission. Armed with this information, hospitals can administer medication or make lifestyle recommendations to keep patients out of the hospital. Considering Medicare payments (or lack of penalties, rather) hinge on readmission rates, hospitals have a financial incentive to leverage their data.

The availability of electronic health records (EHR) makes this sort of analysis possible. "With EHR, we have new types of data we didn't have access to before," said Yannis Paschalidis, professor of engineering and the director of the Center for Information and Systems Engineering at Boston University. "In addition, algorithms and computational tools have improved. We can use larger data sets, come up with decision-making classifiers, and ask more questions."

Paschalidis and colleagues at Boston University’s Center for Information and Systems Engineering recently used machine learning to better predict hospitalizations in heart disease—America's Number One killer—and diabetes cases. Working with 50,000 EHRs from Boston Medical Center and the Brigham and Women’s Hospital, Paschalidis and team could predict heart disease and diabetes-related hospitalizations about a year in advance with an 82% accuracy rate—a marked improvement over the Framingham Risk Score.

"We had information on the drugs patients take, whether they had visited the ER . . . everything in the system over a 10-year period," Paschalidis said. "We could then use a larger set of predictive variables to determine risk, which gave us more accurate predictions."

IBM Research recently developed a new risk score, MELD-Plus, which more accurately predicts short-term mortality of patients with established cirrhosis following a hospital admission. The United Network for Organ Sharing uses current standards—MELD (Model for End-Stage Liver Disease) or MELD Na, which factors in sodium levels—for predicting which patients need a liver transplant more urgently.

To develop an improved MELD, IBM Research used records of nearly 315,000 patients who received care either at Massachusetts General Hospital (MGH) or Brigham and Women’s Hospital (BWH) between 1992 and 2010. The study identified 4781 cirrhosis-related admissions, 778 of which resulted in death within 90 days of discharge.

IBM researcher Uri Kartoun, PhD, launched the project during his three-year postdoctoral training program at Massachusetts General Hospital, working with doctors Stanley Shaw and Kathleen Corey. They found that MELD-Plus predicted mortality rates with about 10% greater accuracy than MELD and MELD Na.

Unlike MELD, which specifies variables in advance, MELD-Plus uses an unbiased feature selection. "We put into the model any possible laboratory types available to us, in addition to demographics, indications, and other data," Kartoun said. "Machine learning selects the most informative variables. In our case, machine learning algorithms pointed to nine variables as the most informative. By using a small subset of variables, you can get the same level of prediction accuracy as using the entire set. It's not a novel methodology, but we find it very effective."

Shortly after joining IBM Research, Kartoun and Kenney Ng, PhD, who manages IBM Research's Health Analytics group, used Kartoun's linear model equation, which he had used on the hospital patient data, on the IBM Explorys Network database, which contains medical records for more than 18 million patients. "We achieved consistency," Kartoun said. Again, MELD-Plus achieved an accuracy rate similar to what he found with the hospital records. PLOS ONE published the full study in late October.

"I'm hoping now that other researchers will try MELD-Plus, and hopefully it's going to perform better for them than other scores," said Kartoun. "For MELD-Plus to be really useful it should receive broader acceptance by multiple institutions."

Predictive Models and Value-Based Care

Now that more than 95% of hospitals have adopted EHR technology, more of them are incorporating the applications and data platforms needed to make better use of EHRs' advanced machine learning and analytics capabilities.

"Within a number of hospitals, there is an effort to develop predictive models using machine learning," said Paschalidis. Boston Medical Center is developing predictive models to predict readmission within 30 days of discharge from general surgery. Texas Health Harris Methodist Hospital Hurst-Euless-Bedford uses predictive analytics to assign members a risk score for congestive heart failure readmission with the aim of keeping patients out of the hospital.

"There are certain tasks that cannot be implemented by a single or multiple human beings," said Kartoun. "Given we have access to millions of EHRs, machine learning algorithms can identify higher risk patients much better than physicians. A hospital has the potential to determine a health strategy—to treat and follow up with those patients differently, and make decisions related to procedures and indications that will improve outcomes."

Filed Under
500 characters remaining