Building the machine learning model isn't enough

Clinical and operational machine learning models are gaining ground at hospitals and health systems throughout the country, and new ones are evolving rapidly.

But at this point, the challenge is not so much development of new models, as effectively evaluating their use, said panelists at the HIMSS Machine Learning and AI for Healthcare Forum this week. (HIMSS is the parent company of Healthcare IT News.)

"As there has been an explosion of data, an explosion of off-the-shelf software you can download, there have been more and more teams just downloading model learning tools to be able to build preliminary models," said Suchi Saria, director of the Machine Learning and Healthcare Lab at Johns Hopkins University, during a fireside chat session Tuesday with STAT News' Casey Ross.  

"What we're seeing is people come up with a preliminary model, they don't know how to evaluate it – because they have a model, they think it's one-and-done," said Saria.  

In the absence of evaluation, she said, teams are trying to put models into practice without an understanding of what success looks like.  

As a researcher, Saria said it can often take astronomically more effort to understand