MIT researchers identified AI challenges healthcare systems must overcome in a paper presented at the Neural Information Processing Systems (NeurIPS 2025) conference in December.
Best-performing AI models on chest X-rays and cancer histopathology images at the first hospital were the worst-performing on up to 75 percent of patients at the second hospital. The AI model failures occurred when models are applied to data other than what they were trained on.
“We demonstrate that even when you train models on large amounts of data, and choose the best average model, in a new setting this ‘best model’ could be the worst model for 6-75 percent of the new data,” says Associate Professor Marzyeh Ghassemi.
The challenge will be how do we leverage AI models working at the Mayo Clinic or MD Anderson at the over thousand healthcare systems including safety net hospitals.