IBM Watson’s skills in oncology? Elementary, according to an investigation

The «intelligent» supercomputer that IBM started selling around three years ago as “Watson for Oncology” claiming that it was capable of recommending the best cancer treatments to doctors was the object of an in-depth investigation by STAT, the science-oriented online magazine recently launched by the Boston Globe. The conclusions are striking: the supercomputer «is still struggling with the basic step of learning about different forms of cancer», and when it comes to recommendations «it is biased toward American patients and methods of care».

Investigative science journalists Casey Ross and Ike Swetlitz talked to many IBM executives, artificial intelligence experts and doctors familiar with the use of Watson for Oncology, from South Korea to Slovakia to South Florida: «The interviews suggest that IBM, in its rush to bolster flagging revenue, unleashed a product without fully assessing the challenges of deploying it in hospitals globally» they write. «While it has emphatically marketed Watson for cancer care, IBM hasn’t published any scientific papers demonstrating how the technology affects physicians and patients. As a result, its flaws are getting exposed on the front lines of care by doctors and researchers who say that the system, while promising in some respects, remains undeveloped».
A doctor said it is still «in the toddler stage».

«So far, the only studies about Watson for Oncology are conference abstracts. The full results haven’t been published in peer-reviewed journals — and every study, save one, was either conducted by a paying customer or included IBM staff on the author list, or both. Most trumpet positive results, showing that Watson saves doctors time and has a high concordance rate with their treatment recommendations» the investigation points out. Many of those positive results are likely true, but excess of hype might end up compromising them: «One former IBM employee says the company could become a victim of its own marketing success — the unrealistic expectations it set are obscuring real accomplishments».