The organization¶
VisioMel began in 2021 when the French Society of Pathology, the French Society of Dermatology, the French Cutaneous Cancers Group, and the National Professional Council of Pathologists decided to build a challenge using clinical variables from a national database on melanoma (RIC-Mel).
The challenge¶
Melanoma is a cancer of the skin which develops from cells responsible for skin pigmentation. It represents 10% of all skin cancers and is among the most dangerous due to high likelihood of metastasizing (spreading). Assessing the risk of melanoma relapse is therefore a vital but difficult task, requiring specialized training and careful examination of microscopic tissue. Machine learning approaches have so far proven helpful for basic tasks like measuring the area of melanomas, but may also help support pathologists in their assessments, especially for melanomas that are more difficult to assess using traditional methods.
A digitized Whole Slide Image (WSI) of melanoma tissue, representative of the images used in the VisioMel challenge to predict relapse risk.
The approach¶
DrivenData worked with the VisioMel team to organize a challenge to predict melanoma relapse within 5 years of the initial diagnosis based on de-identified digitized Whole Slide Images. The data were carefully de-identified and protected under strict user agreements to ensure full patient confidentiality. The train and test sets for the challenge balanced representation of the various risk factors and melanoma characteristics to ensure that the winning model had high predictive ability on varied cases and on unseen data.
The results¶
Challenge participants generated over 600 submissions, and the winning solutions achieved log loss scores of 0.39-0.40 (lower is better) compared to 0.50 for a benchmark model that predicted relapse from tabular clinical features. In terms of area under the ROC curve, winning submissions substantially outperformed the benchmark with AUC scores surpassing 0.80.
The results suggest that computer vision can detect signals in whole slide images that help make more accurate predictions of relapse months to years into the future. VisioMel has also made the images open for ongoing use, practice, and learning on data.gouv.fr here.
"The results show that the algorithms are at least on par with the traditional prognostic factors. In the near future, it will undoubtedly be possible to have a predictive digital signature of melanoma recurrence."
Frédéric Staroz, President of the Conseil National Professionnel des Pathologistes (CNPath)
