The segmentation of white matter hyperintensities of presumed vascular origin on brain MR images.
Small vessel disease plays a crucial role in stroke, dementia, and ageing1. White matter hyperintensities (WMH) of vascular origin are one of the main consequences of small vessel disease and well visible on brain MR images2. Quantification of WMH volume, location, and shape is of key importance in clinical research studies and likely to find its way into clinical practice; supporting diagnosis, prognosis, and monitoring of treatment for dementia and other neurodegenerative diseases. It has been noted that visual rating of WMH has important limitations3 and hence a more detailed segmentation of WMH is preferred. Various automated WMH segmentation techniques have been developed, to provide quantitative measurements and replace time-consuming, observer-dependent delineation procedures.
A review on automated WMH segmentation techniques revealed a key issue: it is hard to compare various techniques3. Each segmentation technique is evaluated on a different ground truth (different number of subjects, different experts, different protocols) and using different evaluation criteria.
This challenge aims to directly compare automated WMH segmentation techniques. The output will be a ranking of techniques that robustly and accurately segment WMH across different scanner platforms and different subject groups.
Participants will containerize their algorithms with Docker and submit these to the organizers. Detailed instructions and easy-to-follow examples are provided and, if needed, the organizers will help with containerization. The organizers will run the techniques on the test data. This guarantees that the test data remains secret and cannot be included in the training procedure of the techniques.
The container / Docker infrastructure guarantees standardized testing in the coming years, and is supported by the UMC Utrecht. The main organizer and the UMC Utrecht agree to commit themselves to future support of this challenge.
The organizers will evaluate all methods according to the evaluation criteria. The results will be published on this website, showing the overall results in a table and detailed results on a separate page per method.
Terms of participation
The WMH Segmentation Challenge is organized in the spirit of cooperative scientific progress. We do not claim any ownership or right to the methods, but we require anyone to respect the rules below. The following rules apply to those who register a team and/or download the data:
- The downloaded data sets, associated reference standard, or any data derived from these data sets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
- All information entered when registering a team, including the name of the contact person, the affiliation (institute, organization or company the team’s contact person works for) and the e-mail address must be complete and correct. Anonymous or incomplete registration is not allowed. If you wish to submit anonymously, for example because you want to submit your results to a journal or conference that requires anonymous submission, please contact the organizers first.
- The data provided may only be used for preparing an entry to be submitted to this challenge. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products, without prior participation in the challenge and approval by the organizers.
- Results of your submission will only be published on the website when a document describing the method is provided.
- If a commercial system is evaluated no method description is necessary, but the system has to be publicly available and the exact name and version number have to be provided.
- The organizers of the challenge will check the method description before your results will be published on the website.
- If the results of algorithms in this challenge are to be used in scientific publications (e.g. journal publications, conference papers, technical reports, presentations at conferences and meetings) you must make an appropriate citation to : Kuijf, H. J., et al. “Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities; Results of the WMH Segmentation Challenge.” IEEE transactions on medical imaging (2019).
- Evaluation of registration results uploaded to this website will be made publicly available on this website (Results section), and by submitting results, you grant us permission to publish our evaluation. Participating teams maintain full ownership and rights to their method.
- Teams must notify the organizers of this challenge about any publication that is (partly) based on the results data published on this website, in order for us to maintain a list of publications associated with the challenge.
- Pantoni, L. Cerebral small vessel disease: from pathogenesis and clinical characteristics to therapeutic challenges. Lancet Neurology 9, 689–701 (2010).
- Wardlaw, J. M. et al. Neuroimaging standards for research into small vessel disease and its contribution to ageing and neurodegeneration. Lancet Neurology 12, 822–38 (2013).
- Caligiuri, M. E. et al. Automatic Detection of White Matter Hyperintensities in Healthy Aging and Pathology Using Magnetic Resonance Imaging: A Review. Neuroinformatics, 13(3), 261-276 (2015).