Details

Task

The segmentation of white matter hyperintensities of presumed vascular origin on brain MR images.

Background

Small vessel disease plays a crucial role in stroke, dementia, and ageing1. White matter hyperintensities (WMH) of vascular origin are one of the main consequences of small vessel disease and well visible on brain MR images2. Quantification of WMH volume, location, and shape is of key importance in clinical research studies and likely to find its way into clinical practice; supporting diagnosis, prognosis, and monitoring of treatment for dementia and other neurodegenerative diseases. It has been noted that visual rating of WMH has important limitations3 and hence a more detailed segmentation of WMH is preferred. Various automated WMH segmentation techniques have been developed, to provide quantitative measurements and replace time-consuming, observer-dependent delineation procedures.

A review on automated WMH segmentation techniques revealed a key issue: it is hard to compare various techniques3. Each segmentation technique is evaluated on a different ground truth (different number of subjects, different experts, different protocols) and using different evaluation criteria.

This challenge aims to directly compare automated WMH segmentation techniques. The output will be a ranking of techniques that robustly and accurately segment WMH across different scanner platforms and different subject groups.

Method

Participants will containerize their algorithms with Docker and submit these to the organizers. Detailed instructions and easy-to-follow examples are provided and, if needed, the organizers will help with containerization. The organizers will run the techniques on the test data. This guarantees that the test data remains secret and cannot be included in the training procedure of the techniques. This workflow has proven successful for previous MICCAI challenges.

Results

The organizers will evaluate all methods submitted before the deadline, according to the evaluation criteria. The results will be presented during the challenge session at the MICCAI, on 14 September 2017. During this challenge session, participating teams should present their method.

Journal publication

Results of the challenge will be summarized in a journal paper, to be submitted after the MICCAI. All participating teams who submitted before the challenge deadline and presented their method at the challenge session at MICCAI will be included in the paper. Each team is allowed two co-authorships on the paper.

Future submissions

The challenge will remain open after the conference deadline. New methods can be submitted and join the challenge. Results as presented at the MICCAI will be frozen and presented separately on this website. Next, there will be an up-to-date ranking available on this website. The container / Docker infrastructure guarantees standardized testing in the coming years, and is supported by the UMC Utrecht. The main organizer and the UMC Utrecht agree to commit themselves to future support of this challenge.

Terms of participation

The WMH Segmentation Challenge is organized in the spirit of cooperative scientific progress. We do not claim any ownership or right to the methods, but we require anyone to respect the rules below. The following rules apply to those who register a team and/or download the data:

  • The downloaded data sets, associated reference standard, or any data derived from these data sets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
  • All information entered when registering a team, including the name of the contact person, the affiliation (institute, organization or company the team’s contact person works for) and the e-mail address must be complete and correct. Anonymous or incomplete registration is not allowed. If you wish to submit anonymously, for example because you want to submit your results to a journal or conference that requires anonymous submission, please contact the organizers first.
  • The data provided may only be used for preparing an entry to be submitted to this challenge. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products, without prior participation in the challenge and approval by the organizers.
  • Results of your submission will only be published on the website when a document describing the method is provided.
  • If a commercial system is evaluated no method description is necessary, but the system has to be publicly available and the exact name and version number have to be provided.
  • The organizers of the challenge will check the method description before your results will be published on the website.
  • If the results of algorithms in this challenge are to be used in scientific publications (e.g. journal publications, conference papers, technical reports, presentations at conferences and meetings) you must make an appropriate citation. Currently, this citation will refer to the this website (http://wmh.isi.uu.nl/), and later to the publication that will describe the results of the WMH Segmentation Challenge.
  • Evaluation of registration results uploaded to this website will be made publicly available on this website (Results section), and by submitting results, you grant us permission to publish our evaluation. Participating teams maintain full ownership and rights to their method.
  • Teams must notify the organizers of this challenge about any publication that is (partly) based on the results data published on this website, in order for us to maintain a list of publications associated with the challenge.

Click here to download this agreement as PDF. This agreement needs to be signed and mailed to wmh@isi.uu.nl, to activate your registration and get access to the data.

References

  1. Pantoni, L. Cerebral small vessel disease: from pathogenesis and clinical characteristics to therapeutic challenges. Lancet Neurology 9, 689–701 (2010).
  2. Wardlaw, J. M. et al. Neuroimaging standards for research into small vessel disease and its contribution to ageing and neurodegeneration. Lancet Neurology 12, 822–38 (2013).
  3. Caligiuri, M. E. et al. Automatic Detection of White Matter Hyperintensities in Healthy Aging and Pathology Using Magnetic Resonance Imaging: A Review. Neuroinformatics, 13(3), 261-276 (2015).