Phases¶

The LUNA25: AI Study took place in two phases, followed by a Post Challenge Phase. Please find descriptions of the Phases below:

• Open Development Phase (Duration: 4 months):
  Anyone can participate in this phase of the challenge. Interested teams can create an account on grand-challenge.org, and register for the LUNA25 challenge at luna25.grand-challenge.org. Afterwards, they will be provided access to download the public training dataset, and in turn, they can start developing and training AI algorithms using their private or public compute resources. Participating teams can also use additional data to train their algorithms, but such data must be publicly available under a permissive open-source license three months prior to the submission deadline, and its source must be clearly stated. Teams can upload and submit their trained algorithms (in Docker containers) for evaluation a maximum of 15 times throughout the challenge. During evaluation, the algorithms are executed on the grand-challenge.org platform, their performance is evaluated on the hidden tuning cohort, and team rankings are updated accordingly on a live, public leaderboard. Facilitating validation in such a manner ensures that any image used for evaluation remains truly unseen, and that AI predictions cannot be tampered with. This allows for bias-free performance estimation.

• Closed Testing Phase (Duration: 1 month):
  After the Development Phase is closed, each registered team can choose to submit a single AI algorithm (presumable their top-performing algorithm) for evaluation on the hidden testing cohort. Based on their performance on this cohort, all new rankings will be drawn and the top 5 algorithms of the LUNA25 challenge will be determined and announced. To qualify as one the top teams, participants must also submit a short paper on their methodology (2-3 pages) and we recommend (but do not enforce) to provide a link to a public Github repository with the source code of the algorithm to ensure fairness, traceability and reproducibility of all proposed solutions.

• Post Challenge Phase: Following the successful edition of the LUNA25 Challenge, we are pleased to offer participants the opportunity to submit their algorithms for evaluation in the Post-Challenge Phase. Submissions will be evaluated on the Hidden Testing Cohort that was previously used during the Closed Testing Phase, allowing results to be benchmarked against the official LUNA25 Closed Testing Phase Leaderboard. To ensure that the benchmark can remain available in a sustainable, maintainable, and fair manner, we ask interested teams to provide a short document describing their methodology. Participation in the Post-Challenge Phase will be subject to a separate legal agreement and a contribution to support the computational resources, data storage, platform maintenance, and continued use of the underlying datasets required for evaluation. If you are interested in submitting to the Post-Challenge Phase, please contact Colin Jacobs for further details.