Deadline for the Exoplanet Imaging Data Challenge Phase 2:
31st of May 2024

EvalAI platform: The results of the Phase 2 (characterization) of the Exoplanet Imaging Data Challenge must be submitted on EvalAI.

The second phase of the Exoplanet Imaging Data Challenge is focused on the characterization of point source: it consists in performing two tasks, each one will have its own type of submission and metrics.

Task 1 (astrometry)

For each target star, the participant must provide one MEF .fits file containing:
(i) the estimated position from the star (in other words, the star is located at the coordinates [0,0]), in pixels;
(ii) [optional] the 1-sigma uncertainties on the estimated position;
(iii) [optional] the corresponding posterior distribution used to estimate the position and its uncertainties.
If the posterior distribution (iii) is not provided, we will assume that the posterior follows a normal distribution.

Note: One MEF .fits file contains the information of all the injections (2 to 3) of a given data set.

Task 1: filename: Please call all your files following this naming convention astrometry_instID.fits, with inst the instrument in lower case (e.g. sphere or gpi) and ID the dataset index (e.g. 1, 2, 3 or 4).

Task 2 (spectrophotometry)

For each target star, the participant must provide one MEF .fits file containing:
(i) the estimated contrast wrt the star;
(ii) [optional] the 1-sigma uncertainties on the estimated contrast;
(iii) [optional] the corresponding posterior distribution used to estimate the contrast and its uncertainties.
If the posterior distribution (iii) is not provided, we will assume that the posterior follows a normal distribution.

Note: One MEF .fits file contains the information of all the injections (2 to 3) of a given data set.

Task 2: filename: Please call all your files following this naming convention photometry_instID.fits, with inst the instrument in lower case (e.g. sphere or gpi) and ID the dataset index (e.g. 1, 2, 3 or 4).

Example of submission files

Task 1:

For example, for submitting your results of the 1st task (astrometry), you must gather the following eight files in .zip format (any name can be used for the .zip file - please do not use any subfolder!):

Each file must at least contain the 1st extension with the position values in x and y of each candidate, following the order given by the first_guess_astrometry_instID.fits file provided with the data sets (and same numpy array format - e.g. 3 rows (ny) of 2 columns (nx) for the xy coordinates of 3 planets).

Task 2:

For example, for submitting your results of the 2nd task (photometry), you must gather the following eight files in .zip format (any name can be used for the .zip file - please do not use any subfolder!):

Each file must at least contain the 1st dimension with the contrast estimates along each spectral channel, for each candidate, following the order given by the first_guess_astrometry_instID.fits file provided with the data sets (and same numpy array format - e.g. 3 rows (ny) of 39 columns (nx) for the spectra of 3 planets).

File format: Every file must be in MEF .fits format. In the EIDC2 Github repository, you will find a short tutorial, as a jupyter notebook, to put your results into a MEF .fits file.

Submission into a .zip file: All of the 8 MEF .fits files must be submitted within a single .zip file, with a flat structure (without subfolder structure, e.g. using the command on Mac > zip -r -X -j archive_name.zip folder_to_compress). One .zip file must be submitted for each task (one for astrometry and one for spectrophotometry).

Note: Each submission must correspond to the results of applying a single algorithm to all the datasets. You can therefore make several submissions, even if only one will show up on the competition webpage leaderboard, all of the submissions will be processed off-line.

IMPORTANT: If you really cannot detect the companion, despite the first guess provided, please put -1 as an entry (for both estimated value and its uncertainties).

Evaluation metrics

For the leaderboard display, the EIDC team has decided to use simple metrics: the metrics consist in mean normalized distances between the estimated value submitted by the participant and the ground truth used for injection. The distance is based on the L2 and L1 norms for astrometry and spectrophotometry, respectively.

Task 1 (astrometry)

For each injected companion, we compute the L2-norm distance (i.e. the Euclidian distance) between the estimated position value (xy_est) and the corresponding ground-truth value (xy_gt):

dist_astro = sqrt| | x_est - x_gt |^2 + | y_est - y_gt |^2 |

We then average all the distances dist_astro obtained for each of the 21 exoplanet injections.

Task 2 (spectrophometry)

For each injected companion, we compute the L1-norm distance between the estimated contrast value (cont_est) and the ground-truth value (cont_gt), for each wavelength (lambda), and normalize it with the ground truth contrast (to not penalize smaller true contrasts in the averaged score):

dist_photo = sum| | cont_est_lambda - cont_gt_lambda | / cont_gt_lambda |, (sum over all the wavelength)

We then average all the distances dist_photo obtained for each of the 21 exoplanet injections.



Further analysis of the results will be conducted off-line by our team. The metric used for the phase 2 of the Exoplanet Imaging Data Challenge are published in the proceeding accessible here.

The final comparison results will be published an SPIE Astronomical telescopes + instrumentation conference proceeding in summer 2024.

Potential error message

Other error message: Please contact us if you encounter any issue when submitting your results exoimg.datachallenge@gmail.com.

EvalAI team: