EvalAI platform: To submit your result(s), the competition is held on EvalAI.

WARNING: The sphere2 data set has strong low wind effect, which might significantly affect the algorithms to process the images. This data set will therfore not be taken into account for the leaderboard ranking. If you encounter problems with this, please put 0 values as inputs and we will not take it into account for the final evaluation.

The second phase of the Exoplanet Imaging Data Challenge consists in performing two tasks, each one will have its own type of submission and metrics.

Deadline: 25th of June 2022

For each submission (one submission per algorithm), the participants must provide one zip file per task.
In each zip file, one MEF (multi-extension .fits file) file per data set (so a total of 8 MEF .fits files must be included in the zip file).

Task 1 (astrometry)

The participants must provide one MEF .fits file containing per target:
(i) the estimated position from the star (in other words, the star is located at the coordinates [0,0]), in pixels; (ii) [optional] the 1-sigma uncertainties on the estimated position; (iii) [optional] the corresponding posterior distribution used to estimate the position and its uncertainties.

If the posterior distribution (iii) is not provided, we will assume that the posterior follows a normal distribution.

One MEF .fits file contains the information of all the injections (2 to 3) of a given data set.

Task 1: filename: Please call all your files following this naming convention astrometry_instID.fits, with inst the instrument in lower case (e.g. sphere or gpi) and ID the dataset index (e.g. 1, 2, 3 or 4).

Task 2 (spectrophotometry)

The participants must provide one MEF .fits file containing per target:
(i) the estimated contrast wrt the star; (ii) [optional] the 1-sigma uncertainties on the estimated contrast; (iii) [optional] the corresponding posterior distribution used to estimate the contrast and its uncertainties.

If the posterior distribution (iii) is not provided, we will assume that the posterior follows a normal distribution.

One MEF .fits file contains the information of all the injections (2 to 3) of a given data set.

Task 2: filename: Please call all your files following this naming convention photometry_instID.fits, with inst the instrument in lower case (e.g. sphere or gpi) and ID the dataset index (e.g. 1, 2, 3 or 4).

Example of submission files

Task 1:

For example, for submitting your results of the 1st task (astrometry), you must gather the following eight files in .zip format (any name can be used for the .zip file - please do not use any subfolder!):

Each file must at least contain the 1st extension with the position values in x and y of each candidate, following the order given by the first_guess_astrometry_instID.fits file provided with the data sets (and same numpy array format - e.g. 3 rows (ny) of 2 columns (nx) for the xy coordinates of 3 planets).

Task 2:

For example, for submitting your results of the 2nd task (photometry), you must gather the following eight files in .zip format (any name can be used for the .zip file - please do not use any subfolder!):

Each file must at least contain the 1st dimension with the contrast estimates along each spectral channel, for each candidate, following the order given by the first_guess_astrometry_instID.fits file provided with the data sets (and same numpy array format - e.g. 3 rows (ny) of 39 columns (nx) for the spectra of 3 planets).

File format: Every file must be in MEF .fits format. In the EIDC2 Github repository, you will find a short tutorial, as a jupyter notebook, to put your results into a MEF .fits file.

Submission into a .zip file: All of the 8 MEF .fits files must be submitted within a single .zip file, with a flat structure (without subfolder structure, e.g. using the command on Mac > zip -r -X -j archive_name.zip folder_to_compress). One .zip file must be submitted for each task (one for astrometry and one for spectrophotometry).

Note: Each submission must correspond to the results of applying a single algorithm to all the datasets. You can therefore make several submissions, even if only one will show up on the competition webpage leaderboard, all of the submissions will be processed off-line.

IMPORTANT: If you really cannot detect the companion, despite the first guess provided, please put -1 as an entry (for both estimated value and its uncertainties).


Evaluation metric

For the leaderboard display, the EIDC team has decided to use a simple metric. Further analysis of the results will be conducted off-line by our team and the results will be published in an SPIE Astronomical telescopes + instrumentation conference proceeding in summer 2022.

For the leaderboard, the metric consist in using simply the distance (in the sense of the L1-norm) between the estimated value submitted by the participant and the ground-truth.

Task 1 (astrometry)

For each injected companion, we compute the L1-norm distance (following the Taxicab geometry) between the estimated position value (xy_est) and the corresponding ground-truth value (xy_gt):

dist_astro = | x_est - x_gt | + | y_est - y_gt |

We then average all the distances dist_astro obtained for each of the 21 exoplanet injections.

Task 2 (spectrophometry)

For each injected companion, we compute the L1-norm distance between the estimated contrast value (cont_est) and the ground-truth value (cont_gt), for each wavelength (lambda):

dist_photo = sum| cont_est_lambda - cont_gt_lambda |, (sum over all the wavelength)

We then average all the distances dist_photo obtained for each of the 21 exoplanet injections.


Potential error message

Other error message: Please contact us if you encounter any issue when submitting your results exoimg.datachallenge@gmail.com.


EvalAI team: