Data Fusion Contest 2022 (DFC2022)

Abstract

The Contest: Goals and Organization

The 2022 IEEE GRSS Data Fusion Contest, organized by the Image Analysis and Data Fusion Technical Committee, aims to promote research on semi-supervised learning. The overall objective is to build models that are able to leverage a large amount of unlabelled data while only requiring a small number of annotated training samples. The 2022 Data Fusion Contest will consist of two challenge tracks:

Track SLM:Semi-supervised Land Cover Mapping

Track BNI:Brave New Ideas

Semi-Supervised Learning

Data availability plays a central role in any machine learning setup, especially since the rise of deep learning. While usually input data is available in abundance, reference data to train and evaluate corresponding approaches is often scarce due to the high costs of obtaining it. While this is not limited to remote sensing, it is of particular importance in Earth observation applications. Semi-supervised learning is one approach to mitigate this challenge and leverage the large amount of available input data while only relying on a small annotated training set.

The semi-supervised learning challenge of the 2022 IEEE GRSS Data Fusion Contest, organized by the Image Analysis and Data Fusion Technical Committee (IADF TC) of the IEEE Geoscience and Remote Sensing Society (GRSS), Université Bretagne-Sud, ONERA, and ESA Φ-lab aims to promote research in automatic land cover classification from only partially annotated training data consisting of VHR RGB imagery.

To this aim, the DFC22 is based on MiniFrance [1], a dataset for semi-supervised semantic segmentation. As in real life Earth observation applications, MiniFrance comprises both labeled and unlabeled imagery for developing and training algorithms. It consists of a variety of classes at several locations with different appearances which allows to push the generalization capacities of the models.

The MiniFrance-DFC22 (MF-DFC22) dataset extends and modifies the MiniFrance dataset for training semi-supervised semantic segmentation models for land use/land cover mapping. The multimodal MF-DFC22 contains aerial images, elevation model, and land use/land cover maps corresponding to 19 conurbations and their surroundings from different regions in France. It includes urban and countryside scenes: residential areas, industrial and commercial zones but also fields, forests, sea-shore, and low mountains. It gathers data from three sources:

Open data VHR aerial images from the French National Institute of Geographical and Forest Information (IGN) BD ORTHO database. They are provided as 8-bit RGB tiles of size ~2,000px x ~2,000px at a resolution of 50cm/px, namely 1 km2 per tile. Images included in this dataset were acquired between 2012 and 2014.

Open data Digital Elevation Model (DEM) tiles from the IGN RGE ALTI database. DEM data gives a representation of the bare ground (bare earth) topographic surface of the Earth. They are provided as 32-bit float rasters of size ~1,000px x ~1,000px at a spatial resolution of 100cm/px, i.e. also 1 km2 per tile. The altitude is given in meters, with sub-metric precision in most locations. This database is regularly updated so images included in the dataset were acquired between 2019 and 2020.
Labeled class-reference from the UrbanAtlas 2012 database. 14 land-use classes are considered, corresponding to the second level of the semantic hierarchy defined by UrbanAtlas. Original data are openly available as vector images at the European Copernicus program website and were used to create raster maps that geographically match the VHR tiles from BD ORTHO. They are provided as integer rasters with index labels (0 to 15 - 8 and 9 being UrbanAtlas classes which do not appear in the regions considered) of size ~2,000px x ~2,000px at a resolution of 50cm/px, namely 1 km2 per tile.

[1] Castillo-Navarro, J., Le Saux, B., Boulch, A. and Lefèvre, S.. Semi-supervised semantic segmentation in Earth Observation: the MiniFrance suite, dataset analysis and multi-task network study. Mach Learn (2021).https://doi.org/10.1007/s10994-020-05943-y

The proposed dataset consists of very-high-resolution imagery, DEM information and semantic maps corresponding to 19 conurbations and their surroundings, spanning over different regions in France. Data are split in three partitions for training, validation and testing, with the particularity that the training partition obeys to semi-supervised learning conditions.

The training partition (labeled + unlabeled), contains a total of 1915 tiles. The largest area corresponds to Nantes/Saint-Nazaire with 433 tiles, while the smallest area is Lorient with only 120 tiles. Data is provided with georeference information.

The validation partition contains eight georeferenced areas with RGB images and DEM information. This partition contains 2066 tiles. Largest area is Lille/Arras/Lens/Douai/Henin including 407 tiles, smallest one is Cherbourg with 113 tiles.

The test partition consists of three areas without georeference information and contains RGB images and DEM information only. This partition includes 1035 tiles.

The semantic segmentation problem posed by MF-DFC22 considers 14 classes. These are defined by the second level of hierarchy of UrbanAtlas, with a total of 12 land use / land cover classes (urban fabric, industrial, mine & construction sites, pastures, forests, etc.), plus a no-data label (0) and clouds & shadows (15). MF-DFC22 raises a class-imbalance challenge, with less represented classes (mine, dump & construction sites, permanent crops, open spaces with little or no vegetation, wetlands and water) and majority classes (arable land and pastures). Note that 2 classes from the original UrbanAtlas nomenclature are not present in the dataset: 8 (mixed cultivation patterns) and 9 (orchards).

Challenge Tracks

Track SLM: Semi-supervised Land Cover Mapping

The DFC22 aims to promote innovation in automatic land cover classification, as well as to provide objective and fair comparisons among methods. The ranking in Track SLM is based on quantitative accuracy parameters computed with respect to undisclosed test samples. Participants will be given a limited time to submit their land cover maps after the competition starts.

Track BNI: Brave New Ideas

Due to the uniqueness of the DFC22, a second track is created which allows to explore new ideas more freely without being limited to land cover classification. In this track, all is possible and all is allowed, as long as it is novel and exciting. Even if it would be possible, results of different submissions will not be compared to each other - neither qualitatively nor quantitatively.

Acknowledgments

The IADF TC chairs would like to thank Université Bretagne-Sud, ONERA, and ESA Φ-lab for providing the data and the IEEE GRSS for continuously supporting the annual Data Fusion Contest through funding and resources.

Contest Terms and Conditions

The data are provided for the purpose of participation in the 2022 Data Fusion Contest and remain available for further research efforts provided that subsequent terms of use are respected. Participants acknowledge that they have read and agree to the following Contest Terms and Conditions:

In any scientific publication using the data, the data shall be identified as “grss_dfc_2022” and shall be referenced as follows: “[REF. NO.] 2022 IEEE GRSS Data Fusion Contest. Online: https://www.grss-ieee.org/technical-committees/image-analysis-and-data-fusion/”.
Any scientific publication using the data shall include a section “Acknowledgement”. This section shall include the following sentence: “The authors would like to thank the Université Bretagne-Sud, ONERA, and ESA Φ-lab for providing the data used in this study, and the IEEE GRSS Image Analysis and Data Fusion Technical Committee for organizing the Data Fusion Contest.
Any scientific publication using the data shall refer to the following papers:

[Castillo-Navarro et al., 2021] Castillo-Navarro, J., Le Saux, B., Boulch, A. and Lefèvre, S.. Semi-supervised semantic segmentation in Earth Observation: the MiniFrance suite, dataset analysis and multi-task network study. Mach Learn (2021).https://doi.org/10.1007/s10994-020-05943-y
Hänsch et al., 2019] Hänsch, R.; Persello, C.; Vivone, G.; Castillo Navarro, J.; Boulch, A.; Lefèvre, S.; Le Saux, B. : 2022 IEEE GRSS Data Fusion Contest: Semi-Supervised Learning [Technical Committees], IEEE Geoscience and Remote Sensing Magazine, March 2022

Instructions:

Participants will submit predictions of semantic maps via the CodaLab Competition page. Each pixel corresponds to one of the class IDs specified above (similar to the reference data of the training set).

Predicted images should be uploaded as follows:

The predictions for a particular tile should be encoded as a TIFF with the Byte (uint8) data type, match the dimensions of the corresponding BD ORTHO image and contain values between 1 and 14 (inclusive).
Each tile of predictions should be compressed in place (e.g., using the GDAL library), and all TIFF files should be submitted in a compressed zip archive. This is a particularly important point, as the competition website will not accept submissions of over 300MB and uncompressed submissions will likely surpass this limit.
Name the TIFF files <bdortho-tilename>_prediction.tif, where is the filename of the corresponding tile in the BD ORTHO database.
Predictions for each concurbations should be placed into corresponding folders with names identical to the folders of the validation/test data.
Please make sure that all TIFF files are readable by Pillow.

A baseline that shows how to use the DFC22 data to train models, make submissions, etc can be found here. See also the Documentation page for the dataset.

This baseline uses TorchGeo, PyTorch Lightning, and Segmentation Models PyTorch to train a U-Net with a ResNet-18 backbone and a loss function of Focal + Dice loss to perform semantic segmentation on the DFC2022 dataset. Masks for the holdout set are then predicted and zipped to be submitted. Note that the this supervised baseline is only trained on the labeled train set containing imagery from the Nice and Nantes Saint-Nazaire regions and results in a mIoU of 0.1278 on the heldout validation set. Participants utilizing semi-supervised learning techniques should seek improve upon this score.