This CoFSM dataset contains the supplemental material of TIP3157450 (Multimodal remote sensing image datasets). The CoFSM dataset contains six types of modal images (multi temporal-optical, infrared-optical, depth-optical, map-optical, SAR-optical andnight-day). Each modal type includes 10 groups of images, and each set of images has corresponding ground truth points.