The Railway Surface Defect Detection (RSDDs) dataset was created to enhance the safety and reliability of railway transportation. This dataset comprises two subsets: Type-I RSDDs and Type-II RSDDs, which were collected from express and common/heavy haul railways, respectively. Type-I RSDDs consists of 67 images, each measuring 160×1000 pixels, while Type-II RSDDs includes 128 images, each measuring 55×1250 pixels. These images were captured under various lighting conditions to simulate real-world railway manufacturing and maintenance environments.