SPLAS

This is a lightweight and versatile robustness benchmark built upon the training set of ImageNet-1K. It contains an overall of 50,000 images, divided in 5 components, evenly distributed over 1,000 classes. It assesses the performance of a classification model in five aspects: accuracy on intrinsically difficult images (SuperHard, SH), images with partial information (PartialInfo, PI), robustness against low resolution (LowResolution, LR), adversarial attacks (AdversarialAttack, AA), and speckle noise (SpeckleNoise, SN).

Compared to previous robustness benchmark, SPLAS is outstanding regarding its size, versatility, and reproducibility. More importantly, the amount of human intervention on the construction is minimal. Particularly, there is no intervention of human’s decision on image inclusion, thus the whole benchmark is constructed based on the understanding of deep learning models.