Datasets for Growing-MoE Evaluations

This dataset is used to evaluate the effectiveness of the Growing-MoE learning framework. The dataset contains tasks across computer vision (CV) and natural language processing (NLP). The dataset includes CV tasks such as CIFAR, ImageNet, Cars, and Flowers, as well as NLP tasks including English Wikipedia and GLUE benchmark.

Our learning framework aims to accelerate training of large Mixture-of-Experts models, which employs a progressive way to learn the model from local to global. We verify its performance with several popular networks, such as DeiT, Swin, GPT-2 on the tasks across CV and NLP. We also conduct transfer learning to prove the versatility and flexibility of our framework.