Dataset for Word Difficulty Prediction

Citation Author(s):
Jadavpur University
Jadavpur University
Sudip Kumar
Jadavpur University
Submitted by:
Avishek Garain
Last updated:
Sun, 10/04/2020 - 12:00
Data Format:
0 ratings - Please login to submit your rating.


Most text-simplification systems require an indicator of the complexity of the words. The prevalent approaches to word difficulty prediction are based on manual feature engineering. Using deep learning based models are largely left unexplored due to their comparatively poor performance. We have explored the use of one of such in predicting the difficulty of words. We have treated the problem as a binary classification problem. We have trained traditional machine learning models and evaluated their performance on the task. Removing dependency on frequency of previously acquired words for measuring difficulty was one of our primary aims. Then we analyzed a convolutional neural network based prediction model which operates at the character level and evaluate its efficiency compared to others.

This dataset contains 40481 data instances. The various column headers are as follows:

  • Word
  • Length
  • Freq_HAL
  • Log_Freq_HAL
  • I_Mean_RT
  • I_Zscore
  • I_SD
  • Obs
  • I_Mean_Accuracy


The other details of the dataset and the method to obtain the difficulty labels are present in the research publication whose link is attached.

For getting open-access to the publication visit

Please cite both the dataset and the conference paper if the dataset comes to any use.


The data is in CSV format. Please check the research paper for obtaining the difficulty label from the I_Z score.


For selecting Bibtex contents, double click on IEEE contents. Then use Ctrl+C to copy. It's a bug and we need to wait till its fixed. Till then this is how you can cite.

Submitted by Avishek Garain on Tue, 10/06/2020 - 03:42

The bug has been fixed.

Submitted by Avishek Garain on Tue, 11/24/2020 - 05:10