Dataset for Word Difficulty Prediction

Most text-simplification systems require an indicator of the complexity of the words. The prevalent approaches to word difficulty prediction are based on manual feature engineering. Using deep learning based models are largely left unexplored due to their comparatively poor performance. We have explored the use of one of such in predicting the difficulty of words. We have treated the problem as a binary classification problem. We have trained traditional machine learning models and evaluated their performance on the task. Removing dependency on frequency of previously acquired words for measuring difficulty was one of our primary aims. Then we analyzed a convolutional neural network based prediction model which operates at the character level and evaluate its efficiency compared to others.

This dataset contains 40481 data instances. The various column headers are as follows:

• Word
• Length
• Freq_HAL
• Log_Freq_HAL
• I_Mean_RT
• I_Zscore
• I_SD
• Obs
• I_Mean_Accuracy

The other details of the dataset and the method to obtain the difficulty labels are present in the research publication whose link is attached.

For getting open-access to the publication visit https://garain.codes

Please cite both the dataset and the conference paper if the dataset comes to any use.