Incomplete big datasets for parallel fractional hot-deck imputation

Citation Author(s):
Iowa State University
Jae Kwang
Iowa State University
In Ho
Iowa State University
Submitted by:
Yicheng Yang
Last updated:
Fri, 07/02/2021 - 16:24
Data Format:
0 ratings - Please login to submit your rating.


Parallel fractional hot-deck imputation (P-FHDI) is a general-purpose, assumption-free tool for handling item nonresponse in big incomplete data by combining the theory of FHDI and parallel computing. FHDI cures multivariate missing data by filling each missing unit with multiple observed values (thus, hot-deck) without resorting to distributional assumptions. P-FHDI can tackle big incomplete data with millions of instances (big-n) or 10, 000 variables (big-p). However, handling ultra incomplete data (i.e., concurrently big-n and big-p) with tremendous instances and high dimensionality has posed challenges to P-FHDI due to excessive memory requirement and execution time. We developed the ultra data-oriented P-FHDI (named as UP-FHDI) capable of curing ultra incomplete data. In addition to the parallel Jackknife method, UP-FHDI enables a computationally efficient ultra data-oriented variance estimation by using parallel linearization techniques. The object of this repository is to illustrate the use of UP-FHDI with different types of big incomplete data.


This repository includes three types of data: incomplete data with massive instances (big-n data), incomplete data with many variables (big-p data), incomplete data with tremendous instances and high dimensionality (ultra data). Overall, there exist six big-n datasets, four big-p datasets, and four ultra datasets. For instructions, see Readme files in the dataset folder for the step-by-step use of UP-FHDI with different types of incomplete datasets.