Knowledge Distillation Method Comparison

Citation Author(s):
Daniel
Hendriks
Karlsruhe Institute of Technology (KIT)
Submitted by:
Daniel Hendriks
Last updated:
Wed, 03/26/2025 - 04:29
DOI:
10.21227/x7nk-b929
Data Format:
*.csv
License:
Creative Commons Attribution
6 Views
Categories:
Artificial Intelligence
Machine Learning
Keywords:
explainability
0
0 ratings - Please login to submit your rating.
ACCESS DATASET CITE SHARE/EMBED

Abstract 

Artificial Intelligence (AI) has increasingly influenced modern society, recently in particular through significant advancements in Large Language Models (LLMs). However, high computational and storage demands of LLMs still limit their deployment in resource-constrained environments. Knowledge distillation addresses this challenge by training a smaller language model (student) from a larger one (teacher). Previous research has introduced several distillation methods for both generating training data and training the student model. Despite their relevance, the effects of state-of-the-art distillation methods on model performance and explainability have not been thoroughly investigated and compared. In our work, we enlarge the set of available methods by applying Critique-Revision Prompting to distillation and by synthesizing existing methods. For these methods, we provide a systematic comparison based on a popular commonsense question-answering dataset. While we measure performance via student model accuracy, we employ a humangrounded study to evaluate explainability. We contribute new distillation methods and their comparison in terms of both performance and explainability. 

Instructions: 

The attached dataset contains two components: a human-grounded study that investigates explainability, and experimental results that measure performance across different model distillation methods.

The file 'study.csv' contains the results from the human-grounded study:

  • The column 'CASE' is the identifier assigned to each participant,
  • the column 'rationale' contains an ID showing which explanation was evaluated,
  • the column 'Completeness' to 'Understandability' contain the ratings of participants regarding the explanation stated in the 'rationale' column,
  • the column 'rational_id' is irrelevant for the analysis and, hence, can be ignored,
  • the column 'gender' to 'employement' contain demographic information of participants.

The file 'experiment.csv' contains the results from the human-grounded study:

  • The column 'run' identifies the seed used to run the code for training the student models,
  • the column 'test accuracy' contains the accuracy of the student model in answering the multitple-choice questions from the CQA dataset's test split [1],
  • the column 'target_rationale' describes which explanations were used to train the student model, where 'few-shot-revision' refers to the explanations resulting from Critique-Revision Prompting and 'few-shot-positive-rationale' refers to explanations without improvements from Critique-Revision Prompting, 
  • the column 'model_size' refers to the number of parameters in the student model, and
  • the column 'training_mode' refers to the training method used to train the student model, where 'task-prefix' refers to multitask training, 'counterfactual-prefix' refers to counterfactual training, and 'both' refers to the training method combining both training methods.

[1] Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019). CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4149-4158). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1421

More from this Author

Knowledge Distillation Method Comparison

Dataset Files

LOGIN TO ACCESS DATASET FILES

 

QUESTIONS?