CodePromptEval
- Citation Author(s):
-
Ranim Khojah
(Chalmers University of Technology and University of Gothenburg)
Francisco Gomes de Oliveira Neto
(Chalmers University of Technology and University of Gothenburg)
Mazen Mohamad
(RISE Reseach Institutes of Sweden | Chalmers University of Technology and University of Gothenburg)
Philipp Leitner
(Chalmers University of Technology and University of Gothenburg)
- Submitted by:
- Ranim Khojah
- Last updated:
- DOI:
- 10.21227/sj94-ez71
- Data Format:
- Links:
33 views
- Categories:
- Keywords:
Abstract
CodePromptEval is a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated. Each data point in the dataset includes a function generation task, a combination of prompt techniques to be applied, the prompt in natural language that applied the prompt techniques, the ground truth of the functions (human-written functions based on CoderEval dataset by Yu et al.), the tests to evaluate the correctness of the generated functions. The prompts in the dataset are carefully designed to apply the five prompt techniques.
Instructions:
The dataset is in csv format. It is recommended to upload the dataset using Pandas dataframes.
