CodePromptEval is a dataset of 7072 prompts designed to evaluate five prompt techniques (few-shot, persona, chain-of-thought, function signature, list of packages) and their effect on the correctness, similarity, and quality of complete functions generated. Each data point in the dataset includes a function generation task, a combination of prompt techniques to be applied, the prompt in natural language that applied the prompt techniques, the ground truth of the functions (human-written functions based on CoderEval dataset by Yu et al.), the tests to evaluate the correctness of the generate

Dataset Files

You must be an IEEE Dataport Subscriber to access these files. Subscribe now or login.

[1] Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner, "CodePromptEval", IEEE Dataport, 2024. [Online]. Available: http://dx.doi.org/10.21227/sj94-ez71. Accessed: Mar. 18, 2025.
@data{sj94-ez71-24,
doi = {10.21227/sj94-ez71},
url = {http://dx.doi.org/10.21227/sj94-ez71},
author = {Ranim Khojah; Francisco Gomes de Oliveira Neto; Mazen Mohamad; Philipp Leitner },
publisher = {IEEE Dataport},
title = {CodePromptEval},
year = {2024} }
TY - DATA
T1 - CodePromptEval
AU - Ranim Khojah; Francisco Gomes de Oliveira Neto; Mazen Mohamad; Philipp Leitner
PY - 2024
PB - IEEE Dataport
UR - 10.21227/sj94-ez71
ER -
Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner. (2024). CodePromptEval. IEEE Dataport. http://dx.doi.org/10.21227/sj94-ez71
Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner, 2024. CodePromptEval. Available at: http://dx.doi.org/10.21227/sj94-ez71.
Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner. (2024). "CodePromptEval." Web.
1. Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner. CodePromptEval [Internet]. IEEE Dataport; 2024. Available from : http://dx.doi.org/10.21227/sj94-ez71
Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, Philipp Leitner. "CodePromptEval." doi: 10.21227/sj94-ez71