ChatGPT Study
- Citation Author(s):
-
Natasha Randall
- Submitted by:
- Natasha Randall
- Last updated:
- DOI:
- 10.21227/4rxb-zv06
698 views
- Categories:
- Keywords:
Abstract
This dataset comprises data created during research on AI-generated code, with a focus on software engineering use-cases. The purpose of the research was to investigate how AI should be integrated into university software engineering curricula. The seven core areas of the research include: An analysis of AI and student-written Java code, an analysis of AI and human-written Python code, the results of predictions made “by eye” to distinguish between AI and student-written code, experiments to determine the capabilities of ChatGPT, individual students' feedback on experiences with ChatGPT, a set of guidelines and detailed experiences with AI tools, and the results of a survey of students regarding the use of AI tools.
Instructions:
The seven core areas of this research are contained in each of the seven folders, and are described below.
Analysis of AI and Student-written Code: AI chatbots (ChatGPT-4, ChatGPT-3.5, Bing Chat and Bard) were used to generate code solutions to Java programming tasks (milestone assignments) taken from the 2021 presentation of the Software Engineering 2 Bachelor's course at TH Köln. Student solutions to the same assignments were stored anonymously. The differences between the AI and human written code solutions to "milestone 0" are analysed, and a simple classification model was trained to distinguish between AI and human written code. For the analysis, two approaches are compared: representing the code using manually defined features, and by OpenAI's text embedding vectors.
AI or Student-written Code Predictions: In order to determine how effectively AI-written code can be detected "by eye", a mixed set of AI- and human-written code solutions to the "Software Engineering 2" 2021 milestone 0 assignment was anonymised and given to two faculty members, who made predictions as to whether each solution was written by either a student or AI chatbot.
Analysis of AI and Human-written Python Code: ChatGPT was used to generate Python code solutions to the HumanEval problem set. The differences between the AI and human-written canonical code solutions are analysed, and a model is trained to distinguish between the two classes. The feature and embedding representation approaches are compared.
ChatGPT Capabilities Experiment: An evaluation of the capabilities of ChatGPT at completing software engineering university course assignments. By prompting ChatGPT-4 with only the original task description and provided document comments, purely AI-written code was generated as the solution to the "Software Engineering 2" 2023 assignment, comprising of the creation of a complex eCommerce system.
Student Experiences with ChatGPT: Feedback from two students on their experiences with using ChatGPT during their software engineering studies. Each use-case is evaluated, and the conversation with ChatGPT is provided.
AI Tools Experiences and Guidelines: Our experiences with generating code using different chatbots, an evaluation of their strengths and weaknesses, and a set of guidelines for using AI tools to write code.
Student Survey: The results from a survey distributed to university students. Questions relate to general AI tool use, AI use within education, and AI use specifically for coding and software engineering.
