Datasets
Standard Dataset
Examining ChatGPT 4's response to college students' psychological inquiries
- Citation Author(s):
- Submitted by:
- Yang Ni
- Last updated:
- Thu, 12/12/2024 - 09:57
- DOI:
- 10.21227/vb4q-5146
- License:
29 Views- Categories:
- Keywords:
Abstract
Data were generated from ChatGPT’s responses to 80 counseling questions that college students asked during a school counseling setting. All the responses generated during these simulated counseling sessions were then analyzed using three primary metrics—warmth, empathy, and acceptance—following APA guidelines. The analysis adopted several natural language processing methodologies for emotion detection and empathy measurement to quantify ChatGPT’s high efficacy in presenting the appropriate emotions and reactions for counseling.
Because this study aimed to measure the performance of ChatGPT and similar large language model (LLM)-based chatbots in the psychological counseling context, the data should come from real-life simulations. Therefore, we first identified a data source consisting of several authentic counseling questions. Then, we decided which LLM to use to obtain responses for our analyses. We ultimately selected ChatGPT-4 because it was the most universally recognized LLM at the time this study was conducted. Using ChatGPT’s online application, we collected three responses for each question [39]. After we collected all the data from ChatGPT’s responses, our analysis focused on evaluating how well they conveyed warmth, empathy, and acceptance. Another vital aspect is determining the randomness level of the ChatGPT responses to the same question. In the analysis, we utilized NLP techniques for our evaluation.
