Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Citation Author(s):
Ranci
Ren
Universidad Autónoma de Madrid
John W.
Castro
Universidad de Atacama
Adrián
Santos
University of Oulu
Silvia T.
Acuña
Universidad Autónoma de Madrid
Submitted by:
John W. Castro
Last updated:
Mon, 03/01/2021 - 11:10
DOI:
10.21227/j3x2-1v53
Data Format:
License:
173 Views
Categories:
Keywords:
0
0 ratings - Please login to submit your rating.

Abstract 

Data Analysis

This folder consists of data obtained from the experiments and the materials we use for analysis. Here we provide the following documents:

  • The folder “Class diagrams”. The class diagrams that realized by participants in three experiments.
  • Code.R. The code we use to analyze the raw data of three experiments. With this code, we generate profile-plots, violin-plots, the summaries of descriptive statistics. 
  • Familiarity Result.xlsx. The data that we obtained from the familiarity questionnaire.
  • Raw Data.xlsx. The aggregation of the raw data from three experiments. 
  • Task 1 diagram.png and Task 2 diagram.png. The two ideal class diagrams correspond to Task 1 and Task 2. To analyze the response variable effectiveness and the quality of class diagrams, we compare these two ideal class diagrams with the class diagram realized by participants.
  • Which Variable Is Statistically Significant?. xlsx. To give a better understanding of the statistical significance of the three experiments to readers, we additionally provided the comparisons of statistical significance toward each metric of each response variable.

Experimental Design

This folder contains the original materials used in the experimental design process. Here we provide the following documents:

  • Experiment Procedure.pdf. The experiment procedure is identical among the three experiments. Here we provide the experiment procedure in detail.
  • Informed Consent.pdf. We require participants to sign the informed consent before the experiment, and the informed consent received by each participant is the same.
  • The folder “Questionnaires”. This folder contains the familiarity questionnaire we used before the execution of each experiment (i.e., Familiarity Questionnaire.pdf), the SUS questionnaire for Task 1 (i.e., SUS Questionnaire_Task1.pdf) and the SUS questionnaire for Task 2 (i.e., SUS Questionnaire_Task2.pdf).
  • The folder “Tasks”. This folder contains the contents of Task 1 (i.e., Task1.pdf) and Task 2 (i.e., Task2.pdf).
  • The folder “Tutorial Slides”. This folder contains the slides we use to give the tutorials of the chatbot SOCIO (i.e., SOCIO Tutorial.pptx) and the web-based online tool Creately (i.e., CREATELY Tutorial.pptx)

 

Instructions: 

EXPERIMENT PROCEDURE

PRE-TEST

1. Each participant creates an account in Telegram

2. Divide the participants into two groups at random

3. Divide the participants of each group into teams of 3 randomly

4. Create two Telegram groups (one with the chatbot SOCIO and the other without the chatbot SOCIO) for each team

 

GROUP 1: Chatbot SOCIO - Web-based online tool Creately

TIME: 10 minutes of tutorial + 30 minutes of experiment + 10 minutes break

DURING-TEST

1. Place the members of each team separately in the classroom

2. The participants fill the familiarity questionnaire

3. Give the tutorial of the chatbot SOCIO

4. Explain task 1 in general and give the participants the description of the task on paper

5. The participants execute task 1 with the chatbot SOCIO

6. The participants fill the first SUS questionnaire

7. 10-minute break

8. Give the tutorial of Creately

9. Explain task 2 in general and give the participants the description of the task on paper

10. The participants execute task 2 with Creately

11. The participants fill the second SUS questionnaire

12. Ask the participants to send the class diagram to the designated email address

 

GROUP 2: Web-based online tool Creately - Chatbot SOCIO

TIME: 10 minutes of tutorial + 30 minutes of experiment + 10 minutes break

DURING-TEST

1. Place the members of each team separately in the classroom

2. The participants fill the familiarity questionnaire

3. Give the tutorial of Creately

4. Explain task 1 in general and give the participants the description of the task on paper

5. The participants execute task 1 with Creately

6. The participants fill the first SUS questionnaire

7. 10-minute break

8. Give the tutorial of the chatbot SOCIO

9. Explain task 2 in general and give them the description of the task on paper

10. The participants execute task 2 with the chatbot SOCIO

11. The participants fill the second SUS questionnaire

The experiment is repeated three times, correspond to the family of three experiments. The process and variables for each experiment are the same. Each participant can only participate in one experiment.