Using the SOCIO Chatbot for UML Modelling: A Second Family of Experiments

Citation Author(s):
Ranci
Ren
Universidad Autónoma de Madrid
John W.
Castro
Universidad de Atacama
Oscar
Dieste
Universidad Politécnica de Madrid
Silvia T.
Acuña
Universidad Autónoma de Madrid
Submitted by:
John W. Castro
Last updated:
Mon, 08/29/2022 - 13:07
DOI:
10.21227/qzdr-nj48
Data Format:
License:
96 Views
Categories:
Keywords:
0
0 ratings - Please login to submit your rating.

Abstract 

CONTENTS OF THIS FILE

 * Data Analysis

 * Experimental Design

 

# Data Analysis

This folder consists of data obtained from the experiments and the materials we use for analysis. Here we provide the following documents:

01-The folder "Class diagrams"

      This folder contains class diagrams realized by participants in three experiments.

02-Code.R 

      Here we provide the code we use to analyze the raw data of three experiments. With this code, we generate profile-plots, violin-plots, the summaries of descriptive statistics. 

03-Familiarity Result.xlsx

      Here we provide the data we obtained from the familiarity questionnaire.

04-Raw Data.xlsx

      Here we provide the aggregation of the raw data of the three experiments.

05-Task 1_Ideal Class diagram.png and 06-Task 2_Ideal Class diagram.png

      Here we provide two ideal class diagrams corresponding to task 1 and task 2. To analyze the response variable effectiveness and the quality of class diagrams, we compare these two ideal class diagrams with each of the class diagrams realized by participants.

07-How to calculate the quality.pdf

      Here we provide two examples of SOCIO and Creately to illustrate how we compare the ideal class diagram and the class diagram obtained in the experiment.

08-The folder "Thematic Analysis"

      This folder contains the code and data that we use to generate the thematic results graphs.

 

# Experimental Design

This folder contains the original materials used in the experimental design process. Here we provide the following documents:

01-Experiment Procedure.pdf

      Here we provide the experiment procedure in detail. The experiment procedure is identical among the three experiments. 

02-Informed Consent.pdf

      Here we provide the informed consent in detail. We require participants to sign the informed consent before the experiment, stating that they granted us permission to record their data via Telegram. We ensured that the participation of all subjects in this research was completely anonymous, and no information they shared could be electronically traced to them

03-The folder "Questionnaires"

      This folder contains the familiarity questionnaire we used before the execution of each experiment (i.e., Familiarity Questionnaire.pdf), the SUS questionnaire for Task 1 (i.e., SUS Questionnaire_Task1.pdf) and the SUS questionnaire for Task 2 (i.e., SUS Questionnaire_Task2.pdf).

04-The folder "Tasks"

      This folder contains the specific content of Task 1 (i.e., Task1.pdf) and Task 2 (i.e., Task2.pdf).

05-The folder "Tutorial Slides"

      This folder contains the slides we use to give the tutorials of the chatbot SOCIO (i.e., Tutorial_SOCIO V1.pdf) and the web-based online tool Creately (i.e., Tutorial_Creately.pdf).

Instructions: 

EXPERIMENT PROCEDURE

Pre-experiment:

1. Each participant creates an account in Telegram

2. Divide the participants into two groups at random

3. Divide the participants of each group into teams of 3 at random

4. Create two Telegram groups (one with chatbot SOCIO V1 and the other without chatbot SOCIO V1) for each team.

 

During the experiment:

GROUP 1: Chatbot SOCIO V1 - Web-based online tool Creately

Time: 10 minutes of tutorial + 30 minutes of experiment + 10 minutes break + 10 minutes of tutorial + 30 minutes of experiment

Process:

1. Place the members of each team separately in the classroom

2. The participants fill the informed consent

3. The participants fill the familiarity questionnaire

4. Give the tutorial of chatbot SOCIO V1

5. Explain task 1 in general and give the participants the description of the task on paper

6. The participants execute task 1 with chatbot SOCIO V1

7. The participants fill the first SUS questionnaire

8. 10-minute break

9. Give the tutorial of Creately

10. Explain task 2 in general and give the participants the description of the task on paper

11. The participants execute task 2 with Creately

12. The participants fill the second SUS questionnaire

13. Ask the participants to send the class diagram to the designated email address

GROUP 2: Web-based online tool Creately - Chatbot SOCIO V1

Time: 10 minutes of tutorial + 30 minutes of experiment + 10 minutes break + 10 minutes of tutorial + 30 minutes of experiment

Process:

1. Place the members of each team separately in the classroom

2. The participants fill the informed consent.

3. The participants fill the familiarity questionnaire.

4. Give the tutorial of Creately

5. Explain task 1 in general and give the participants the description of the task on paper

6. The participants execute task 1 with Creately

7. The participants fill the first SUS questionnaire

8. 10-minute break

9. Give the tutorial of chatbot SOCIO V1

10. Explain task 2 in general and give them the description of the task on paper

11. The participants execute task 2 with chatbot SOCIO V1

12. The participants fill the second SUS questionnaire.

The experiment is repeated three times, correspond to the family of three experiments. The process and variables for each experiment are the same. Each participant can only participate in one experiment.

 

Funding Agency: 
Spanish Ministry of Science, Innovation and Universities, Madrid Region R&D program
Grant Number: 
PGC2018-097265-B-I00, RTI2018-095255-B-I00, P2018/TCS-4314