Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

CONTENTS OF THIS FILE

•  Data Analysis
•  Experimental Design

Data Analysis

This folder consists of data obtained from the experiments and the materials we use for analysis. Here we provide the following documents:

01-The folder "Class diagrams"

      This folder contains class diagrams realized by participants in three experiments.

02-Code.R 

      Here we provide the code we use to analyze the raw data of three experiments. With this code, we generate profile-plots, violin-plots, the summaries of descriptive statistics. 

03-Power calculation (appendix A).R

      Here we provide the code needed for a-priori power analysis.

04-Cummulative meta-analysis (appendix B).R

      Here we provide the code needed for the cumulative meta-analysis analysis.

05-Familiarity Result.xlsx

      Here we provide the data we obtained from the familiarity questionnaire.

06-Raw Data All.xlsx

      Here we provide the aggregation of the raw data of the three experiments. 

07-Task 1_Ideal Class diagram.png and 08-Task 2_Ideal Class diagram.png

      Here we provide two ideal class diagrams corresponding to task 1 and task 2. To analyze the response variable effectiveness and the quality of class diagrams, we compare these two ideal class diagrams with each of the class diagrams realized by participants.

09-Which Variable is Statistically Significant?.xlsx

      To give a better understanding of the statistical significance of the three experiments to readers, we additionally provide a comparison of the statistical significance for each experiment.

10-How to calculate the quality.pdf

      Here we provide two examples of SOCIO and Creately to illustrate how we compare the ideal class diagram and the class diagram obtained in the experiment.

Experimental Design

This folder contains the original materials used in the experimental design process. Here we provide the following documents:

01-Experiment Procedure.pdf

      Here we provide the experiment procedure in detail. The experiment procedure is identical among the three experiments. 

02-Informed Consent.pdf

      Here we provide the informed consent in detail. We require participants to sign the informed consent before the experiment, stating that they granted us permission to record their data via Telegram. We ensured that the participation of all subjects in this research was completely anonymous, and no information they shared could be electronically traced to them

03-The folder "Questionnaires"

      This folder contains the familiarity questionnaire we used before the execution of each experiment (i.e., Familiarity Questionnaire.pdf), the SUS questionnaire for Task 1 (i.e., SUS Questionnaire_Task1.pdf) and the SUS questionnaire for Task 2 (i.e., SUS Questionnaire_Task2.pdf).

04-The folder “Tasks”

      This folder contains the specific content of Task 1 (i.e., Task1.pdf) and Task 2 (i.e., Task2.pdf).

05-The folder “Tutorial Slides”

      This folder contains the slides we use to give the tutorials of the chatbot SOCIO (i.e., SOCIO_Tutorial.pptx) and the web-based online tool Creately (i.e., Creately_Tutorial.pptx).

06-SMS procedure.xlsx

      Here we provide the Systematic Mapping Study procedure with peer-review opinions.

07-SMS selection process.xlsx

      Here we provide the selection process of primary studies in Systematic Mapping Study and the number of papers encountered in each process.

08-SMS-HCI conf&journal.docx

      Here we provide the list of HCI conferences and journals where we conducted SMS search.

09-SMS procedure.docx

      Here we provide the SMS procedure, including objectives, search string selection, databases and search protocol, and search process.