Supplementary Material - Supporting Usability Inspection Education with a Tailored ChatGPT: An Empirical Study on Learning Heuristic Evaluation

Citation Author(s):
Lennon
Chaves
Romualdo
Costa
Márcia
Lima
Tayana
Conte
Submitted by:
Lennon Chaves
Last updated:
Fri, 02/21/2025 - 07:35
DOI:
10.21227/td3m-q457
License:
Creative Commons Attribution
102 Views
Categories:
Education and Learning Technologies
Keywords:
HCI Education, Usability Inspection, Heuristic Evaluation, Learning Process
0
0 ratings - Please login to submit your rating.
ACCESS DATASET CITE SHARE/EMBED

Abstract 

Experimental Setup:

We conducted the experiment over two separate days. On the first day, the participants inspected the object Canva (Web version), while on the second day, they inspected the object Skoob (Web version). Our study had 31 participants (undergraduate students enrolled in the HCI - Human-Computer Interaction Course).

Initially, participants were introduced to the experiment's objective and the procedure for the activity to be performed. On the first day, the class was divided into two groups G1 and G2, each assigned a different usability inspection method. G1 performed the usability inspection manually, while G2 used Solaria-GPT. On the second day, when evaluating a new system, the groups switched methods: G1 used Solaria-GPT, while G2 conducted the manual inspection. The experiment lasted 1 hour and 20 minutes per session. All participants received a usability inspection guide according to the assigned inspection method.

Instructions: 

Instrumentation Description:

The instruments were used to conduct a controlled experiment.

  • Informed Consent Form (ICF) – used to request participants' permission and present the study’s terms.
  • Participant Characterization Forms – applied to assess participants' experience levels in usability evaluation.
  • Inspection Guide – detailing the steps for inspecting both objects (Canva e Skoob) using manual and Solaria-GPT approaches. Each object had a dedicated guide according to the approach.
  • Spreadsheet for recording defects – used to document identified defects, including a brief problem description, location, and violated heuristic for manual inspection. For Solaria-GPT inspection, additional fields are included, such as violated heuristic returned by Solaria-GPT and level of agreement or disagreement with Solaria-GPT response.
  • Post-evaluation Questionnaire – applied to assess perceived knowledge after the experiment and collect open-ended feedback from participants.

Files Description:

Each instrument is represented by the following files:

  • Informed Consent Form (ICF)
    • Instruments_Informed_Consent_Form.docx
  • Participant Characterization Forms
    • Instruments_Participant_Characterization_Forms.docx
  • Inspection Guide:
    • Instrument_Inspection_Guide_Canva_Manual.docx
    • Instrument_Inspection_Guide_Canva_Solaria-GPT.docx
    • Instrument_Inspection_Guide_Skoob_Manual.docx
    • Instrument_Inspection_Guide_Skoob_Solaria-GPT.docx
  • Spreadsheet for recording defects
    • Instruments_Spreadsheet_Recording_Defects_SolariaGPT.xlsx
    • Instruments_Spreadsheet_Recording_Defects_Manual.xlsx
  • Post-evaluation Questionnaire 
    • Instrument_Post_Evaluation_Questionnaire.docx

Steps to Reproduce:

Preconditions (Before the experiment)

    • All the participants must assign and agree with the Informed Consent Form (ICF)
    • All the participants must answer the Participant Characterization Forms
    • Split the participants in two groups (G1 and G2), according to the participant characterization forms. Follow a balanced and randomized experimental design

First Day (Canva Object)

    1. Use the Inspection Guide, according to the group (G1 or G2) - G1 uses Instrument_Inspection_Guide_Canva_Manual.docx and G2 uses Instrument_Inspection_Guide_Canva_Solaria-GPT.docx
    2. Ensure all participants report the defects found in the Spreadsheet for recording defects.
      • G1 uses Instruments_Spreadsheet_Recording_Defects_Manual.xlsx
      • G2 uses Instruments_Spreadsheet_Recording_Defects_SolariaGPT.xlsx
    3. After the experiment, G2 must answer the Post-evaluation Questionnaire
      • Represented by Instrument_Post_Evaluation_Questionnaire.docx

Second Day (Skoob Object)

    1. Switch the groups' treatment: G1 will perform the Solaria-GPT assisted inspection, while G2 will perform the Manual inspection.
    2. Use the Inspection Guide, according to the group (G1 or G2) - G1 uses Instrument_Inspection_Guide_Skoob_Solaria-GPT.docx and G2 uses Instrument_Inspection_Guide_Skoob_Manual.docx
    3. Ensure all participants report the defects found in the Spreadsheet for recording defects.
      • G1 uses Instruments_Spreadsheet_Recording_Defects_SolariaGPT.xlsx
      • G2 uses Instruments_Spreadsheet_Recording_Defects_Manual.xlsx
    4. After the experiment, G1 must answer the Post-evaluation Questionnaire
      • Represented by Instrument_Post_Evaluation_Questionnaire.docx

Dataset Files

LOGIN TO ACCESS DATASET FILES

 

QUESTIONS?