�USABILITY �TESTING�METHODS

PERTEMUAN 9 – EVALUASI UI/UX

TIM DOSEN EPAP 2018

METHODS

�Usability Testing (UT)

Overview

• Usability testing refers to evaluating a product or service by testing it with representative users.
• Usability testing involves observing users while they perform tasks with a hardware or software system.
• The goal is to identify any usability problems, collect qualitative and quantitative data and determine the participant's satisfaction with the product.

• https://www.usability.gov/how-to-and-tools/methods/usability-testing.html

User Testing vs Usability Testing

User Testing vs Usability Testing

When should you do:

• User Testing: Right after you have got the idea.
• Usability Testing: As soon as you have drawn those random hand drawn sketches on paper

What to expect from:

• User Testing: Do people need the solution I am working on?
• Usability Testing: Can people easily use the app ( for a given set of task) I am working on?

​

• http://blog.canvasflip.com/index.php/2016/06/12/user-testing-vs-usability-testing/

​

User Testing vs Usability Testing

What to ask during:

• User Testing:
• How do you currently do <problem you are solving>?
• Did you even think of a better way of doing this?
• Will you like to do this task <the way your solution works>?
• Will you pay money (for transaction apps) / share content for this solution?
• Usability Testing:
• Can you try doing this <new way of solution>?
• How would you like to login to this solution?
• Can you get <a small task in your solution> done in 10 seconds?

• http://blog.canvasflip.com/index.php/2016/06/12/user-testing-vs-usability-testing/

​

Advantages

• You can get feedback that reveals possible design flaws and other issues.
• You can get reliable measures of usability (see summative usability testing).
• Experienced test facilitators can elicit feedback from users to help understand why they had problems.
• Low and medium-fidelity prototypes are cost-effective to test.
• It is easy to have project manager and developers as observers.
• You can produce video clips from test sessions to show problems.

• http://www.usabilitybok.org/usability-testing

Disadvantages

• Not all problems will be found with small samples of users.
• You may not have access to users that match the user profile.
• Not all tasks may be "right" for all users.
• Lab testing takes users away from their natural work environment.
• Technical setup may be complex and require domain experts and additional time for setup and debugging.
• An inexperienced facilitator can influence the results by using too many hints, asking biased questions, or providing nonverbal cues about the tasks.

• http://www.usabilitybok.org/usability-testing

Procedure

Usability testing consists of five primary phases:

1. Planning
2. Pretest or pilot
3. Test sessions
4. Post-test or debrief
5. Analysis, interpretation, and presentation of the results.

• http://www.usabilitybok.org/usability-testing

Materials Needed

In general, the materials needed to run a usability test include:

• The system (paper sketch, model, display mockup, software, website)
• Physical or portable test lab (camera setup, observation room)
• Written test materials (informed consent, questionnaires, task scenarios, observation data sheets)
• Technical setup (servers, "live" or simulated test data)
• Connections for remote observers

• http://www.usabilitybok.org/usability-testing

Procedure (1)

Planning

• Scope:  Indicate what you are testing
• Purpose:  Identify the concerns, questions, and goals for this test. 
• Schedule & Location:  Indicate when and where you will do the test.
• Sessions:  Describe the sessions taken.
• Equipment:  Indicate the type of equipment you will be using in the test.
• Participants:  Indicate the number and types of participants to be tested you will be recruiting. >>More<<

• https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html

Procedure (1)

Planning

• Scenarios: Indicate the number and types of tasks included in testing.
• Metrics:  Subjective metrics, include the questions you are going to ask the participants. >>More<<
• Quantitative metrics: Indicate the quantitative data you will be measuring in your test. >>More<<
• Roles:  Include a list of the staff who will participate in the usability testing and what role each will play. 

​

• https://www.usability.gov/how-to-and-tools/methods/planning-usability-testing.html

Participant

• Usability Testing 🡪 representative users
• Number of participant:
• Norman Nielsen: 5 users (qual.) or 20 users (quant.) >>more<<
• Janet M. Six and Ritch Macefield: >>more<<�Problem-discovery study: 5-10 users�Comparative study: 10-12 users��

​

​

Num of Participant (Nielsen)

• https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/

Num of Participant (Nielsen)

• https://www.nngroup.com/articles/quantitative-studies-how-many-users/

Num of Participant (Six & Macefield)

• https://www.uxmatters.com/mt/archives/2016/01/how-to-determine-the-right-number-of-participants-for-usability-studies.php

Metrics

Effectiveness

• Accuracy
• Completeness

Efficiency (resources expended)

• Time

Satisfaction (comfort and acceptability)

• comfort and acceptability questionnaire

​

​

• Error rate
• Successful task completeness

​

• Time on task

Data Gathered

Procedure (1)

Planning

• Usability Testing Plan Template: https://www.usability.gov/how-to-and-tools/resources/templates/usability-test-plan-template.html

Procedure (2)

Pre-test or Pilot Testing

• Conduct pilot tests with internal users to debug instructions and tasks, verify that the hardware and software are working, and determine if there is adequate time for the session.
• Resolve any technical or logistical problems with the test plan and setup. Fix any problems with written test materials.
• Finalize the schedule and send it to all the observers.

Procedure (3)

Test Session

1. Welcome the user

Brief the user. Sign informed consent form(s), and the nondisclosure agreements (NDAs) if needed,

• User fill out pretest questionnaire

Which can be used to verify screening information and gather additional background information.

• Observing users while they perform tasks

Do not give hints or assistance unless necessary. Record time spent on each task, errors and problems user may face

​

• http://www.usabilitybok.org/usability-testing

Moderating Technique (1)

Some common moderating techniques include:

• Concurrent Think Aloud (CTA) is used to understand participants’ thoughts as they interact with a product by having them think aloud while they work. The goal is to encourage participants to keep a running stream of consciousness as they work.

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (1)

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (2)

Some common moderating techniques include:

• In Retrospective Think Aloud (RTA), the moderator asks participants to retrace their steps when the session is complete. Often participants watch a video replay of their actions.

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (2)

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (3)

Some common moderating techniques include:

• Concurrent Probing (CP) requires that as participants work on tasks—when they say something interesting or do something unique, the researcher asks follow-up questions.

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (3)

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (4)

Some common moderating techniques include:

• Retrospective Probing (RP) requires waiting until the session is complete and then asking questions about the participant’s thoughts and actions. Researchers often use RP in conjunction with other methods—as the participant makes comments or actions, the researcher takes notes and follows up with additional questions at the end of the session.

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Moderating Technique (4)

• https://www.usability.gov/how-to-and-tools/methods/running-usability-tests.html

Procedure (4)

Post-test or Debrief

1. User fill out post-test questionnaire �At the end of the session, ask the user to complete a post-test questionnaire.
2. Interview the user

to confirm they are representative of the intended user group, to gain general opinions, and to ask about specific problems encountered.

• Assess the results

Assess the results of the results of quantitative metrics, such as accuracy and completeness.

• http://www.usabilitybok.org/usability-testing

Kuesioner

• Teknik pengumpulan data dengan memberikan sejumlah pertanyaan yang sudah pasti diberikan dan dalam bentuk tercetak.
• Pertama yang harus dilakukan evaluator adalah menentukan tujuan dari kuesioner : informasi apa yang akan diambil ?
• Keunggulan metode ini yaitu cepat, dapat mencapai sasaran untuk kelompok pengguna yang besar dan dapat dianalisis dengan teliti, sedangkan kekurangannya kurang fleksibel. 

Jenis Kuesioner

• General, �pertanyaan untuk mengetahui latar belakang pengguna dan memasukannya ke dalam subyek populasi.
• Open-ended, �pertanyaan yang membutuhkan jawaban dan opini sendiri yang berguna untuk mengumpulkan informasi subyektif secara umum tetapi sulit untuk dianalisis.

Jenis Kuesioner

• Scalar, �user diminta untuk memutuskan pernyataan tertentu berdasarkan skala angka, biasanya berhubungan dengan setuju atau tidak setuju terhadap pernyataan yang diberikan.

Jenis Kuesioner

• Multi-choice, �responden diberikan pilihan respon secara eksplisit dan hanya boleh memilih satu saja atau sebanyak yang diperlukan. ��Misal : Software apa yang pernah Anda gunakan ?

Jenis Kuesioner

• Ranked, �mengurutkan item dalam daftar dan berguna untuk mengindikasikan preferensi user. ��Misal : rangkinglah kegunaan dari metode perintah

Popular Usability Questionnaire

1. System Usability Scale (SUS)
2. USE Questionnaire: Usefulness, Satisfaction, and Ease of use
3. Software Usability Measurement Inventory (SUMI)
4. Post-Study System Usability Questionnaire (PSSUQ)
5. Purdue Usability Testing Questionnaire

​

Procedure (5)

Report the results

1. Background Summary:  Include a brief summary including what you tested (website or web application), where and when the test was held, equipment information, what you did during the test (include all testing materials as an appendix), the testing team, and a brief description of the problems encountered as well as what worked well.

​

• https://www.usability.gov/how-to-and-tools/methods/reporting-usability-test-results.html

Procedure (5)

Report the results

2. Methodology:  Include the test methodology, explain how you conducted the test.
3. Test Results:  Analyze the data recorded.
4. Findings and Recommendations:  List your findings and recommendations using all your data (quantitative and qualitative) 

• https://www.usability.gov/how-to-and-tools/methods/reporting-usability-test-results.html

Report Template

Usability Test Report Template:

https://www.usability.gov/how-to-and-tools/resources/templates/report-template-usability-test.html

Further Reading

http://gen.lib.rus.ec/book/index.php?md5=061EBDB34205DD7DF79A26E9F806B5BA