How to measure UX with standardized questionnaires�

Martin Schrepp

Content

• Why is it important to measure UX?
• Standardized UX questionnaires and benchmarks
• Some basic principles you should know
• Summary and further infos

Why is it important to measure UX?

UX is a key factor for the market success of products:

• Products get more and more similar concerning functionality over time
• UX is often the factor that makes the difference
• Objective measures of UX quality are thus important

If you can not measure it, you can not improve it.

- Lord Kelvin

Why is it important to measure UX?

Cool and fancy!

I used all new technology features.

Developer

No customer has ever complained on the usability. Thus, there is no need to invest.

Nice and complex. I love it! I can show my expertise.

Product Owner

Unexperienced user

Strange that our users hate it? It has such cool functions.

IT expert of customer

UI might be difficult to understand.

UI Designer

I do not understand that. How the hell should I work with that?

Expert user

Without objective measures you need to fight opinions! That can be hard!

As long as nobody escalates we do not invest. Other things are more important.

DEV Manager

How did this work last time. I can not remember and it is so unintuitive!

Infrequent user

Why is it important to measure UX?

Our recent UX survey with 200 users showed that the Usability score is 49. Compared to a benchmark containing 450 products only 10% scored lower!

UX Researcher

It is hard to argue against numbers (which does not mean that people will not try it)!

☹

☹

☹

☹

☹

Product Team (DEV, PO, UID)

☹

I still don’t believe it.

We need to do something!

Which questions can be answered by measuring UX?

Is the UX of our product good enough to be successful in the market?

Is our product better or worse than the competition in terms of UX?

Has our new version improved in terms of UX compared to the previous one?

What specifically do we need to do to improve UX?

Standardized questionnaires

Each measurement method needs to fulfill certain quality criteria to provide interpretable results:

• Objectivity
• Reliability
• Validity

​

To fulfill these criteria the items (=questions) of the UX questionnaire must be constructed carefully. Often the construction process takes more than a year.

​

There a many such standardized questionnaires available that can be used.

​

For purely qualitative research questions you can define the questions directly. If a quantitative result (number!) is needed, this requires the usage of a carefully constructed UX questionnaire.

Example: Product Satisfaction (PSAT)

Simple satisfaction metric. Only a single item.

​

Overall, how satisfied are you with <produkt name>?

Strongly

dissatisfied

Dissatisfied

Neutral

Satisfied

Strongly

satisfied

PSAT Score: Number of satisfied or strongly satisfied participants / Number of all participants

​

PSAT is a simple marketing metric that is easy to interpret. It is NOT a pure UX metric!

Example: Net Promotor Score (NPS)

Reichheld (2003) ‘The one number you need to grow’, Harvard Business Review

Single item survey. Measures loyalty of customers.

How likely is it that you would recommend <product or company name> to a friend or colleague?

Not at all likely

Very likely

Detractor

Passive

Promoter

NPS = Percentage Promoters – Percentage Detractors

​

Managers love NPS because they believe: High NPS causes higher revenue in the future!

BUT: NPS requires huge data volumes for stable measurement!

Example: System Usability Scale (SUS)

Brooke, 1996: „A quick and dirty usability scale“

Strongly disagree

Strongly agree

I think that I would like to use this system frequently.

I found the system unnecessarily complex.

I thought that the system was easy to use.

I think that I would need the support of a technical person to be able to use this system.

I felt confident using the system.

I found the various functions in this system were well integrated.

I thought there was too much inconsistency in this system.

I would imagine that most people would learn to use this system very quickly.

I found the system very cumbersome to use.

I needed to learn a lot of things before I could get going with this system

10 items with agreement/disagreement.

Very common questionnaire. Very stable measurements.

​

No scales.

Provides one score representing overall UX in the range from 0 to 100.

​

Focus on pragmatic UX aspects, mainly efficiency and learnability.

​

Many translations available.

Example: User Experience Questionnaire (UEQ)

Laugwitz, Schrepp & Held, 2008

Semantic differential with 26 items. Very broadly applicable (from household appliances to business software).

​

There is a short version with 8 items.

​

6 Scales:

• Attractiveness: General assessment good/bad.
• Efficiency: You can complete your tasks without unnecessary effort.
• Perspicuity: Easy to understand and learn.
• Dependability: User can control the interaction well.
• Stimulation: It is interesting to work with.
• Novelty: Unusual and creatively designed.

​

Website: www.ueq-online.org

Many translations available.

Example: Visual Aesthetics for Web-Sites Inventory (VISAWI)

Mooshagen & Thielsch, 2010

Statements with agreement/disagreement, 18 items.

Measures visual beauty of a web page. But also works for other types of products.

​

The is a short version with 4 items.

​

4 Scales:

• Simplicity: Does the layout appear clear and structured?
• Diversity: Does the layout appear original and dynamic?
• Colorfulness: Does the color choice seem aesthetic?
• Craftmanship: Does the design seem well thought-out and professional?

​

Website: visawi.uid.com/

Benchmarks: How good is our product in terms of UX?

UEQ Benchmark: 6 Scales with range -3 to +3.

Benchmark based on > 400 product evaluations.

SUS Benchmark: Single scale, range 0 to 100.

Benchmark based on > 200 industrial product evaluations.

PSAT has a benchmark (updated once per year) per industry. For example, industry average 76% for computer software and 2021.

Benchmarks: How good is our product in terms of UX?

UMUX-Lite benchmark business software: Direct comparison to common products.

From: https://measuringu.com/business-software-ux2022/

Your product

Some basic principles you should know if you collect data using surveys and questionnaires?

Different users, different opinions

• Graph shows evaluation of 3 products with the System Usability Scale.
• 10 item questionnaire to measure usability, with a range of 0 (worst) to 100 (best).
• Obviously the assessment of the products differ, but we observe nearly full range of opinions for some of them.

Source: Rummel & Schrepp, 2018

We need to ask a larger target groups!

• Small confidence interval = High precision
• Large confidence interval = Low precision

​

• The more similar the responses, the narrower the intervals
• The more responses, the narrower the intervals

Confidence intervals

• Assume that you repeat your survey infinitely often with different random samples.
• Each repetition provides a different estimation of the scale values.
• The (unknown) true value would lie in 95% of the cases inside the (95%) confidence interval.

Don’t trust results from 5 feedbacks!

A small simulation

7,924 users, Mean: 3.21

Draw random samples of 50 users and calculate the mean

Which questionnaire should I use?

The choice of the questionnaire(s) depend on what is important for users.

• Which questions should we ask?
• UX has many facets! Different UX researchers or designers have often a quite different understanding of relevant UX qualities.
• The word cloud on the right shows items 1248 items from 40 UX questionnaires reduced to the main adjective in the item. Font size corresponds to frequency, color is just decoration.
• A single questionnaire can not cover the complete UX. Available questionnaires measure a special part of UX.

Source: Schrepp, 2020

Which UX aspects are important for a product?

The importance of UX aspects for the overall UX impression depend mainly on the product type!

Source: M. Schrepp, J. Kollmorgen, A.-L. Meiners, A. Hinderks, D. Winter, H. B. Santoso, J. Thomaschewski. On the Importance of UX Quality Aspects for Different Product Categories, International Journal of Interactive Multimedia and Artificial Intelligence, (2023), http://dx.doi.org/10.9781/ijimai.2023.03.001

Differences between product types

• Different UX aspects are important for different product types!
• But the importance ratings of UX aspects seems to be quite similar in different cultures.
• Graph on the right shows the importance ratings of 16 UX aspects for 15 product categories.
• Blue bars: German sample. Green bars: Indonesian sample.
• Big differences between product types, not much influenced by culture.

If we measure UX we should use a questionnaire that covers most of the important UX aspects!

My scores are low – What should I do?

• UX scores and benchmarks tell you how good or bad your product is compared to others.
• But they do not tell you what to do in order to improve!

Combine UX metrics with comments!

For example:

What did you like about the product?

What should be improved?

Use all available channels to get a better picture before you plan changes!

• Customer contacts
• Customer reviews
• Usability tests

​

Do not base important decisions about future developments on one single source of input!

How to collect feedback?

Button somewhere in the UI.

Opens typically a dialog with a survey when it is clicked.

Please give feedback

Overall, how satisfied are you with <product>?

Submit

Cancel

Please tell us about your experience

Strongly

Dissatisfied

Strongly

Satisfied

• Does not disturb users.
• Focus is on comments, can be clicked several times by the same user.
• Dialog should be used to keep context.
• Survey must be very short.

How to collect feedback?

Request for feedback is triggered automatically by time spent in the app or by a user action.

Please give us some feedback

OK Close

• Disturbing, but motivates to give feedback.
• Avoid to ask too often! We do not want to ruin UX!
• If you can not guarantee that the same user is not asked too often, do not use this method!
• Dialog should be used to keep context.
• Survey must be very short.

Please give feedback

Overall, how satisfied are you with <product>?

Submit

Cancel

Please tell us about your experience

Strongly

Dissatisfied

Strongly

Satisfied

Campaigns

Please take part in our survey

Link that is shown in the UI only for some time.

Longer survey shown in browser.

Alternative 1: Send link per eMail.

Alternative 2: Advertise link per social media channels.

• Allows longer surveys.
• Requires a higher completion time. People might drop out if it is too long! So concentrate on the really important questions!
• Always combine metrics (= standardzed questionnaires) and comment fields!
• Metrics tell you how good/bad your product is compared to others, comments help you to understand why this is the case!

Each method reaches a specific group of users! For each data collection method there always is a bias!

Be aware of the bias!

User base = All users of your product.

If you would be able to ask them all, the outcoming score will represent the truth!

Users that click on a feedback button

Users that answer to a eMail campaign

Users that answer to a system triggered intercept

Be aware of the bias!

Users give feedback if they have something to say! Often double-peaked, i.e. very positive and very negative opinions are overrepresented.

More people with a neutral opinion are reached! Typically, single peaked distribution.

Passive mechanism

Active mechanism

Key takeaways!

• Surveys are an efficient way to capture feedback from your users
• UX is about subjective opinions of users, surveys are a good method to ask
• Try to combine standardized questionnaires and comments!
• The scores from the questionnaires explain how bad/good your product is, the comments help to find the reason
• If you collect data using different methods be careful when you compare the results. Each method has an inherent bias
• This also applies when you compare to a benchmark
• There are many standardized UX questionnaires available.
• Think what are the most important UX qualities for your product and if these are captured by the intended questionnaire. A single questionnaire may not cover all UX aspects relevant for your product!

Some advertisement

Available at Amazon as eBook and printed book

​

ISBN-13 : 979-8736459766

Additional infos:

​

UEQ homepage: https:www.ueq-online.org

​

MeasuringU (Jeff Sauro, Jim Lewis): https://measuringu.com/

Questions?

Be aware of the bias!

• Does not disturb the user

​

• Relative constant flow of answers over time

​

• Users with very negative and very positive impressions are overrepresented

​

• Feedback reacts fast on negative (for example downtimes, bugs, bad designed new screens) and positive changes (for example, cool new functions, easier access). Thus, not very stable over time.

Passive mechanism

Active mechanism

If you compare data coming from different mechanisms, be careful! There can be differences due to different target groups of users you reach with different mechanisms.

• Disturbs the user, some users react quite negative

​

• A lot of feedback when you start this, but willingness to give feedback goes down fast

​

• Much more balanced feedback. Still a bias, but usually smaller than with a purely passive mechanism

​

If you measure UX over time, this does not matter. Choose one mechanism and do not change it! Since bias stays the same changes in the measurements are resulting from the product.

Are questionnaires a good tool to measure UX?

Advantages:

• We can directly ask users about their opinion
• We can ask for many different facets of UX (efficiency, ease of learning, aesthetics of design, fun of use, etc.)
• We can reach larger groups of users with limited effort
• Benchmarks can provide a comparison to other tools in the market

Disadvantages:

• Results indicate potential problems, but not how to solve them
• Often difficult to recruit enough users
• Difficult to decide if the target group is somehow representative for all users of a product