1 of 25

Card Sorting or Tree Testing?�Best Methods for Testing Information Architecture

Leah Catania

Content Strategist, Technical Communications

TraceLink, Inc.

PRESENTED BY

2 of 25

Introduction

Define and test the product and documentation information architecture (IA)
Research other company’s IA choices
Put together usability research and testing with low resources or access to customers

Content Strategist at TraceLink, Inc.
7+ years of technical writing experience
8+ years studying and performing usability testing
7+ years of industry experience in pharmaceutical supply chain (heavily regulated)

3 of 25

Agenda

Basics

Usability testing
Information architecture (IA)

Card sorting

What
When
How
Analysis
Example

Tree testing

What
When
How
Analysis
Example

Tools

Paper
Flare
Usability tools (e.g. Optimal Workshop)

Today, I’ll be touching on the basics of usability testing and information architecture, diving into the what, when, and how of card sorting and tree testing, and we’ll look at an example of each. And then I’ll go into some of the pros and cons for the different methods and tools you can use for each of these types of usability tests.

Summary:

You don’t know much about building information architecture, but you do know that you should probably test what you have. But testing is time consuming and expensive. Or is it? Card sorting and tree testing are two fantastic methods of forming and validating information architecture decisions, and they both require relatively little resources to set up and conduct, even if you don’t have access to user testing software like Optimal. Learn how these testing methods can be useful to building IA for your products and documentation, as well as which method you should use when. I’ll also walk through the basics for how to conduct them (including using Flare if you don’t have other options), software that can help you analyze the results, and how to make recommendations based on the results.

4 of 25

Basics

5 of 25

What is usability testing?

Measures how effectively and efficiently users can actually use your product or documentation
Provides feedback before the product has been developed, which saves money in the long run
Participants must represent real users and perform real tasks
Provides both qualitative and quantitative data
Can be performed by any member of the product team (not just UX)

How many of you have actually participated in usability testing at your jobs before?

So, what actually is usability testing?

In general, it’s how we evaluate and measure how effectively and efficiently users actually use your products or documentation.
Ideally, testing can provide this feedback before the product is developed or the documentation is written, which will save the company money in the long run, and is usually the best argument for introducing usability testing to your software development lifecycle.
In order to get useful results, you need to make sure that the participants of any usability test represent real users and that you’re asking them to perform real tasks. Which, of course, means you have to know who your users are and exactly what they do in your product. There’s no point in jumping into usability testing if you haven’t done the groundwork user research first.
It’s important to remember that usability testing provides both qualitative and quantitative data. Stakeholders are often focused on the quantitative results, but without proper qualitative results, it can be hard to interpret the numbers.
Now, some of you might be wondering why we’re even talking about usability testing at a tech writer’s conference, but usability testing really can be conducted by any member of the product team who has an interest in doing it and is willing to put in the work to do it in an unbiased way.

Writers are often the product’s first users, and are involved enough to understand the product but removed enough from the design process to remain objective, which is important. We can also help write a clear, understandable test plan, script, and tasks.
And of course, our documentation and content strategy decisions should be based on real data as well, so even if you aren’t able to help incorporate usability testing into your SDLC, you probably have more control over the documentation development life cycle and be able to incorporate it there.

6 of 25

What is information architecture?

Grouping or arranging different pieces of information in a way that makes sense
Labelling those groups and pieces of information in an intuitive and understandable way
Creating a path for sensemaking and wayfinding
All members of the product team contribute to IA

Now for the other half of the basics: information architecture.

IA is the practice of grouping or arranging different pieces of information in a way that makes sense.
It also includes labelling those groups and pieces of information in an intuitive and understandable way.
All of this creates a path for users to make sense of the information in front of them and find their way around your product or documentation.
The most noticeable place that involves IA in your products is the navigation, but IA goes much deeper than that, all the way down to the way your order and organize fields in a form.
All members of the product team contribute to IA.

UX designers can be more visually-focused and less knowledgeable on IA.
Writers are constantly organizing information in our documentation, and that skillset also lends itself to influencing the IA of the actual product.

If the team doesn’t actively think about the IA during the initial research and design phases of product development, then you’ll likely end up with confused users who aren’t sure where they should go.

7 of 25

How do we usability test IA?

Traditional usability tests will also give you information about your IA
To test the IA before designing the experience, use:

Card sorting
Tree testing

These methods can also help evaluate the IA of an existing product
Ensures the product is organized around what makes sense to users instead of what makes sense internally to the company

So, how do we usability test IA?

Traditional usability tests, using wireframes or hi-fidelity designs definitely give you information about IA, such as whether users can find where they need to go, whether they selected the correct menu option, etc.
But we can very effectively test the IA before the team begins designing the experience or writing the documentation by using card sorting, tree testing, or both.
These methods can also very effectively evaluate the IA of an existing product or documentation set.
By performing these types of usability testing before you begin designing, you can make sure that the product or documentation is actually organized around what makes sense to the users instead of what makes sense internally to the company.

This sounds like common sense right? But all too often, companies don’t even realize the assumptions they hold about the way to organize information and tasks. Think about every corporate website you’ve seen where it’s organized around the departments inside the company, but you can’t find the information you need because you don’t know which department is in charge of it. This happens to me every time I visit my town’s website. I can never find where the town dump information is, or the dog license information, because it’s listed under a specific department, and I have no idea which one. This happens all the time, and using card sorting and tree testing can be a very valuable way of demonstrating to stakeholders than their mental model for organizing information isn’t the same as their users.

8 of 25

Card sorting

9 of 25

What is card sorting?

Participants sort through various items (e.g. objects, actions) and place them into groups that make sense to them
Provides insight for how your users think about grouping the information
Open card sorts ask participants to form and label the groups themselves
Closed card sorts provide groups for the participants to sort the cards into

In a card sort, you have a series of items – like objects or actions – that you’re trying to figure out how to organize. You give a set of cards to the participant with one item on each and ask them to sort them into groups, whatever way makes the most sense to them.
Knowing how participants sort the items, and which items they sort together, gives you insight into how your users think about grouping the information in your product or documentation.
There are a couple different types of card sorts:

Open card sorts ask participants to form and label the groups themselves
Closed card sorts provide groups for the participants to sort the cards into
Hybrid card sorts give participants groups but also allow them to create their own groups.
Unless the team has enough evidence to be very confident about the top-level organization, open card sorts are usually best, because they provide more unbiased results form users. However, they can make the analysis more difficult, since each person thinks slightly differently and participants usually won’t create the exact same groups.

10 of 25

When is card sorting most useful?

Building a brand new product or documentation set
The product already exists, the team knows the IA has problems but isn’t sure which direction to go to improve the IA
Compare how different people think about the same set of information

11 of 25

How to run a card sort

Choose the items to sort:

30-60 items
Use diverse terminology
Make sure they’re all at the same hierarchical level

Recruit 15-20 participants, representative of users
Moderated or unmoderated

Best practices:

Users can change groups around as much as they like
Users can create a “not sure” group of leftovers
Ask users to label their groups after making them
Ask users qualitative questions at the end

To run a card sort:

First you need to select the items to sort.

Keep them under 60 ideally. Too many more than that and it will be too hard for the participants to sort them all. You really only want this to take an hour at most.
Make sure the items use diverse terminology. The terms don’t need to, and in fact, should not, match the UI.

Use synonyms as much as possible so that people don’t just automatically match the words they see in front of them. You want them to actually think about the concepts behind the words instead.
For example, if you’ve got a bunch of items with configuration actions on them, make sure you mix up the verb you use: “Configure,” “Set up”, “Edit”, “Update” etc.

Make sure that the items are all at the same hierarchical level, so they can actually be easily grouped.

For example, if you’ve got a bunch of produce as your items, for a farmer’s stand website, make sure they’re all things like “strawberries, blueberries, zucchini,” etc. and there aren’t any “category-level” items like “fruits.” Participants will struggle with trying to group items that are at different hierarchical levels.

Then you need to find your participants, who need to be representative of your actual users, if you can’t get actual users as participants.

Card sorting is more quantitative than traditional usability tests, so you do need a higher number of participants to draw accurate conclusions from the results. 15-20 is usually enough to see enough overlap in the groupings.

Card sorting can be done unmoderated, but keep a few things in mind:

If you’re struggling to find enough participants, moderated sessions can give you more qualitative feedback to help analyze the results because the moderator can ask more questions in-the-moment about the grouping choices participants are making and record their thoughts out loud as they go through the session.
Moderators can also ask participants to make more reasonably sized groups at the end: if they have a couple large groups, you can ask how they would split them if they had to or if they have many small groups, you can ask them how they were merge them if they had to. This can help a lot with standardizing the groups during your analysis.
Unmoderated card sorts also generally need to be run online with some kind of tool to record the groupings, like Optimal.

For some best practices to keep in mind:

Make sure participants know that they can change around their groups as much as they like until they confirm this is their “final” group selection.
Make sure participants also know that they can set items to the side if they don’t understand the term or aren’t sure where it belongs.
Only ask the users to label their groups after they complete them so their labels don’t bias their groupings.

These labels aren’t actually good for UI labels, so don’t rely on them for that, but they can be very enlightening about the participant’s thought process for what all the items in the group had in common.

Ask qualitative questions at the end, like:

Were any items especially easy or difficult to place?
Did any items seem to belong in two or more groups?
What thoughts do you have about the items left unsorted (if any)?

12 of 25

Analyze the results

Look for patterns and common groups:

By the label
By which items are grouped together

Create a standardization grid to quantify how many participants placed an item in the same group
Combine the patterns with the qualitative analysis to form recommendations

13 of 25

Example

12 participants, sourced internally
Moderated, conducted on paper
Grouped 52 card items centered around TraceLink’s administrative functionality
Came up with clear common groups
Recommendations led directly to the IA of the administrative area of our new platform

A few years ago, we ran a card sort to determine how to best organize our administrative functionality.
We had to source our participants internally, making sure everyone who participated were familiar with administering TraceLink on our customers’ behalf.
We only had 12 participants, but every session was moderated, so we got a lot of valuable qualitative feedback, and there was very strong alignment in the grouping sin the results that made it fairly easy to interpret even with fewer participants.
The recommendations from this card sort, led directly to the IA we have now on our new platform for our administrative functionality.
I’ve included all of the resources from this card sort in the presentation materials, including the test plan, raw data, and final recommendations.
<Open the Items & Raw Data sheet and walk through:

The Items
The standardized groups
The standardization grid. >

14 of 25

Tree testing

15 of 25

What is tree testing?

Participants follow tasks to find information in your navigation hierarchy without any UI elements or visual indicators

Provides insight into:

How easily users can find information in your product or documentation
Labelling
Where users go to find information that the team didn’t expect

16 of 25

When is tree testing useful?

A follow-up to card sorting
Evaluating or benchmarking existing IA
Comparing two sets of labels (A/B tree testing)

Most importantly, tree testing is a great follow-up to card sorting.

Card sorting gives you a good idea of where to start, and a first stab at the labels. Once you come up with a proposal for the navigation and organization, you can use a tree test to validate that what you’ve come up with actual works, or find out where it doesn’t work and adjust accordingly.

You can also use tree tests to evaluate or benchmark an existing IA.

You might have anecdotal stories about a couple problem areas, but need evidence to demonstrate that to stakeholders, which tree tests can provide relatively easily.
Or, you’re planning to make changes to the IA and you want benchmark results to prove that the changes have made a positive difference for users, so you can run a tree test on the existing IA, and then run another tree test on the new IA.

Often, we end up with different IA options and need to choose the best one. This could be because stakeholders have differing opinions, or even because your card sort was clearly split in a few key areas.

Running a test to compare two trees, like an A/B test, can be extremely useful for determining which option works best.

17 of 25

How to run a tree test

Build the tree:

Text-only labels from the UI
Entire hierarchy

Write the tasks:

Use different terminology
Keep them simple
Choose high-priority goals to focus on
Define correct answers

Recruit 20-50 participants, representative of users
Moderated or unmoderated

Best practices:

Include 10 or less tasks
Run a pilot test
If doing 2 trees, have 2 separate participant groups and keep the tasks the same
Ask users qualitative questions at the end

To run a tree test:

First, you need to build out the tree:

Make sure you’re taking the exact labels from the UI.
And make sure you’re including the entire hierarchy for whatever you’re trying to test.

Even if you just have a question about one section, it can be beneficial to include all sections, because you might not realize that users actually have trouble deciding between the two sections in the first place.
Make sure you include all levels of the hierarchy as well, so that participants have to go as deep as they would in your navigational structure to get to the screen they need.

Then you need to write out the tasks, like you would for a traditional usability test.

I’m not going to go into too much detail on how to write good tasks, but I’ll be happy to talk your ear off about it later if you’re interested.
A couple of key points to keep in mind:

Use different terminology in the task than you have in the tree. For example, if you use something like “Update” in the tree, make sure your task talks about “modifying” instead, again to prevent people from just looking for a word match.
Keep the tasks simple. Give just enough context that participants understand what they’re tryin to do, but extra context doesn’t distract them from understanding the main point.
Choose a handful of high-priority goals or problem areas to focus on.

You don’t need to make sure that you have task for every menu item. You’ll learn what makes people select other menu items as they’re going through the tasks.

Make sure you define all correct answers, especially if there are multiple.

Because tree testing is so heavily focused on the quantitative analysis, which we’ll talk about in a minute, you need a higher number of participants to gather useful results, and again, you always want to make sure they’re representative of your real users, even if they aren’t your actual users.
Like card sorting, tree testing can be done moderated or unmoderated.

Moderated allows for more qualitative results: recording participants’ thoughts as they’re moving through the tree, ask them more questions about why they chose certain items.
Moderated can also help you interpret the results if you have less participants.
Again, like card sorting, unmoderated usually requires some kind of tree testing software, like Optimal Workshop, because you need to be able to record every click the user takes.

As a best practice:

You usually want to keep the number of tasks down to 10 or less, so you don’t overwhelm participants.
Make sure you run a pilot test to ensure the tasks are understandable.
If you run 2 tests with 2 different trees, always have separate participant groups. Just like in A/B testing, if you show the same person a second tree, they will be biased by their experience with the first tree, and the results won’t be as useful.

Keep the tasks the same for consistency.

Ask qualitative questions at the end, such as:

How confident were you that you found the right places to complete the tasks?
How easy or difficult was it for you to find the right places to go to complete the tasks?
Do you have any other thoughts or feedback you'd like to give?

18 of 25

Analyze the results

Quantitative:

Success rate
Directness
Time taken
First clicks
Final destinations

Qualitative:

Use to help interpret the quantitative results

There are many quantitative metrics that you should plan to log when you create your tree test:

Success rate – How many participants succeeded at finding the correct destination in each task.
Directness – How many clicks the participant took to find the correct destination compared to the ideal number of clicks.
Time taken – How long participants took on each task.
First clicks – Which option the participants clicked on first for each task.
Final destinations – Which options the participants selected as their final destination for each task.

If you’re using an online tool like Optimal Workshop, it will record each of these metrics for you, which significantly cuts down on the work. If you don’t have that option, make sure you record each session, so that you or someone else can go back and record each of these metrics. Attempting to record them all during the session, even as a separate observer, won’t work very well!

As with the card sort, any qualitative results you gathered will help you interpret the quantitative results.

19 of 25

Example

18 participants, sourced internally
Moderated, conducted through Flare HTML outputs
Walked through tasks in 2 different trees (9 each)

2 options for the IA of a new documentation type

Found common themes about labelling and where certain tasks belonged
Implemented the recommendations before the first publication of the document

We’ve run quite a few tree tests at TraceLink. Since we have Optimal Workshop, it’s one of the quickest and easiest ways we have to gather input on our IA, and I like use it anytime there’s disagreement amongst the team or the stakeholders on the direction we should go.
This example is from a couple years ago, but it was specifically about a new type of documentation we were developing, rather than the product, so I thought it would be interesting to you all. We didn’t have Optimal Workshop at the time, so it was all done by hand, using Flare to build the trees.
The new documentation type had a lot more information than we’ve ever included in a single Help Center before, and we had 2 different ways we could have organized the information: by use case or by task. So we set up 2 different trees, and sourced our participants from anyone inside the company who had a working knowledge of the pharmaceutical supply chain, but didn’t know anything about the Russia compliance requirements.
We were only able to find 9 participants for each test, but we ran every session with a moderator to gather all of the qualitative feedback we could.
In the end, the qualitative feedback turned out to be more valuable than the quantitative feedback, since we had a lower number of participants, but it still gave us enough information to improve our labelling and the placement of certain tasks, and the test did definitively tell us that the tree organized by use case was much easier for users to understand.
Since this was for the documentation, we were able to implement the changes before the writers actually began working on the document, which caused less churn for them, and led to a better doc for initial publication.
<Open one of the Tree Flare Outputs and demo how it was built and how it works.>
<Open the recommendations presentation to show some of the findings.>

Slides: 15, 16, 17

20 of 25

Tools

21 of 25

Paper

Pros

Tactile
Easy for participants to understand
Easier to get participants to think out loud
More personal
Low cost to build

Cons

Must be in-person
Harder to record results
Harder to record the entire session for later qualitative analysis
Requires manual analysis or manual input

22 of 25

Flare

Pros

Allows for remote sessions
Easy to record sessions for later analysis
Less complex for participants to understand
Easy to build
Low cost to build

Cons

Only works for tree tests
Must be moderated, or the build output needs to be more complex
Requires manual analysis or manual input

23 of 25

Usability tools (e.g. Optimal Workshop)

Pros

Allows for remote, unmoderated sessions
Instant analysis after each session
Easy-to-understand visual representation of results
Quick to build

Cons

Can be expensive
Require some onboarding to the tool in order to build
Tempting to rely only on unmoderated sessions
Amount of ways to analyze data can be overwhelming

Finally, we have usability tools built specifically for card sorting and tree testing. One of the most popular is Optimal Workshop, which we use at TraceLink. It’s significantly reduced the time and effort needed to run these tests, which allows us to run many more of them. We typically run 2 to 3 tree tests in a year, depending on what we’re working on at the time. For a team without a dedicated usability tester or user researcher, that’s a lot!

Optimal is really user-friendly for the test and the participants, and makes it extremely easy to send out the tests for unmoderated sessions, which means you can get more participants with less time spent recruiting and moderating. After each session, the data is added to the overall results, and everything is calculated for you and presented in vry interesting and visually appealing graphics, which are fantastic for presenting results to stakeholders.

The biggest con of course, is the cost, and it’s something you’ll have to argue your case for when it comes to budgeting season. We actually logged all of the time spent on doing that documentation tree test I showed you earlier as a way to make our case for paying for a couple licenses, and it worked.

Really quickly, I just want to show you some visuals that you can get out of Optimal Workshop for card sorting and tree testing, because this is really the coolest part:

Card sort: https://app.optimalworkshop.com/optimalsort/5q66u6s8/4f8h377s/shared-results/8jc5v382xo6q4rnz57pkzhwcky02kk41

Show the Standardization Grid and the Dendrogram

Tree test: https://app.optimalworkshop.com/treejack/5q66u6s8/jq7w8v1f/shared-results#/t/results/overview

Password: ProtectPatients-2020
Show the Overview briefly
Show the Task analysis and the Pie tree of Tasks 1 and 2 – both high success rates, but one has much lower directness.
Briefly explain the First Click, Paths, and Destinations tabs – more of a drill-down to research specifics about the overall task results

24 of 25

Questions?

Extra Information

25 of 25

Resources