SUMO

Information Architecture Project – Phase 2

March 2012

​

Susan Farrell – Nielsen Norman Group�Bram Pitoyo – bram@mozilla.com

Overview

• Project goal and scope
• What we’ve done
• Findings, insights and recommendations
• What now?
• What next?

​

Ask question and take notes:

https://docs.google.com/document/d/1CVYmD0pjWMT3fGE13no8-EJcyiF-GfuMQ-d95O0kCXY/edit

Project goal

Outfit SUMO with a new design, based on a new Information Architecture / taxonomy that:

• Is scalable
• Has a robust structure
• Has navigation built to be extensible
• Is based on research and data

​

We focused on what SUMO users are doing,�in order to optimize their tasks, by making navigation and search results improvements.

Project scope

Analyze Data and Propose IA Structure

Remote Test IA Structure

This presentation

Test Paper Prototypes

Turn Paper Prototypes into Wireframes, Test

Develop Wireframes into Web Pages

Introduce New Website to the Community

What we’ve done

• Sort SUMO KB articles (OptimalSort)
• Test new task-based navigation (Treejack)
• Test new product-based navigation
• Revise IA, and test combined navigation

KB article sort

https://docs.google.com/spreadsheet/ccc?key=0AqTssbJSHVTndDNac0VkdHFOY1BwejN0Zk0zRXZDVUE#gid=0

• 220 articles, split into thirds
• Sorted into 19 pre-made topics
• 18, 17 & 16 respondents, respectively
• Testers: help forum and KB contributors
• We planned to do one more sort, but all articles in “Other” &“?” fit into obvious topics
• Result: renaming recommendations, cross-referenced topics a.k.a. "see also"
• Sorted KBs populated the new IA

Task-based navigation test

https://docs.google.com/spreadsheet/ccc?key=0AqTssbJSHVTndF9iTGgxTjBYS1oxQXlNbTkwR21Qb2c#gid=8

Product-based navigation test

https://docs.google.com/spreadsheet/ccc?key=0AqTssbJSHVTndEtLMGt3TzRMS19wR0tLM2ZxS3BPY0E

• 50 respondents: intercepted users of SUMO
• 13 tasks (the same ones we ran before)
• With TreeJack, we were able to find out:
• Where people looked - (first-click)
• Mental-model disconnects - ambiguous and misleading titles ("evil attractors" a.k.a. "red herrings")
• Time it takes to get to the information, and path directness (efficiency)

Task- & Product-based

test findings

More KB articles that need:

• Creation (knowledge gaps)
• Title/URL renaming
• Content tweaking (minor and major)
• Concept consolidation and clarification
• A bigger conversation (frequent wrong targets)
• “Basic Troubleshooting”, “Firefox is slow”
• Diagnostics for self-help needed�(process instead of search)

Task- & Product-based

test findings

Tasks that had problems unrelated to the IA:

• #1: You want to change your Firefox home page to Wikipedia. (People expect home page and search engine controls to be located on the surface of the browser)

​

• #3: Find out what’s causing Firefox to stop working sometimes. (The article most people chose sounds like the right answer but doesn't have the right content for the task)

​

• #7: You want to shop at Amazon but you don’t want Firefox to automatically sign you in. (We suspect many people don't know whether it’s Firefox that does this or the website.)

Task- & Product-based test findings

• A couple of red herrings:
• Basic troubleshooting
• How do I fix Firefox
• Is there a place where we can tell users that Profiles, not reinstalling, is often the solution?�Some people want to reinstall to resolve problems.
• Profiles vs. Persona
• Navigation Toolbar Items (expectation that controls are there)

​

Combined navigation test

https://docs.google.com/spreadsheet/ccc?key=0AqTssbJSHVTndHNSRmtBeS1oSEh2SEZyNzBIel9QcEE

50 respondents (users), 13 tasks (reworded),�and a tweaked tree

​

You want to change your home page to Wikipedia.�You want to change the page that first opens when you start the browser.

​

You want to shop at Amazon but you don’t want Firefox to automatically�sign you in. Find out how to change it so it won’t do that.�Find out how to change your browser so it doesn’t automatically log you in to websites.

​

Today something on Facebook doesn’t seem to be working right. Is it Firefox or Facebook? Where would you look to see if anyone else is having that problem?�Something on a website doesn’t seem right. Find out how to determine whether it’s Firefox’s problem or the website’s problem.

​

You need to give this computer to your mother, so you want to erase all your personal info from Firefox. Find out what you need to do to make sure it’s gone.�Find out how to erase information about you that Firefox has saved.

Task success scores:

some caveats

• We tweaked our sitemap after each test.
• We reworded some tasks for our Combo test - which makes the tweaked-task results not directly comparable with the first three tests - in order to determine whether the task wording or the names of things in the IA were causing some red herrings.
• The new sitemaps we used in Treejack contain just the task- and product-based structures. The deliverable has Hot Topics, News, auxiliary navigation, etc.
• The old (January 2012) IA was tested with all navigation and quick links intact. It was more complex but had a lot of quick links on the home page.
• We determined problematic articles by tallying the wrong answers for each article.

Task 1:

(Changed task). You want to change the page that first opens when you start browsing with Firefox. Find instructions for that.

Old: 75%. Task: 53%. Product: 67%. All: 79%.

​

We know that this is a malware problem, but as long as we have this problem, people will still look for this page.

​

The SearchReset Add-on is a great start.�Can we link to more self-help resources?

​

Is my home page my search engine?

Task 2:

Find out how you can make Firefox work faster.

Old: 1%. Task: 77%. Product: 61%. All: 86%.

​

Crash/hang issue. Contributors know the difference, but users may not. (Content issue).

​

Basic Troubleshooting / How do I fix Firefox / Firefox is Slow are easy to find but don’t lead users through a process.

Task 3:

Find out what’s causing Firefox to stop working sometimes.

Old: 60%. Task: 47%. Product: 27%. All: 69%.

​

(Hang vs. crash) Too many possible targets. Consolidate articles.

Task 4:

You bought a computer without Firefox on it. Where would you go to get it?

Old: 53%. Task: 81%. Product: 93%. All: 92%.

​

In the top 304 search terms for a 3-month period in late 2011:

• 37410 searches for download and/or update
• 53944 total, including version and language queries

​

We link to download and update wherever people expect to see them:

• Specific task on home pages (SUMO and software landing pages)
• In the first-level nav (no need to click a topic)
• On software landing pages (Download and Install, Getting Started;�no need to click a KB article)

​

This bucket is not just a link to marketing. SUMO’s is bigger and includes problems and solutions as well as docs. (Doc search needs discussion).

​

Task 4:

You bought a computer without Firefox on it. Where would you go to get it?

It turns out that people were downloading malware-infected Firefox.

​

It is unusually difficult to find links to download on SUMO. Most companies don’t have separation between support and download the way Mozilla does. It makes sense to host the download page on Support, but the two ought to link back in any case.

​

If we can write about the importance of downloading software from trusted source (see also: Getting Started), we might help solve the “Changing home page,”�“Changing search engine” and “Firefox is slow” problems.

Task 5:

(Changed task). Find out how to erase all the information about you that Firefox has saved.

Old: 57%. Task: 70%. Product: 67%. All: 71%.

​

Consolidate. Call it “Delete history, cache and other personal information”. Users understand history, cache, and cookies as being related to security.

​

Privacy and security are hot topics that are not well understood technically. What makes it all go away?�How can users be confident that they’re protected?

​

Users don’t relate this to “autofill” and “autocomplete”, even though they’re part of the problem�(arguably, a more important part)

Task 6:

Your 9-year-old needs to use the Internet for homework research. Find out how to prevent her from visiting inappropriate websites with Firefox.

Old: 34%. Task: 73%. Product: 55%. All: 71%.

​

There were a lot of queries about “blocking”, “unblocking”, “trusting” and “parental controls” (4708 we know in the Top 304 queries, approaching 6000 in the Top 1000)

​

The most effective solutions lie at the OS level rather than add-ons or features that Firefox will build, so we need to create tutorials for them.

Task 7:

(Changed task). You want websites to ask you to type in your username and password every time, instead of having Firefox do it for you. Find instructions for that.

Old: 1%. Task: 39%. Product: 40%. All: 63%.

​

Managing passwords is the most difficult task, perhaps because it’s not well understood. Needed: education, better visibility in product, content improvement, naming.

​

Is it a checkbox on the website, or a Firefox setting?�Who’s really responsible for it (and is it related to cookies)? If it’s a Firefox setting, why is it not exposed?

​

Only 2 people skipped this question after “Amazon” was removed, so task wording was an issue.

​

​

Task 8:

Find out how to get your Firefox bookmarks from your old computer onto your new one.

Old: 55%. Task: 79%. Product: 83%. All: 83%.

​

Import/export bookmarks and Sync keep getting intertwined. (Content issue for Sync)

​

We have made Bookmarks very prominent in the IA. The problem is not that people couldn’t find bookmarks. It’s that they picked Sync often.

Task 8:

Find out how to get your Firefox bookmarks from your old computer onto your new one.

Sync is one (and perhaps the preferred) way for people to move bookmarks around. Currently, it is not sanctioned. How can we support this?

​

People are learning about syncing email and contacts on their smartphones by deciding which device “wins”. Could we Sync one-way, and/or one-time (rather than “the truth is in�the Cloud”)?

Task 9:

Change Firefox so that it searches Bing instead of Google.

Old: 8%. Task: 71%. Product: 77%. All: 87%.

​

Search engine and home page controls are not on the surface of the browser, which is where some people expect them to be. (Direct manipulation)

​

People are also confused because their home page�is a search engine.

​

You can’t change your search engine while you are tinkering with that control, the same way you can’t change your home page from your home page.

Task 10:

Find out what Firefox's “private browsing” is and what it protects you from.

Old: 67%. Task: 78%. Product: 76%. All: 82%.

​

Some people clicked “private browsing” for a solution to erasing their password. Although sites ask users for passwords during private browsing, it is not the right solution, because it doesn't erase saved information. (Password security issue)

​

Private browsing is not well understood, but people looked there for security and privacy concerns.

Task 11:

(Changed task). Something on a website doesn’t seem right. Find out how to determine whether it’s Firefox’s problem or the website’s problem.

Old: 11%. Task: 80%. Product: 76%. All: 69%.

​

Non-Mozilla website issues (including webmail): 67,518

Facebook-related alone: 36,738

Games: 3,112

Total: 75,414

​

We created a bucket for this called “Fixing problems with websites”, because there are a lot of search queries. Users need to know where to look before they ask questions and crowd our support forums.

Task 11:

Something on a website doesn’t seem right. Find out how to determine whether it’s Firefox’s problem or the website’s problem.

Similar to usernames and passwords, users might not know what causes website problems: the browser, or the website?

​

• We need a way to self-diagnose
• We will also need content that tackles specific problems (Zynga, CityVille, FarmVille, Facebook Games)
• When people search for websites, they may be treating SUMO’s search box as a Google search box. How many are having technical problems with the site vs. how many just want to find that site? (Intercept, measure)

Task 12:

You want to have the same Firefox tabs, bookmarks, history and passwords on your work computer as you do on your home computer. Find instructions for this.

Old: 37%. Task: 62%. Product: 74%. All: 76%.

​

Everyone looked everywhere for Sync. They’ll need to see more than one word.

​

Could Sync be considered a feature rather than a service? (Sync causes a lot of confusion with import/export Bookmarks). Plus, Firefox Home (Sync?) for Apple iOS does the same thing.

Task 13:

You want to install Firefox Mobile on your smartphone. Find out whether it will work on your model.

Old: 54%. Task: 65%. Product: 70%. All: 68%.

​

Articles about installing and getting started with Firefox mobile should offer Sync information.

​

Compatibility information should exist on the Mobile blog, too.

Combo navigation test:

Findings

• Final score is better than the separated tests, even though the structure is bigger.

​

• Changes in task wording also seemed to help.

Combo navigation test:

Findings

People chose the task-based path most of�the time:

​

• There are many test influences that had to do with both positioning and information scent. Ordering matters a lot on lists of content.

​

• Task-based is preferred for usability because a lot of users want to do things, not find/look around for things. (Mental model match)

Combo navigation test:

Findings

Input/Feedback is going to be a link on SUMO.�We will need to:

• Build infrastructure for tagging input. The problem list will be auto-categorized for us. We can perhaps use the same forum tags for it and treat it like bug tracking (to avoid double/no work on an issue).
• File bugs and resolve problems about top/emergent things in this categorized list.
• We can measure success with this list.
• Mailing lists around Mozilla will be good places to send relevant issues to. They should be able to subscribe to Inputs about their products.

What now?

• Some things need more research

​

• New content needed around what users don’t understand

​

• Contributors need to solve many of these issues, to make their jobs easier

​

• Implications for content and SEO

What needs more research

• Navigation comprehension testing. We would show pictures of navigation with wording and ask people: “what do you expect to get when you click on this link?” This is a way for us to investigate whether the user’s mental model matches up with ours.

​

• Topics: home page vs. search engines, autofill/autocomplete & clearing password

What needs more research

• The product picker: one will be displayed (icons), but a link that stays on all pages needs to have an attractive label name. What should autodetect do for us?

​

• Preferred first-click targets?

​

• How many people are searching SUMO instead of navigating to websites; why?

​

​

What needs more research

Title of home page and product landing page must not sound too similar:

​

• Mozilla Support | Software Help
• Mozilla Support Home | Software Help
• Support Home | Software Help
• Mozilla Support Home | Products and Services Help

​

People understand that software is the thing you install, product is the box you buy.�B2G will be tricky.

What users need

and don’t understand

• Name confusion seems pretty consistent in all tests, and it’s at the article level (e.g. Persona and Profiles). Our articles need to be named and linked in ways that will help steer people to the best places and away from things that won’t help them.

​

• Product names were still confusing to some, even with additional info about their purpose. Firefox Home might benefit from an Apple icon next to it. Firefox Mobile could use an Android icon.

What users need

and don’t understand

• Users complained a lot in the comments about similarly named things in one bucket and not knowing which one leads to the solution.

​

• Article consolidation (rolling into one) when the names sound similar to each other will help users choose:��Clearing data, managing bookmarks / Sync, troubleshooting performance & diagnostics.

Task 7 example of similarity problem

You want websites to ask you to type in your username and password every time, instead of having Firefox do it for you. Find instructions for that.

Old: 1%. Task: 39%. Product: 40%. All: 63%.

​

Passwords, forms, search, and history – control what Firefox suggests

10 Forms autofill: How to manage auto-complete features in Firefox

Form autocomplete

Form autocomplete entries are not saved

Awesomebar - How to manage Firefox suggestions and search history

20 Password manager: Remember, delete, and change saved passwords in Firefox

6 How do I make Firefox remember my usernames and passwords?

7 Username and password not remembered

How to clear Search bar history

What users need

and don’t understand

• The users’ mental model of “settings, options, controls and preferences” will need to be resolved somehow. How do we define the toolbar controls that one can set directly from the UI?

​

• Our UI is in flux, and it might introduce new elements (the gear).

What users need

and don’t understand

• What’s the obvious difference between controls and add-ons? For example, search engines can appear in toolbar or options.

​

• The UI treats these the same in many ways, so for the user, there’s no meaningful difference between things you can change and other things you can change. We called them all “customization.” Users just want to change search engines from the search bar. They don’t care who owns the change mechanism.

What users need

and don’t understand

Our tests identified a gap between feeling safe when using the web and actually being safe.

​

Delete all personal information: A lot of people chose “Clear History” as the right answer.

​

There is an opportunity to differentiate Firefox from competitors by making it a browser that not only has good privacy features, but also helps users be better informed and more secure.

​

Needed: UI feedback, affordances, teaching about privacy controls, mental models, and security best practices.

Issues contributors need to solve,

to help make their jobs easier

• Renaming will help users find articles faster and reduce the amount of support questions. Old names should be kept in the <meta> field, and old bookmarks will lead to the same articles.

​

• The landing pages will help contributors browse for things faster, too. We can make cheat sheets to landing pages, and add tags to refine it further.

Issues contributors need to solve,

to help make their jobs easier

• The tag set we have identified and tested in OptimalSort isn't complete (although they were designed to cover most situations).

​

• Contributors will have to change the way they work a bit, but the new process is going to be easier to use.

Implications for

content and SEO

• A lot of the search engine and content tuning tasks will be ongoing processes that need a lot of maintenance.
• Best bets, synonyms, preferred terms for predictive search, search query analysis for emerging issues
• Title/URL, summary and meta-description,�heading and link text tuning

​

• Do these initially to increase clickthrough and search accuracy, but also create processes and do some training around content tuning best practices.

Style guide process

• The style guide is a living document that resolves differences�in style.

​

• The style guide (for writing, adding tags, etc.) should be workshopped to get buy-in from content team members and to familiarize everyone with the style issues.

​

• Draft guide first: just the problems and controversies
• Meet and go over the guide
• Apply the guide to real content (each person does this)
• Compare and contrast results
• Discuss differences, choose the best ways
• Revise guide with examples and remove ambiguities

​

​

What’s next

Budget time and file bugs, on the to-do list:

​

• Content strategy (retitle, tag, summarize, cross-link, add, tweak, clarify, consolidate)

​

• Search engine tuning and optimization, as well as search engine tuning workshops (thanks to Matt_g�for adding notes)

​

• Development tasks (UX bugs, the ability for tags to have landing pages, auto-populate see also’s, KB and forum changes around links to landing pages and tags.)

What’s next

Present emerging issues and UX suggestions to various product teams:

​

• Our analytics is quantitative, not qualitative.�What product teams never see is “how often somebody searches for a solution to a problem then says ‘I can’t find an answer, so I’m going to switch to a Google product’.” (Source: Matt_g)

​

• SUMO is in a perfect position to make these UX improvement suggestions, because it completes the user feedback loop, in order to make the products�better and easier to use.