In-Product Learning 🎃
Dave McCarter
Principal Design Technologist
LUNCH & LEARN
Liz Hill
Product Design Lead, Education
Introductions
Overview
Approach
Explorations and Prototypes
What You Can Do
What's Next
Wrap-up
Questions
Agenda
Liz Hill
INTRODUCTIONS
3
Dave McCarter
INTRODUCTIONS
4
It all started with a SPOOKY convergence of signs
Several designers had worked on the same concept over the years
User research findings had consistent themes- users need more guidance when setting up features
Other companies were providing the same support we kept exploring and not actioning on
Today…
OVERVIEW
6
MongoDB's products are highly technical, and we’re not consistent in how we provide supporting content to our users.
Adding to the problem, existing in-product help can get out of date quickly, confusing users in the process.
Because of this, users often seek help from resources outside of the product they're using to be successful.
PM & PUX Feedback
OVERVIEW
7
Users struggle to select the right configuration for their use case with some features.
Users need support for advanced features and settings, troubleshooting, and best practices.
Helping users in the right ways can have a huge impact on their success.
8
Reading and Retention
OVERVIEW
9
Users will forget ~70% of what you teach them within a day so we need to provide users with supportive text for flows they may not regularly engage with. (Ebbinghaus Forgetting Curve)
On the average web page, users read at most 28% of the words during an average visit so each word counts.
Distractions
OVERVIEW
10
Showing the non-relevant or distracting information to users can pull them away from elements that are crucial to their success, or from trying products or features that they would find useful.
Presenting users with links to the resources at the wrong times sometimes distracts users, causing them to not complete their tasks.
An Example
OVERVIEW
11
We wanted to know…
How can we help users at the right times, in the right ways?
12
Three Facets of IPL
APPROACH
13
We found that solving these problems required considering multiple angles.
We looked at three types of approaches that together, could make a huge difference for users.
Clear & Relevant Content
APPROACH
14
Show users too much information and it's overwhelming. Show them irrelevant information and they become distracted. Give users too little information and they might struggle to complete their task.
To be successful, we needed think about best practices for writing clear, relevant, and concise content for users.
Design Patterns
APPROACH
15
Sometimes how information is presented to users can be just as important as what that information is.
We needed to explore new ways to present contextual information to users to aid them in their tasks without overwhelming them.
AI
APPROACH
16
AI presents a unique opportunity to help our users, especially in cases that are hard to anticipate.
AI can lower the barrier to entry for certain complex and technical tasks. We should be thinking about how we can leverage AI to simplify tasks.
Prototypes
We started exploring…
17
Creating an Index
PROTOTYPES – CREATE INDEX
18
We noticed that the create index flow is light on information that helps users complete their task. It offered a great canvas to test several approaches to helping users contextually.
Create Index: Approach 1
Using Current Patterns
19
PROTOTYPES – CREATE INDEX
Approach 1
20
We started with a prototype applying our existing patterns for in-product documentation and learning to the Create Index flow.
PROTOTYPES – CREATE INDEX
Approach 1
21
Users had no problem finding details on topics like index options. However, they would need to open those docs in a separate window to confidently complete the flow.
PROTOTYPES – CREATE INDEX
Approach 1
22
Users leveraged example values in the JSON editor to quickly reference what potential values they could use, and to understand the meaning of "1" and "-1."
PROTOTYPES – CREATE INDEX
Approach 1
23
However, the inclusion of the more niche "2d" and "2dsphere" values confused several users, causing them to re-evaluate their understanding of the field.
PROTOTYPES – CREATE INDEX
Approach 1
24
Users easily identified that the inline code hints we provided directly referenced their data model. It was an effective way to contextually surface their data model.
PROTOTYPES – CREATE INDEX
Approach 1
25
Code hinting was also well received for static sets of values. Users quickly began expecting IDE-style hinting to be available across all prototypes, and were dissatisfied when it was not available.
PROTOTYPES – CREATE INDEX
Approach 1
26
Some users were confused by the use of MongoDB-specific terms like "rolling index build process" and "failover," causing their confidence in creating an index to decrease.
Create Index: Approach 2
Full Code Editing
27
PROTOTYPES – CREATE INDEX
Approach 2
28
The second exposed users to the code to create an index. It aimed to be more natural than three separate JSON fields, and help users move to programmatic and command line interfaces later in their journey.
PROTOTYPES – CREATE INDEX
Approach 2
29
A benefit to showing the code is that unlike with JSON, JavaScript allows for inline comments. Users loved the clear, inline documentation within the editor, and that they can easily un-comment the options that they want to use.
PROTOTYPES – CREATE INDEX
Approach 2
30
Users also liked that only the most common index options were shown, and that the optionality, types, and the default behavior for each option were clearly provided.
PROTOTYPES – CREATE INDEX
Approach 2
31
However, this approach created a heavy cognitive load.
Users felt that this approach would be slower to use since they need to pay attention to the code syntax.
PROTOTYPES – CREATE INDEX
Approach 2
32
Users also expressed concern and uncertainty about edge-case behaviors, such as what would happen if they changed their collection namespace.
PROTOTYPES – CREATE INDEX
Approach 2
33
Overall, this approach turned out to be our most controversial prototype. Some users were excited by the prototype, and the power scripting allows. Others preferred a more guided approach where the behavior was unambiguous.
Create Index: Approach 3
The Options Panel
34
Approach 3
PROTOTYPES – CREATE INDEX
35
The third prototype introduced an options panel, allowing users to view documentation on options directly within the flow.
Approach 3
PROTOTYPES – CREATE INDEX
36
Once users saw the options tab, they were able to accurately anticipate that the panel would appear, and what information the panel would contain before interacting with it.
Approach 3
PROTOTYPES – CREATE INDEX
37
The information it contained was well liked by users, with several noting how they loved having the information they need without needing to visit the documentation.
Approach 3
PROTOTYPES – CREATE INDEX
38
The "Add" buttons were also universally well-liked, with users noting them as helpful and intuitive to use.
Approach 3
PROTOTYPES – CREATE INDEX
39
This was the clear favorite prototype across the users we spoke with, combining clear, contextually relevant content within the page, with an intuitive interface.
Create Index: Approach 4
Index Creation Assistant
40
Assistant
PROTOTYPES – CREATE INDEX
41
We also explored helping users create the index they need automatically.
The assistant allows users to to create a new index definition by entering the query they would like to cover.
Assistant
PROTOTYPES – CREATE INDEX
42
Despite no direct mention, users assumed it used AI because of cues like the words “Assistant” and “Generate.”
As a result, users expected they could use natural language to create an index.
Assistant
PROTOTYPES – CREATE INDEX
43
When they did generate an index definition, they valued the ability to review and edit the generated outputs before the index was built.
Create Time Series Collection
Supporting Content
44
Creating a Time Series Collection
PROTOTYPES – CREATE TS COLLECTION
45
After hearing that users struggle to select the right options when setting up a Time Series Collection (timefield, metafield, granularity) we decided to test a flow with additional guidance for these selections. This flow included simplified language and expandable descriptions.
Time Series
PROTOTYPES – CREATE TS COLLECTION
46
We used descriptive titles for the input labels and included the field name at the end.
This gave users more clarity on the options before even reading the descriptions.
Time Series
PROTOTYPES – CREATE TS COLLECTION
47
Users had the ability to expand and collapse descriptions. This interaction was easily understood by users, and allowed us to include more relevant information for difficult concepts.
Time Series
PROTOTYPES – CREATE TS COLLECTION
48
Descriptions included 3 parts to help users understand the input.
Time Series
PROTOTYPES – CREATE TS COLLECTION
49
Overall, the focus on simpler and clearer content helped users complete this flow.
The expanding descriptions design pattern also proved to be an excellent tool to add to our toolbox.
Artificial Intelligence
Atlas Chatbot
50
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
51
Due in part to the success of the MongoDB Docs chatbot, we wanted to explore if a chatbot within Atlas could be an effective tool to help users learn about, and use Atlas.
Since modern chatbots are dynamic, we needed to take a novel approach to prototyping a chatbot so that we could see how user would actually react to, and engage with one.
So… we built one!
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
52
Appearing through a button at the bottom-right of the screen, the prototype was contextually aware, knowing some details about any cluster you looked at. It used RAG to answer questions accurately, and gave answers relevant to the user's preferred programming language by default.
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
53
Users trusted the chatbot more than ChatGPT to accurately answer MongoDB and Atlas-specific questions.
They also liked that their preferred language was taken into account.
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
54
Users expected it would be able to retrieve details about their MongoDB cluster, and any other resources.
They noted that they expected the chatbot to have access to any details that they did.
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
55
They also expected it to be aware of what they were looking at on the page.
It met their expectations when the chatbot knew details about the cluster they were viewing.
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
56
Users broadly trusted the security of a chatbot within the Atlas UI with access to their resources and data, citing that they trusted MongoDB to take the appropriate security and privacy precautions.
Atlas Chatbot
PROTOTYPES – ATLAS CHATBOT
57
The high user expectations of an Atlas Chatbot became clear.
AI Chatbot
PROTOTYPES
58
Because the chatbot was within Atlas, users expected it to provide more than documentation, and expressed disappointment when it didn't deliver.
Some noted that they wouldn't use the chatbot at all if it didn't provide convenient access to information about their account or resources.
AI Chatbot
PROTOTYPES
59
An Atlas chatbot MVP would need to be advanced to meet users' lowest expectations.
AI Chatbot
PROTOTYPES
60
To delight users, we would need to go even further.
Prototypes
Other Ideas & Concepts
61
Define Anything
OTHER IDEAS & CONCEPTS
62
We explored some other ideas that we decided not to test, but might be great to explore in the future!
One of which was to leverage AI to allow users to get an explanation of anything they want in Atlas that we called "Define Anything."
Audit/Other information
PROTOTYPES
63
We explored some other ideas that we decided not to test, but might be great to explore more in the future!
Support Drawer
OTHER IDEAS & CONCEPTS
64
Several designers have explored using a drawer to surface additional learning material in places like Search and FGA, rather than linking users to Docs. ��This pattern has been received favorably in user testing and in performed well in a Search experiment.
65
<Docs sidebar?>
66
Support Drawer
OTHER IDEAS & CONCEPTS
67
As the Design Systems team builds out the support drawer for the new Atlas nav this concept will continue to be explored and expanded upon.
With all we've tested and learned…
Where do we go now?
68
You can start using these learnings today!
69
Clear & Relevant Content
RECOMMENDATIONS
70
Content can have just as much, if not more impact on usability as the design patterns you use.
Be sure that the content included on the page is top of mind when designing, performing user interviews, and making UX decisions.
Clear & Relevant Content
RECOMMENDATIONS
71
It's often better to be clear, than to be comprehensive.
Ensure that copy only includes necessary detail, and uses simple terminology when possible. Putting details related to niche use-cases or highly technical explanations can confuse users in otherwise clear tasks.
Clear & Relevant Content
RECOMMENDATIONS
72
Adding links should be deliberate, not the default.
Ensure each link is purposeful and relevant to its context. Users can become distracted by links that aren't directly relevant to their task or they need to search through a Docs page for the relevant section.
Additionally, most users are comfortable finding the docs they need without a link. Evaluate if a link is necessary before adding one.
Clear & Relevant Content
RECOMMENDATIONS
73
Make sure an interface can stand on its own.
Directly include the details needed to complete the most common use-cases in your designs.
Users are delighted when they can complete their task confidently without needing to visit external resources.
Design Patterns
RECOMMENDATIONS
74
Don’t interrupt users.��Guide cues and modals can interrupt users flow of work. Be deliberate with visual hierarchy to draw users attention to what’s most important.
Introduce guide cues as part of a pull revelation and use them sparingly. Make good use of titles and descriptions.
Design Patterns
RECOMMENDATIONS
75
Break things up- Don’t overwhelm users by giving them all of their options at once.
Use progressive disclosure, collapse advanced options, ensure you’re chunking longer tasks into smaller more manageable pieces.
AI
RECOMMENDATIONS
76
Consider more specific AI related features
AI can be used to help users overcome technical hurdles and simplify workflows
Be careful when using terminology that could be indicate a non AI feature is using AI like “generate” or “assistant”
77
What’s Next?
78
Content Guidelines
WHAT'S NEXT
79
Pulling together a set of content guidelines to review with a cross-functional team to encourage consistency in our content experience and ensure we’re using best practices when considering the content in MongoDB’s user experiences
Patterns
WHAT'S NEXT
80
Design Systems is building out:
The “Show More” pattern when inputs need more context than a 2 line description can offer
The “Options Tab” to show various options available in JSON editors
The support drawer for supplemental content not directly related to one input
AI
WHAT'S NEXT
81
We recommend not proceeding with an AI chat interface in the Atlas UI unless desired features are included. ��Pursue AI chat support in logged out experiences where the chatbot meets users’ expectations.
Continue to investigate more targeted AI features.
WRAP UP
It Takes a Village
82
Andrea Bascetta
Ksenia Samokhvalova
Yuxin Cheng
Allison Mui
Elena Woiciechowska
Ben Perlmutter
Nick Larew
Shayla Hill
Kellie Carter
Charlie Xu
Kaylee Won
Robert Androue
Eliza Spang
Julia Oppenheim
Benjamin Flast
Christopher Shum
Frank Sun
Nishith Atreya
Joe Niemiec
Julian Gilliatt
Tonya Edmonds
Sahi Muthyala
Justin Grinstead
Brooke Yalof
Questions?
Thank You!
— ARCHIVE —
85
PROTOTYPES
Approach 1
Creating an Index
86
CREATE INDEX EXPLORATION
Approach 1
87
Labels were updated, and descriptions for each input were added to make sure each input's purpose was clear.
CREATE INDEX EXPLORATION
Approach 1
88
The addition of input description text was relied on heavily by users interacting with the designs.
Across our prototypes, their inclusion was an effective aid to user understanding of each field.
WHAT WE FOUND
CREATE INDEX EXPLORATION
Approach 1
89
Links to documentation on the options users would need to specify were included inline with the description for the field.
CREATE INDEX EXPLORATION
Approach 1
90
Users knew where to find information on topics like "Index Options," but need to use those docs to complete their task and feel confident about their choices.
They also expressed comfort with finding the information they needed using Google if a link wasn't present.
WHAT WE FOUND
CREATE INDEX EXPLORATION
Approach 1
91
Example values were included in the field to help users quickly identify possible values, and what they meant.
The modal auto-focuses on the input, and selects the key that a user would need to edit when it is opened.
CREATE INDEX EXPLORATION
Approach 1
92
Users leveraged the example values to quickly reference and understand the meaning of the "1" and "-1" values.
However, the inclusion of the more niche "2d" and "2dsphere" values confused several users and caused them to second-guess their understanding of what the input was defining.
Additionally, some users initially mistook the examples as valid, default values.
WHAT WE FOUND
CREATE INDEX EXPLORATION
Approach 1
93
The collection's inferred schema is used to provide contextual hinting to reduce the user's need to open a separate window to reference their data model.
CREATE INDEX EXPLORATION
Approach 1
94
Most users were able to quickly identify that the options in the hinting menu referenced their own data model.
Hinting quickly became expected by users. When one prototype failed to show this functionality, users expressed dissatisfaction with its omission.
Hinting within code editors is well liked and understood by users, and would provide a much-needed, contextual way to surface their data model.
WHAT WE FOUND
CREATE INDEX EXPLORATION
Approach 1
95
Similarly, we also provided hinting for possible values for field definitions, as well as index options, which was similarly well received by users.
CREATE INDEX EXPLORATION
Approach 1
96
Instead of using links and info sprinkles, details on the rolling index build process were included directly on the page.
CREATE INDEX EXPLORATION
Approach 1
97
Users weren't turned away by the inclusion of additional information inline on the page, and most users took the time to read about the option.
However, some users were confused by the use of MongoDB-specific concepts like "rolling index build process" and "failover," causing their confidence completing the task to decrease.
WHAT WE FOUND
PROTOTYPES
Approach 2
Creating an Index
98
The second prototype took a more novel approach:
PROTOTYPES
Approach 2
Creating an Index
99
<Create index prototype 1 image(s)>
PROTOTYPES
Approach 2
Creating an Index
100
This was the clear favorite for most users. They found it clear and convenient to create an index with just the information on the page.
PROTOTYPES
Approach 3
Creating an Index
101
The third prototype explored helping users learn our programmatic tools while completing their task in the UI.
PROTOTYPES
Approach 3
Creating an Index
102
<Create index prototype 1 image(s)>
PROTOTYPES
Approach 3
Creating an Index
103
This approach was the most polarizing among users.
PROTOTYPES
Approach 4
Creating an Index
104
Lastly, we tested providing users with an "index assistant" experience.
PROTOTYPES
Approach 4
Creating an Index
105
<Create index prototype 1 image(s)>
PROTOTYPES
Approach 4
Creating an Index
106
<What we found>
Time Series
PROTOTYPES – CREATE TS COLLECTION
107
We started by moving the instructions for database name and collection name into the input descriptions.