State of OpenStack User Experience

December 2016

Piet Kruithof

Danielle Mundle

​

“Requests for more communication and cooperation between operators and developers were frequent. ‘There’s still a bit of a gap between developers and operators. Although that is narrowing, understanding operators’ pains in certain areas is key in developing a better product as a whole.’”

​

​

Tretheway, H. J. (2016). OpenStack User Survey: A snapshot of OpenStack users’ attitudes and deployments. Austin, TX: OpenStack Foundation.

​

“Open source software is software developed by and for the user community.”

​

https://opensource.org/

​

Overview

The goal of this presentation is to provide an overview of research that was conducted on behalf of the OpenStack community.

• Efforts spanned both application developers, operators and architects
• Areas of focus included information needs, quota management, OpenStackClient and novice user experience
• Research areas were selected based on a combination of issues identified in the user survey, feedback from operators and requests from PTLs

The current focus on user research is intended to identify the specific issues that impact adoption of OpenStack

• For example, the user survey is very good at identifying the “what” but not the “why”

What went right

Our body of knowledge for our customers (operators, architects, etc) has grown significantly over the past six months.

The team was able to build cadence and conducted six studies on behalf of the community

We were able to collaborate with both the Foundation and Intel’s marketing teams to help communicate our efforts

• GUI Guidelines
• Persona Guidelines
• Enhancing the OpenStack User Experience

General Themes

�December 2016

​

What needs improvement

Governance: Users felt that Technical Committee should govern more

• Needs to be treated more as a product vs. collection of projects because of the amount of cross project work that needs to be done. (notes from deployment study)
• Example: some services refer to “projects” while other refer to “tenants” in the API.
• “Technology is good, but no synergies between the sub-projects.” (OpenStack User Survey)
• “The governance of the OpenStack project seems very fragmented.” (OpenStack User Survey)
• Open source zealots

Disconnect on value of open source: There is a significant difference on how upstream developers and operators perceive the value of open source

• Upstream developers: Writing code allows us to drive product direction! Yay!
• Operators/consumers: No vendor lock-in, Ability to look at raw code, (perceived) ability to help drive production, responsibility for maintaining product is spread across contributing companies

​

What needs improvement

Difficult to implement cross project efforts: A significant number of issues within the community span projects. However, there is no mechanism to force projects to comply.

• Documentation, consistent API calls, product direction, Information gathering

Lack of focus on users: This is an issue with open source projects where the assumption is that developers are creating a project for themselves. Closed loop.

• “Open source software is software developed by and for the user community.” (opensource.org)
• Similar to proprietary software, but more pronounced.

​

​

Cloud Operator Information Needs

�August 2016

​

Cloud Operator Interviews: Information Needs

Why this research matters:

• Documentation has been consistently identified as an issue by operators during the user survey. However, we wanted to understand the entire workflow associated with identifying and consuming information to resolve issues associated with operating production clouds.

Study design:

• This research consisted of interviews with seven cloud operators from different industries with varying levels of experience to determine how they find solutions to problems that arise.
• What do you try first? What is your last resort?
• How is sharing information handled?
• What would make it easier for you to solve issues?

Cloud Operator Interviews: Information Needs

Findings:

• Most problems are solved through personal knowledge: Operators typically already have information in their heads to solve problems and only need to consult outside sources 20 percent of the time. Cheat sheets and quick references are rarely used.
• Advanced problems required expert advice: Obtaining a different perspective from an expert coworker or “guru” typically within the organization is a common approach for solving challenging problems.
• Internal sharing of solution information is common: Information shared among team members or internal personnel is more common than community sharing. Externally, information sharing across the OpenStack community takes place on mailing lists or Internet Relay Chat (IRC) channels. Slack is sometimes used for communicating with customers during a problem-solving session.
• Notes embedded in code are not particularly popular: The time required to scan and assess problem-solving notes embedded in the code makes this a method of last resort, because it is sometimes prone to errors due to differences in interpretation.
• More detailed error messages would be helpful: Operators felt that including rich, detailed information in error message text would be helpful in solving problems or scoping issues.

​

​

​

​

​

Cloud Operator Interviews: Information Needs

Recommendations:

• Improve logging: Current log content doesn’t help operators sort issues very well, nor does it easily identify the root cause of an error. For example, a log entry may display an error for Keystone, the OpenStack identity service, even though the error originated in another service.
• Improve the quality and findability of the content: Rework the documentation, blogs, support sites, and other resources, and make information more accessible through Google and other search engines.
• Create a knowledge base of problems with suggested solutions: A knowledge base could help operators learn about issues and gain knowledge before a similar problem is encountered in their environment.
• Consolidate best practices for key tasks: Given that there are often multiple ways of solving a problem, a set of best practices could enhance performance of common tasks for typical environments.

​

Infrastructure Architect Information Needs

�October 2016

​

Infrastructure Architect Interviews: Information Needs

Why this research matters:

• Cloud infrastructure architects investigate different cloud solutions on behalf of the their enterprise and make recommendations on which solution to adopt. Like operators, they have a complex process for synthesizing information, and this research expands upon that theme with a different user persona.

Study design:

• This research consisted of interviews with five cloud architects, spanning from consultant to team manager roles.
• What does the process for recommending a cloud platform look like for you?
• What information do you need at each step, and how do you acquire it?
• What would make your role as a cloud architect easier?

​

Infrastructure Architect Interviews: Information Needs

Findings:

• Cloud architects rely mostly on tribal cloud knowledge to weigh pros and cons for a platform, given a set of requirements.�
• Basic cloud platform information is available and easily found; the main focus is on determining needed capabilities and understanding the environment.
• “Could X platform work if we want to do Y?”
• “Does an elegant solution missing features outweigh a difficult one that meets all our needs?”�
• Often architects “research as they go” – scoping different cloud solutions as they learn more
• A “waterfall” approach (complete all research, then start forming a solution) can be time consuming
• Talking to a variety of stakeholders is important to capture their needs and perspectives�
• From an information standpoint, the most desirable resources were case studies or opinion pieces from other architects on their experience considering cloud platforms.

​

​

​

Infrastructure Architect Interviews: Information Needs

Recommendations:

Communal learning approaches may best improve access and use of information for cloud architects:

• Access to opinions and recommendations from other cloud architects
• Knowing what kinds of problems they were trying to solve, what approaches they tried, and whether or not they were successful
• This information could take the form of a blog or digest�
• Written use cases may also be helpful for this type of information
• For example, information about company field, size, and type of solution arranged by platform could be framed as case studies for cloud architects�
• More domain-specific knowledge would also improve their ability to understand nuances between cloud solutions
• Training and/or certificates geared toward architects (not just operations-focused training)
• More information on commercial apps and other ways users are consuming cloud resources in various contexts

Application Developer Usability: Initial Experience Using Horizon

�July 2016

​

Application Developer Usability: Initial Experience Using Horizon

Why this research matters:

• The ability to learn and use a system is particularly critical for individuals experiencing it for the first time. To complement classroom training lectures, individuals received a hands-on experience using Horizon (the OpenStack user portal) to simulate the frequent task of launching an instance.

Study design:

• This study brought together 17 students through the OpenStack Innovation Center (OSIC) in San Antonio, TX, to perform a set of tasks and evaluate their experience in accomplishing their goals.
• Were you able to complete the task?
• How easy or hard was it to complete?
• What are your likes and dislikes about your experience today?

​

Application Developer Usability: Initial Experience using Horizon

Findings:

• Metadata difficulties: A number of participants found that the process for selecting available metadata was cumbersome and confusing.
• NIC order task not intuitive: Participants could not immediately figure out how to reorder the NIC items by dragging and dropping them.
• Image and volume selection improvements: Some participants had trouble determining how to delete storage at the same time an instance is deleted, or if selected volumes were persistent by default.
• User interface enhancements: Many of the tasks could be simplified by changing the way items are presented on the user interface.

​

Application Developer Usability: Initial Experience using Horizon

Recommendations:

• Clarify the metadata task: Repositioning the search bar to switch places with the custom bar, so that the filters are closer to the selection items, would clarify the process.
• Increase system feedback: Providing a more graphical representation of instances and subnets would help users visualize the entire network.
• Enhance some areas of the interface: Confusion over some of the fundamental tasks could be minimized by fairly minor changes to the user interface.

​

​

Usability Evaluation: Searchlight for Horizon

�October 2016

​

Usability Evaluation: Searchlight for Horizon

​

Why this research matters:

• The Searchlight plug-in for Horizon aims to provide a consistent search API across OpenStack resources. To validate its suitability and ease of use, we evaluated it with cloud operators who use Horizon in their role.
• This research allows the Searchlight team to make changes to improve its performance before use of the plug-in �is widespread.

Study design:

• Five operators performed tasks that explored Searchlight’s filters, full-text capability, and multi-term search.
• If you wanted to know what images required at least 2048 RAM, what would you do?
• How would you go about displaying all cloud resources associated with the IP address 172.24.4.11?
• How would you find everything associated with BOTH of the following terms: "web" and "python?"

​

Usability Evaluation: Searchlight for Horizon

​

Findings:

• Searchlight was generally seen as an improvement over the old Horizon search method�
• Participants especially liked the full-text search, which did not require the use of a filter as previous iterations of Horizon search�
• All participants felt comfortable using Searchlight after about 10-15 minutes of focused use
• It was considered relatively easy to use after a few “trial-and-error” searches�
• Many were accustomed to the CLI and preferred to use keyboard input over the mouse
• Some actions required the use of the mouse, which wasn’t a dealbreaker but slowed them down��

​

​

Usability Evaluation: Searchlight for Horizon

​

Recommendations:

• Remain inside the “Search” heading of Horizon while exploring resource information
• Abruptly changing the user’s current location within the tool can disrupt their mental model of the navigation�
• Highlight matching search terms and surface data that may be “hidden” layers deep
• This may provide greater transparency on why items are returned for a search and help users locate the target information faster�
• Communicate the possibility of full-text search
• Previously impossible in Horizon, users may assume they are required to select a filter to use Searchlight
• To reduce this false assumption, consider changing the text from “Click for Filters” to “Filter or Full-Text Search”�
• Increase visibility of search syntax help
• The small icon was not easily noticed, consider enlarging it or repositioning near the settings icon

​

​

​

Cloud Operator Interviews: Managing Quotas at Scale

�October 2016

​

Cloud Operator Interviews: Managing Quotas at Scale

​

Why this research matters:

• The study was initiated following operator feedback identifying quotas as a challenge to manage at scale.

“To prevent system capacities from being exhausted without notification, you can set up quotas. Quotas are operational limits. For example, the number of gigabytes allowed for each tenant can be controlled so that cloud resources are optimized. Quotas can be enforced at both the tenant (or project) and the tenant-user level.”�( http://docs.openstack.org/admin-guide/dashboard-set-quotas.html)

Study design:

• One-on-one interviews with cloud operators sought to understand their methods for managing quotas at �production scale.
• How do you set up quotas for a new project? What if you need to modify them?
• What’s the hardest thing about quota management?
• What could be changed about quotas to make your work easier as an operator?

​

Cloud Operator Interviews: Managing Quotas at Scale

​

Findings:

Study participants approached quota management and handling in several different ways and had ideas for �improvements, including:

• Automate processes: Quota management typically involves many tedious tasks. Participants suggested automating these kinds of processes to free up time for dealing with more complex issues.
• A few participants manage quotas entirely manually, and saw benefit in having automated tools for certain quota allocation and modification tasks�
• Streamline user requests: Additional details provided by users issuing quota requests make it easier to promptly evaluate and approve appropriate requests.
• Although generally not seen as optimal solutions, ticketing systems were commonly implemented for user-generated quota requests�

​

​

Cloud Operator Interviews: Managing Quotas at Scale

​

Recommendations:

• Centralize quotas: Instead of forcing operators to set quotas across project APIs, centralize quotas. This theme was also identified in the OpenStack Foundation’s User Survey in which increasing consistency across projects was seen as a primary goal.
• Single point of view: Create a single point view of project quotas, clearly indicating the status of each.
• Self-configuration and delegation: Let operators configure resource allocations for individual domains. Also, let domain administrators delegate quotas across their range of projects.

​

​

​

Cloud Operator Interviews: OpenStackClient Workflows

�May 2016

​

Cloud Operator Interviews: OpenStackClient Workflows

Why this research matters:

• Consistency across projects has been identified as an issue in the user survey:
• “It really boils down to having openstack act more like a single initiative vs's a collection of projects.”

​

Study design:

• This usability study, conducted at the OpenStack Austin Summit, observed 10 operators as they attempted to perform standard tasks in the OpenStack client.

​

Cloud Operator Interviews: OpenStackClient Workflows

Findings:

• Quick recognition of value: Operators quickly recognized the benefits of the OpenStackClient.
• Lack of commands: A potential adoption barrier was identified: the lack of commands included in the OpenStackClient.
• Additional help: Operators expressed a need for additional help. “Help” was the third term used in the open-ended questions (behind “command” and “user”). The need for help was also noted in the OpenStack Foundation’s User Survey as a high-priority item.

Cloud Operator Interviews: OpenStackClient Workflows

Recommendations:

• Increase in commands: Expand the number of commands available in the OpenStackClient to further drive adoption and better support the services in use.
• Improved help: Improve the help to make it easier to find detailed help information and search various topics.
• Better explanations: Explain the relationships more effectively between components, such as projects, domains, users, roles, and so on.

​

​

​

Cloud Operator Interviews: OpenStackClient Workflows

�October 2016

​

Cloud Operator Interviews: OpenStackClient Workflows

Why this research matters:

• Consistency across projects has been identified as an issue in the user survey:
• “It really boils down to having openstack act more like a single initiative vs's a collection of projects.”

​

Study design:

• This usability study, conducted at the OpenStack Barcelona Summit, observed 9 operators as they attempted to perform standard tasks in the OpenStack client.

​

Cloud Operator Interviews: OpenStackClient Workflows

Findings:

• Learning Curve: There was a learning curve where operators needed to become familiar with the commands and structure. The learning curve flattened out significantly after participants learned the combined commands and structure
• Help: Operators didn’t search help so they receive far too much content within the command prompt
• Value: All operators clearly saw value and answered that they anticipated using the OpenStackClient based on their experience from the study

​

Cloud Operator Interviews: OpenStackClient Workflows

Recommendations:

• Quick Start: Add an operator “quick start” to the OSC docs page. Should be immediately discoverable and include use case-based examples to tie commands together into a single code snippet
• Client Help: Improve discoverability of help within the client. Focus should be on teaching operators how to search help so they receive feedback for specific features
• Increase development on the OSC: This is based on the overwhelming number of operators that anticipated using the client following the study

​

​

​

Cloud Operator Interviews: Deployment

�December 2016

​

Cloud Operator Interviews: OpenStackClient Workflows

Why this research matters:

• Deployment has been consistently identified as an issue by operators during the user survey and impacts both adoption and operations of OpenStack. We wanted to do a deep dive with operators to identify the specific issues impacting deployment.
• “I feel difficulties like deployment of OpenStack at a very large level are still not so easy; the product is not very stable, and migration of whole infrastructure with a new release of OpenStack is still bit challenging.”

Study design:

• Areas of discussion included organizations, tools, workflows and pain points associated with deploying OpenStack
• Research areas were selected based on a combination of issues identified in the user survey, feedback from operators and the project teams
• 1:1 interviews
• Participants included the Rey the Cloud Operator persona
• Notes were taken by the team in an etherpad and analyzed using MaxQDA

​

Cloud Operator Interviews: OpenStackClient Workflows

Findings:

• Relevant information was difficult to find or incomplete: This is not exclusive to docs and can include content such as blogs or chat rooms. Poor documentation is an overall theme in the community
• Ability to triage using the logs was difficult: There was generally too much or too little information. Error codes provide very little detail.
• Teams tended to use what they know: Teams discussed moving to other tools, but very few invested the time to do so
• Teams tended to hire adaptable members with ability to learn: Only one participant mentioned hiring with OpenStack-specific skills

Cloud Operator Interviews: OpenStackClient Workflows

Recommendations:

• Improve logging: Again, not exclusive to deployment
• Improve documentation: Optimize for Google searching and update content
• Easy/Customisable: Deployments should be simple to use, but allow for customization

​

​

​

OpenStack Personas

�(ongoing)

OpenStack Personas

Why the personas matter:

• In order to share the knowledge about the target users, we have created these representations of our key audience segments based on qualitative and quantitative user research. The goals are to create more empathy for our customers and to better display the different types of customers performing different jobs.

Study design:

• The personas were created from a series of three working sessions in Austin and Seattle. The contributors included operators, product managers and user researchers from companies like BestBuy, IBM, HP, Intel and Rackspace.

Role Ecosystem

OpenStack Personas

Model Companies

OpenStack GUI Guidelines

�(ongoing)

GUI Guidelines

Why the guidelines matter:

• The are a number of OpenStack projects that have GUIs including Horizon, Irony and Fuel. The GUI guideline is intended to help:
• Drive a consistent interaction models across OpenStack projects in order to avoid forcing users to relearn an interface.
• Create a family look and feel for OpenStack projects
• Prevent developers from having to spend time creating a new patterns for each project

​

GUI Guidelines

GUI guidelines

Additional Thoughts

�

Consumers and upstream developers value open source for different reasons

• Operators commented that they value open source because it avoids vendor lock-in, allows them to impact product direction and provides access to code
• Upstream contributors view open source as an opportunity to contribute to developer-driven communities
• It’s not entirely clear whether the community of upstream developers always understand this distinction.