How to Configure

Your Cloud and Tempest

for

Interoperability Testing

Catherine Diep

Solutions Architect & Performance Engineer

IBM

Chris Hoge

Interop Engineer

OpenStack Foundation

May 19, 2015

OpenStack Summit, Vancouver

Why Test?

• How do you know your cloud works as expected?

​

• Your installation is up and running, but is it healthy?

​

• Just finished an upgrade, but did anything go wrong?

​

• Do you qualify for logo usage?

Defcore, Interoperability, and You

A Committee formed during the OpenStack IceHouse Summit in Hong Kong by Board Resolution on 11/4/2013

• OpenStack DefCore Committee has established the principles and first artifacts required for vendors using the OpenStack trademark (bylaws changes passed Jan, 2015 election).

https://wiki.openstack.org/wiki/Governance/Foundation/2014ProposedBylawsAmendment

​

• DefCore sets base requirements for all OpenStack products by defining
1. Designated sections of code.
2. Must-pass tests of capabilities.

​

• These definitions use community resources and involvement to drive interoperability by creating the minimum standards for products labeled “OpenStack”

http://git.openstack.org/cgit/openstack/defcore/

Introducing Refstack

There are three parts in the Refstack project:

• refstack client is a command line tool that runs test tools (Tempest or non-Tempest), summarizes results, and communicates with the refstack api. Vendors can use refstack client to run the tests themselves in a controlled and repeatable way.
• refstack api is a central repository for the collection of results... it is currently hosted at api.refstack.net. It can also be hosted locally within any organization's infrastructure.
• refstack ui is a web interface for interacting with data collected with the api and client ( http://refstack.net ).

A source of tools for OpenStack interoperability testing. Data collected by these tools can be used to validate against the DefCore required capabilities.

Test tools installation

• Installation
• Tempest can be installed manually or via Refstack.
• Refstack provides a one-step setup command (setup_env) which takes care of installation of all dependencies including Tempest. (https://github.com/stackforge/refstack-client/blob/master/README.rst)
• By default, Refstack will install a specific Tempest SHA which was used by the DefCore committee to define the must-pass tests.
• Once refstack-client is installed, Tempest is located at <refstack-client home directory>/.tempest.

​

• Running tests
• You can run tests directly from the Tempest directory to validate/debug the tempest.conf file & test cases. The results are saved in the Tempst .testrepository directory.
• Alternatively, you can run tests using refstack-client when you are ready to collect data for submission or uploading to Refstack. Test results will be parsed to a json list of passing tests.

​

• More details of how to run tests can be found at:

https://git.openstack.org/cgit/openstack/defcore/tree/2015.04/procedure.rst

​

Cloud Asset Configuration

• User accounts
• At the minimum, create two test users from two different tenants
• For example demo1 and demo2 users and tenants.
• Images
• Two image ids are needed. They can be the same image uploaded twice. We suggest Cirros. It’s small and gives you all the functionality you need.
• Networks
• One shared network visible to all test users -or-
• One network per tenant.
• “Member” role
• Create a non-admin “Member” role if it does not exist in your cloud
• Assign this role to the test users.
• “Swift Operator” role
• If testing object storage, you’ll also need a role that can create Swift assets.

​

Tempest Configuration

• Enabled Services - What’s running in your cloud.

​

• Endpoints - Where the services are running.

​

• Credentials - Who can log into the cloud.

​

• Assets - Resources you need to verify functionality.

​

Tempest Configuration Guide

Much of this guidance comes from the docs recently added to the Tempest repository:

http://docs.openstack.org/developer/tempest/configuration.html

​

tl;dr: Three ways to configure users and assets

• Stored credentials in tempest.conf.
• Locking accounts.
• Stored credentials in tempest.conf with tenant isolation.

​

tl;dr: You can’t not read that document, and it’s not that long.

​

​

Essential tempest.conf settings

• Tempest default settings
• Tempest assigns default values to most of the configuration parameters, and shows them inline
• For example, “#resize = false” indicates that by default resize is not supported.
• If you want to run the resize related test cases, you can either edit the existing line or add a new line.

#resize = false

resize = true

• Enabling services
• Make sure to enable the service you want to test.
• For example, the default value for Neutron service is “false”.
• User accounts, for non-locking account configuration.
• Currently, at the minimum, 3 users accounts belongs to 3 tenants are needed.
• One of the users is an admin user.
• For interoperability, we want to eliminate admin-only tests.
• Images
• Two different images ID are needed for image_ref and image_ref_alt.
• Some tests will be skipped if the two image ID are identical. It’s ok to upload the same image twice.

​

​

Essential tempest.conf settings

• Networks
• Simplest network configuration is to use
• fixed_network_name = shared_private
• Log file
• Default is empty, but you want logs!
• log_file = tempest.log
• log_dir = /tmp
• Resizing (essential for interoperability testing).
• Default is false
• resize = true
• Endpoint Types - varies depending on cloud network configuration
• default is publicURL, may need to change
• for example: endpoint_type = internalURL
• Volume Device Name
• default is vdb, may need to change
• for example: volume_device_name = vdc
• Roles in Swift
• Varies by installation, defaults are Member and ResellerAdmin
• for example operator_role = swiftoperator
• and reseller_admin_role = admin

Alternative: Locking Accounts

• Set test_account_file = etc/accounts.yaml

​

• Set allow_tenant_isolation = false

​

• Configure accounts.yaml with a set of pre-created users
• For every two accounts you create, you can run one thread.
• Every user needs it own tenant, or tests may fail in strange ways.

​

• One shared network as defined in fixed_network_name, or one network per tenant as defined in accounts.yaml

​

• You actually need locking accounts if you have non-standard Swift roles.

Alternative: Locking Accounts

​

• Example yaml entry:

- username: 'testuser'

tenant_name: 'test1'

password: 'pass1'

resources:

network: 'private-net'

roles:

- '_member_'

- 'swiftoperator'

Running tests with Tempest

There are various ways to initiate a Tempest test run

• testr run

​

• ./run_tempest.sh -C <tempest conf file>
• This script is a testr wrapper that will install and use a virtualenv

​

• ./run_tempest.sh -C <tempest conf file> -- --load-list <test-list-file>
• This will run only the list of tests provided

​

• ./run_tempest.sh -C <tempest conf file> -- tempest.api
• This will run only Tempest API tests, matching on name prefix

​

Interpreting Test Results

• Output from tempest runs in .testrepository directory
• File names numeric increasing in value.
• Format called subunit, lots of tools for parsing it.
• For example, to create a csv of results:

subunit-1to2 .testrepository/6 | subunit2csv > 6.csv

​

• Check your logs for information and errors
• log_dir and log_file parameters in tempest.conf

Running tests with refstack-client

• refstack-client test

Run a full Tempest API test set with the following command:

./refstack-client test -c <path your tempest.conf file> -vv

Run only the tests listed in a file with the following command:

./refstack-client test -c <path your tempest.conf file> -vv \

-t "--load-list <your test-list file>"

​

• refstack-client upload

Once you’ve completed a test run and you are happy with the results, you can upload it to the API server.

./refstack-client upload <Path of results file>

This defaults to uploading the data to refstack.net however you can override that by using the --url flag if you want to keep track of results on your own running API server. When the results are finished uploading you will see something like this;

Test results uploaded!

URL: http://refstack.net/#/results/79e70f27-47ca-424a-a575-fd24d79686f4

​

Test results on Refstack UI

Why run Refstack?

• Simple result collection for review and sharing.

​

• Anonymous way to share the capabilities of your cloud.

​

• Helps us to discover widely used features and define what interoperability means. Run all the tempest.api tests, not just Defcore, and report to Refstack.

​

• Your results make a difference!

Thank you!

​

​

Questions?