Moblab Introduction & Use Manual

Rev: 1.0 (20180425)

Status: REVIEWED / PUBLISHED

Forum: Moblab-discuss

Moblab release note: https://sites.google.com/a/chromium.org/dev/chromium-os/testing/moblab/releasenotes


Introducing - Moblab

What is a Moblab?

Why do I need to setup a Moblab testing environment in my company?

What are the benefits of having a Moblab on site?

Requirements

1. What are the hardware requirements for setting up a Moblab?

2. Now that I have all the required hardwares, how do I connect them?

3. How to configure my network to accommodate Moblab?

Setting up the Moblab

1. Boot up

2. Find Moblab IP Address

3. Config Moblab

4. Your Moblab is ready!

Setting up DUT

1. Install test image

2. Connect to Moblab

3. Configure DUT on Moblab

Using Moblab

How to use Moblab for testing by using Run Suite

Use Wmatrix to view test results (except CTS)

Advanced Moblab Tutorials

a. Running tests from the AFE

b. Staging Custom Images & Client-side Tests on Moblab

c. Custom Client-Side Tests Support

d. Advanced Tasks

e. Advanced Commands

f. Information on how Moblab works

g. MobMonitor

Introduction

How to use the Mob Monitor

h. Setting up SSH Access from a Remote Machine

FAQ

Moblab Setup

How do I join the Moblab mailing list?

How do I report issues of Moblab ?

Can I access the Moblab server from another machine on the network?

Can I access the Moblab server from a DUT (Device Under Test)?

Where can I download the latest Moblab recovery image?

How do I immaculate Moblab to restart from the beginning?

How do I access the Google cloud storage bucket from a web browser?

I reboot my Moblab, and now I can't get to the Moblab AFE from a browser, and looks like Apache is not up, what is happening?

Mob*Monitor says "Moblab has less than 10 percent free disk space. Please free space to ensure proper functioning of your Moblab." what should I do?

Why does my Moblab run out of space so often?

Getting "The device you inserted does not contain Chrome OS" message when trying to use recovery image on Moblab.

Running tests

Can I mix and match hardware platforms for hosts on the test subnet?

My DUT kept on beeping during startup (when auto start from "verification is off" screen), it is really annoying, can I silence the DUT?

I've kicked off a suite of tests. I'm not sure if anything is happening? How do I know something is running?

Introducing - Moblab

What is a Moblab?

Moblab is a test system in a box: a customized Chrome OS image loaded onto a Chromebox. The Moblab tests are the same tests that the Chrome OS team at Google runs in the Chrome OS lab.

So, what are the tests?

When qualifying a new board for release, the test engineering team for Chrome OS performs the following tasks:

Why do I need to setup a Moblab testing environment in my company?

These tests if done manually take multiple days to complete. To accelerate board bring-up and ensure quality, the Chrome OS team is in the process of automating the tests so that partners can run them on site.

What are the benefits of having a Moblab on site?

Moblab requires virtually no administrative overhead.

Storage, network communication, and software updates are all managed automatically. The latest software is used for testing, and versions of the operating system and test suites are always in sync.

Any Autotest suite or individual test case can be run on a single box.

This includes test suites such as build validation tests (BVT), hardware component qualification tests, and custom automated tests*. Moblab can run any test written using the Autotest framework.

*Depending on the type of component being tested, additional equipment may be required. For example, a WiFi signal is required for wireless connectivity tests.

The Autotest framework enables running across multiple DUT (Device Under Test) and scheduling tests. Simpler tools and test frameworks only support manually launching tests, or running them on a single DUT. With Autotest, you can also define pools of multiple DUTs, to group DUT and set scheduling priorities.

Moblab also supports different pool of DUT for testing simultaneously by using pool labels, You can run storage component qualification for first DUT pool and memory component qualification for second DUT pool by using one Moblab.

Moblab provides a common platform for test infrastructure. Using the same hardware, software, and test builds makes communication between partners and Google more efficient. When issues are found, having a common platform makes it much quicker and easier to reproduce them.

Community support provides greater security and faster response time for questions and issues. Source files are available for inspection and customization, and partner contributions are welcome. The only feature unavailable to custom builds of Moblab is automatic server updates.

Requirements

1. What are the hardware requirements for setting up a Moblab?

Item

Hardware

Specification

Minimum Requirement

Mandatory for CTS

Notes

1

Chromebox

Google Chromebox(Wukong) with Core i7, 32GB RAM, 512GB NVMe SSD And dual Network Ports.

1

1

2

DUT (Device Under Test)

1

20

  • CTS requires 20 DUTs to have best efficiency
  • AVL DUT number depends on components, please contact Google AVL POC  

3

Switching Hub (optional for multiple DUTs)

NETGEAR ProSAFE 24-Port Fast Ethernet Switch

1

Do NOT use managing hubs, switches and routers. Size the switch appropriately for the number of DUT’s you have attached, for your particular test application.

4

Ethernet Cable

multiple

multiple

5

Keyboard

1

1

6

Mouse

1

1

7

Monitor

with HDMI/DisplayPort supported

1

1

8

Network requirement

100MBps Upload / 100MBps Download

  • Proxy is NOT support, please keep direct link to internet

** Please contact Synnex for Moblab procurement, other accessories can be sourced from local market

Synnex Technology International Corp.

Sanny Chiu

E-mail: sannychiu@synnex.com.tw

Tel : 886-2-2506-3320  Ext  2033

2. Now that I have all the required hardware, how do I connect them?

Here are the simple diagrams that show you how to connect the devices, depending on the number of DUTs are being tested.

Please note the lower ethernet port on Moblab should connects to DUT. See physical setup picture below.

It does not matter which network port you use on moblab to connect to the internet or switch, but the connection to the internet should be active when you boot the moblab as it uses the internet connection to calculate the network configuration.

One DUT: 

Multiple DUTs:

Physical Setup:

*Important*

3. How to configure my network to accommodate Moblab?

This diagram shows a typical configuration of Moblab on a partner’s network:

Setting up the Moblab

1. Boot up

STEP 1 - Power on Moblab

STEP 2 - Make sure Moblab is in Developer mode

you should see booting screen like this

STEP 3 - At welcome screen, please select Ethernet as your network

2. Find Moblab IP Address

Note: Moblab IP will only be used when you want access Moblab remotely.

STEP 1 - Login with any account or simply browse as guest

STEP 2 - Enter VT-2 and login as moblab 

you should not be prompted for password.

Ctrl-Alt-F2

 localhost Login: moblab  

STEP 3 - Check network configurations

 moblab@localhost ~$ ifconfig

eth0 : IP address from your internet network (upper network port)

eth1 : blank, network info for the lower network port

*Note: No meter which network port connects to the internet, it will be assigned as eth0

lo: localhost of Moblab

lxcbr0: IP address (inet) = ***.***.***.*** (this is the DHCP service of Moblab)

STEP 4 - Switch back to Chrome OS UI and browse any web page on internet to make sure internet works.

Ctrl-Alt-F1

3. Config Moblab

Before starting this section, you should have:

STEP 1 - Open browser, enter localhost/ or IP of your Moblab

STEP 2 - Click Moblab on top-right corner to enter setup page

STEP 3 - Click Configure to start

Moblab will show its IP as server IP and verify internet connections, if not, click Query to refresh

STEP 4 - Click Next when confirmed.

Please note Moblab doesn't support connections via proxy, please consult with your IT support for solution

STEP 5 - Enter your boto key info

STEP 6 - Enter image storage bucket URL

the format is gs://[YOUR_BUCKET_URL]/

STEP 7 - Click Finish then OK to commit all changes.

Moblab will restart some services in background and show all updated info

4. Your Moblab is ready!

You don't need to change anything in Config Settings or Launch Control Key unless you receive instructions from your Google contact.

Setting up DUT

1. Install test image

STEP 1 - Install a test image on DUTs

Installation

Test Images

STEP 2 - Switch to developer mode after re-image completed

2. Connect to Moblab

STEP 1 - Connect DUT to Moblab by using USB-Ethernet adapter.

Please refer diagram above to connect 1 or more DUTs to Moblab

STEP 2 - Boot up DUT

STEP 3 - Enter VT-2 (Ctrl-Alt-F2) and login as root

 Login: root

 Password: test0000  

STEP 4 - Get the IP address of DUT, if you have multiple DUTs, you'll need to repeat the steps in this section to get all the IP addresses of the DUTs

Attention: IP & MAC address of eth0 on DUT based on USB-Ethernet adapter, swapping adapter may cause unexpected error

3. Configure DUT on Moblab

STEP 1 - Open browser, enter localhost/ or IP of your Moblab, click Moblab on top-right corner

STEP 2 - Click Manage DUT, you should see DUT listed in the page, IP & MAC address should match the info you found from DUT setup

(Always check IP address, MAC address and labels to ensure you're adding right DUT for autotests)

STEP 3 - Select DUTs and Add Selected DUT's then click Apply, Moblab will add DUT for autotest.

Click Refresh to get DUT labels

If you want to add more labels for DUT, select it again then scroll action column to "Add Labels Selected DUT's", add labels splitted by comma, then click Apply

Checking “Is pool label” will add pool label to selected DUT(s). Pool label can help you to group up different DUTs easily.

Using Moblab

How to use Moblab for testing by using Run Suite

STEP 1 - Open browser, enter localhost /or IP of your Moblab, click Moblab on top-right corner

STEP 2 - Click Run Suite

STEP 3 - Select the board that you are testing

( name of the board that's connected to Moblab with your DUT should appear here )

STEP 4 - Select the build that you are going to test with

STEP 5 - Select the suite

Note that you can also have the option to run specific test jobs within the suite package.

Ex. arm.android.bluetooth, arm.android.app.usage

STEP 6 [Optional] - Select only when you have multiple DUT pools

STEP 7 - Click Run Suite

All test jobs can be found at Moblab front page

Use Wmatrix to view test results (except CTS)

WMatrix present a better test result viewing UI for Moblab users.

The Test Results UI might look a little overwhelming. It includes all tests that currently existing in Chrome OS autotest.git.  But might not be relevant to Partners initially. We recommend to start with 'Suite List" in the "Quick links" section.

Depends on the type of test suites you have run on your Moblab unit, the type of suite in the subsequent screens may differ. You may have faft, storage qual, bvt show up.  Here is one example from our faft tests. From here you could drill down further to see aggregated test results.

Aggregated test results of one particular suites across build and devices. Here is a simple example of one device and one build test results. If you have your mouse hover over the "red" section on the right hand side, you will see all the failed tests associated with runs on that build (R52-8350.69.0). Clicking on the red "1" in the screen below will lead you to the detailed failure screen so you can investigate further.

NOTE: If you are running a build that is old, you might see an empty UI when you click on the Suite Name from screen shown earlier. One work around is to add "&days_back=15" (or whatever days you know that are required to get to your build comparing to today's build) to see the result. We are aware of this flaw and working on a fix for next release.


Advanced Moblab Tutorials

a. Running tests from the AFE

We encourage you use Moblab Admin to run individual test and test suites, if you need more customized configurations for your test, here is the instruction of using AFE running tests. Please only use Moblab Admin for CTS.

Additional information on Autotest: For descriptions of each parameter in the Create Job view or details of how the Autotest scheduler runs jobs, click here.

Running your test from the AFE

STEP 1 - Click the tab to open the Create Job view.

STEP 2 - Type a name for the job. The string should be meaningful when viewing a list of jobs.

STEP 3 -Type the name of the test image to use in the Image URL/Build field.  If no image is specified and you are not running a suite job, the latest ToT image is used. For some  tests Image URL/Build is a required field. In such cases add the release name here in the following format: <board>-release/RXX-YYYY.ZZ.0 eg. samus-release/R42-6812.88.0.

STEP 4 - Select the tests to run.

Advanced Options can be adjusted for your tests

If you select a suite, the hostless (suite job) attribute will automatically be selected in the “Advanced Options” section.

If you want to run a suite of tests on a pool of hosts you have defined in “Manage DUT” section, specify the pool_name in the Pool: (optional) field.

STEP 5 - Click the names of registered hosts for running the tests.

STEP 6 - Click Submit Job.

b. Staging Custom Images & Client-side Tests on Moblab

In order to test a locally built test image you must first build one in the chroot. Instructions for this can be found in the Chromium OS developer guide.

Notes

Staging a custom image on a local Moblab

To stage on a local Moblab that is accessible by your chroot you simply use the ‘cros stage’ command:

cros stage </path/to/board/build/chromiumos-test-image.bin> <Moblab IP address>

Example:

In this example I am pushing an image I built in my chroot to the Moblab device.

~/trunk/src/scripts $ cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin 100.XX.XX.XX

The path requires the format of /board/build/chromiumos-test-image.bin.

Once completed ‘cros stage’ will output the name of the image to use when kicking off tests via the run_suite command or the AFE. For this example: peppy-custom/R41-6576.0.2014_12_12_1535

Notes:

Staging an official image with custom tests on a local Moblab

You can also stage official images from Google Storage with test packages from your chroot onto a local Moblab.  The command, cros stage always pulls test packages from the chroot regardless of the image source.

For this workflow, you must have previously built an image for this board at some point. For information about doing custom test development please refer to the Advanced:  Custom Client-Side Tests Support.

cros stage gs://<bucket-name>/<board>-<builder type>/<build name>/ <Moblab IP address>

Example:

In this example I am pushing an official image with test packages built in my chroot to the Moblab device.  I would have already built a squawks test image at some point.

~/trunk/src/scripts $ cros stage gs://chromeos-image-archive/squawks-release/R42-6750.0.0 100.XX.XX.XX

/trunk/src/scripts $ cros stage gs://chromeos-image-archive/squawks-release/R42-6750.0.0 100.XX.XX.XX

Once completed ‘cros stage’ will output the name of the image to use when kicking off tests. For this example: squawks-custom/R42-6570.0.0

Staging a custom image to Google Storage

You can also push images to Google Storage. The advantages to this is that the image will not be deleted for 24 hours, and that remote Moblab’s not in the same local network can access the image as well.

cros stage  </path/to/board/build/chromiumos-test-image.bin> <gs_base_path>    --boto_file=<boto_file_path>

Example:

cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin gs://chromeos-moblab-peppy --boto_file=~/peppy.boto

Once completed ‘cros stage’ will output the name of the image to use when kicking off tests just like when staging on a local Moblab.

Testing the custom image

Specify the outputted image name when launching the test:

c. Custom Client-Side Tests Support

NOTE: Custom Client-Side Tests Workflow is only supported when using cros stage with a Moblab address and NOT when using an Google Storage URL as the staging destination.

Cros Stage also enables developer to run Custom Client Side Tests on their Moblab. It does this by packaging up the tests that are built and stored in the board’s buildroot (/build/<board>/) of the chroot.

  1. First make your test changes in the autotest repo (src/third_party/autotest/files).
  2. If this is a new test, add it to an ebuild to ensure it is built at build time.
  1. If this is an existing test, note the ebuild it belongs to.
  1. Build your test changes. From a chroot:
  1. Run cros stage from the chroot to stage your image and test packages.

Note you do not need to build a new image. Therefore you can use the same image repeatedly while doing test development.

For more information:

d. Advanced Tasks

Using command line tools to run tests

You can also use the Autotest command line interface (CLI) to run manual tests. The following Python utilities are available to Moblab users for running tests:

For a more extensive list, click here.

To run a test or test suite from the command line, use the following syntax:

run_suite.py --board=<board_name> --build=<build_name> --suite_name=<suite_name> --pool=””

Example:

run_suite.py --board=peppy --build=peppy-release/R37.5847.0.0 --suite_name=smoke --pool=””

To obtain the names of test builds, use gsutil to browse your Google Cloud Storage bucket.

You can check job status and results through the AFE. This page describes the status values for hosts and jobs.

Writing Custom Tests

The Moblab server runs the open source git Autotest framework. The framework supports writing custom tests in Python using the Autotest APIs.

For details and examples of coding to the Autotest APIs, view the general user documentation.  

Please note that although Moblab uses the Autotest framework, code refactoring has made it so that many examples do not work with our code base. However, many of these documents are still helpful at a high level.  

Chromium OS-specific information can be found here and our repo is located here.

Booting from a Recovery Image

If the Moblab server hangs and the device can’t be manually rebooted, it is possible to boot from a recovery image. For Chrome OS partners who have deployed Moblab, the recovery image is located in a folder in Google Cloud Storage. It is not generally available on the Chrome OS image release site.  

To boot Moblab from a recovery image:

  1. Copy the Moblab recovery image to a USB memory stick using cros flash:  

cros flash usb:// remote/stumpy_moblab/latest-dev/recovery

  1. This example copies the latest recovery image in the dev channel to a USB drive.
  2. From the Moblab device (in Developer Mode), press Ctrl-Alt-F2 to switch to the second virtual terminal.
  3. Run crossystem dev_boot_usb=1
  4. Reboot the Moblab device.
  5. Insert the USB memory stick with the recovery image into an open USB port on the Moblab device.
  6. When Chrome OS displays the Developer Mode screen, press Ctrl-U to boot from a USB device

e. Advanced Commands

These commands have been added to the Moblab user profile so you can run them anywhere on the system. Please note that this list is valid as of version 6343.

Note: These custom tests in these directories will be overwritten if Moblab is power washed or if a new test with the same name is checked into the source.

f. Information on how Moblab works

Bringing up the test environment

After Moblab is connected to a network and booted, the Moblab server performs the following network configuration tasks:

  1. Connects to the test subnet through lower network port.
  2. Requests subnet addresses 192.168.231.x from the DHCP server.
  3. Configures a static IP address for the Moblab server of 192.168.231.1 through lower network port.
  4. Connects to the corporate network through the on-device Ethernet port.
  5. Obtains an IP address from the DHCP server on the corporate network. This is the external IP address of the Moblab server.
  6. Configures a gateway service for host Internet access.
  7. Enables IP forwarding.
  8. Adds a NAT module to the Chrome OS kernel.
  9. Enables automatic updates of Chrome OS and the test suite.
  10. Uses SSH to configure the server settings.

One or more hosts are connected to the test subnet by either a switch or a hub. Moblab communicates with the test subnet using the lower network port. When the Moblab server boots up, it waits up to 5 minutes to establish an SSH connection through the lower network port and subnet. Then it assigns dynamic IP addresses to all hosts, ranging from 192.168.231.100 to 192.168.231.199.

Access to the partner’s corporate network enables automatic software updates for the Moblab server. It also enables access to Google Cloud Storage for downloading test images and server updates, and uploading logs. The corporate network should provide security features (firewall, VPN, etc.) to protect Moblab and the test subnet from potential attacks from outside the lab.

Automatic updates

The Autotest scheduler manages and coordinates build updates. On each Moblab device, it manages two separate types of build images:

The scheduler waits for a time of low or no activity on the Moblab server to update the test image. If tests are running when an image is updated, they continue to run.

g. MobMonitor

Introduction

The Mob Monitor is a tool available on Moblab devices that provides:

Using the Mob Monitor can help you to:

How to use the Mob Monitor

To access the Mob Monitor web interface, go to http://localhost:9991 on your Moblab (even if your Apache server is not up, e.g. when http://localhost is not functioning, you can still use this address to get to Mob Monitor)

The web interface presents all health information in single table. Each compartment in the table describes the state of a single service running on Moblab. In the above image, we see health information for the following illustrative services:

A service may be described by the monitor as being:

The Mob Monitor colorizes each service based on their health status.

When a service is Quasi-Healthy or Unhealthy the Mob Monitor will display Warnings and Errors for that service.

Looking at the above image, we see that:

  1. If no repair actions are listed, follow the instructions given in the issue description.
  2. If repair actions are given, click to run!

Most repair actions will require no user input, you simply have to click the button and wait ~10-15 seconds for the Mob Monitor to execute the action in the background and for the web interface to refresh.

Some repair actions do require input. If that is the case, a dialog will pop up when the run button is clicked.

The dialog will present the following information:

In the event that something terrible happens with your Moblab device (related to the Mob Monitor or not!), the Mob Monitor provides a Collect Logs button in the upper right corner of the UI. Clicking this button will open a separate tab that will provide a package of relevant system logs for download to your local machine.

Note, this operation may be slow if there are many logs on the Moblab.

Some things to keep in mind when using the Mob Monitor:

h. Setting up SSH Access from a Remote Machine

This process will walk you through how to set up ssh access into Moblab.

  1. From a remote machine with access to the Chrome OS source tree, download the ssh keys from the Chromium git repository at the following link:  https://chromium.googlesource.com/chromiumos/chromite/+/master/ssh_keys
  2. Copy both keys into ~/.ssh/ on the remote machine where you want to enable SSH access to the Moblab server.
  3. Edit the SSH configuration file  (~/.ssh/config) on the remote device as follows:
    Host <moblab_server_IPaddress>
    CheckHostIP no
    StrictHostKeyChecking no
    IdentityFile %d/.ssh/testing_rsa
    Protocol 2
  4. Save the file.
  5. Test ssh access using the following commands: ssh moblab@<moblab_server_IPaddress>

FAQ

Moblab Setup

How do I join the Moblab mailing list?

Moblab has a public discussion group, moblab-discuss@chromium.org. Please use this link to join this public group.

How do I report issues of Moblab ?

Before filing an issue, please confirm you’ve completed actions below:

Please have all screenshots ready and rename to descriptions in bold.

Use Partner Issue Tracker and select Moblab template to report issue.

Can I access the Moblab server from another machine on the network?

Yes, you can remote access your Moblab by using browser and Moblab IP you get during setup.

Can I access the Moblab server from a DUT (Device Under Test)?

Yes, you can access Moblab from DUT by using browser, however, this is only for debugging, you should never run any test on Moblab from DUT.

Where can I download the latest Moblab recovery image?

Moblab recovery images are publicly shared at gs://chromeos-moblab-public/moblab-images. Please refer to the release notes to find out the latest release.

How do I immaculate Moblab to restart from the beginning?

Please follow this instruction to powerwash your Moblab.

Attention:  If you’re trying to immaculate Moblab by re-image, you should always use Moblab recovery image, NOT Chromebox recovery image.

I reboot my Moblab, and now I can't get to the Moblab AFE from a browser, and looks like Apache is not up, what is happening?

Mob*Monitor says "Moblab has less than 10 percent free disk space. Please free space to ensure proper functioning of your Moblab." what should I do?

Why does my Moblab run out of space so often?

Moblab offloads test results every 24 hours to google cloud storage. If your network connectivity is down, the results will accumulate and if you are running long running tests, the results will build up and eat up your storage space.

Can I purchase Chromebox from somewhere and build Moblab by myself ?

No, Moblab (Wukong) was a customized Chromebox with more powerful computing power and some enhancements on its hardware, we don’t support any self-build Moblab and you should only purchase them from designated channel.

Can I setup Moblab testing environment out of Taiwan or US ?

Yes, as long as the network environment is stable and unblocked, Moblab should work in anywhere.

Please note, we don’t support any Moblab setup in China due to uncertainty network issues.  

Running tests

Can I mix and match hardware platforms for hosts on the test subnet?

Yes, you may connect different DUTs to one Moblab unit. When running hostless tests (all "suites" are run hostlessly), during "Create Job" step, if you specify "Image URL/Build" by pointing to a particular type of image for a particular type of DUT, Moblab will automatically only push subtests to corresponding DUT based on "Image URL/Build" type.

My DUT kept on beeping during startup (when auto start from "verification is off" screen), it is really annoying, can I silence the DUT?

Yes, go to VT2 (ctrl+alt+F2), log in as root, and type the following in the command line will silence the DUT:

$> /usr/share/vboot/bin/set_gbb_flags.sh 0x19

I've kicked off a suite of tests. I'm not sure if anything is happening? How do I know something is running?

Go to the Job List page in AFE and click on Running Jobs. This will give you a list of jobs that are currently running.