Moblab Introduction & Use Manual
Rev: 1.0 (20180425)
Status: REVIEWED / PUBLISHED
Moblab release note: https://sites.google.com/a/chromium.org/dev/chromium-os/testing/moblab/releasenotes
Moblab is a test system in a box: a customized Chrome OS image loaded onto a Chromebox. The Moblab tests are the same tests that the Chrome OS team at Google runs in the Chrome OS lab.
So, what are the tests?
When qualifying a new board for release, the test engineering team for Chrome OS performs the following tasks:
These tests if done manually take multiple days to complete. To accelerate board bring-up and ensure quality, the Chrome OS team is in the process of automating the tests so that partners can run them on site.
Moblab requires virtually no administrative overhead.
Storage, network communication, and software updates are all managed automatically. The latest software is used for testing, and versions of the operating system and test suites are always in sync.
Any Autotest suite or individual test case can be run on a single box.
This includes test suites such as build validation tests (BVT), hardware component qualification tests, and custom automated tests*. Moblab can run any test written using the Autotest framework.
*Depending on the type of component being tested, additional equipment may be required. For example, a WiFi signal is required for wireless connectivity tests.
The Autotest framework enables running across multiple DUT (Device Under Test) and scheduling tests. Simpler tools and test frameworks only support manually launching tests, or running them on a single DUT. With Autotest, you can also define pools of multiple DUTs, to group DUT and set scheduling priorities.
Moblab also supports different pool of DUT for testing simultaneously by using pool labels, You can run storage component qualification for first DUT pool and memory component qualification for second DUT pool by using one Moblab.
Moblab provides a common platform for test infrastructure. Using the same hardware, software, and test builds makes communication between partners and Google more efficient. When issues are found, having a common platform makes it much quicker and easier to reproduce them.
Community support provides greater security and faster response time for questions and issues. Source files are available for inspection and customization, and partner contributions are welcome. The only feature unavailable to custom builds of Moblab is automatic server updates.
Mandatory for CTS
Google Chromebox(Wukong) with Core i7, 32GB RAM, 512GB NVMe SSD And dual Network Ports.
DUT (Device Under Test)
Switching Hub (optional for multiple DUTs)
Do NOT use managing hubs, switches and routers. Size the switch appropriately for the number of DUT’s you have attached, for your particular test application.
with HDMI/DisplayPort supported
100MBps Upload / 100MBps Download
** Please contact Synnex for Moblab procurement, other accessories can be sourced from local market
Synnex Technology International Corp.
Tel : 886-2-2506-3320 Ext 2033
Here are the simple diagrams that show you how to connect the devices, depending on the number of DUTs are being tested.
Please note the lower ethernet port on Moblab should connects to DUT. See physical setup picture below.
It does not matter which network port you use on moblab to connect to the internet or switch, but the connection to the internet should be active when you boot the moblab as it uses the internet connection to calculate the network configuration.
This diagram shows a typical configuration of Moblab on a partner’s network:
STEP 1 - Power on Moblab
STEP 2 - Make sure Moblab is in Developer mode
you should see booting screen like this
STEP 3 - At welcome screen, please select Ethernet as your network
Note: Moblab IP will only be used when you want access Moblab remotely.
STEP 1 - Login with any account or simply browse as guest
STEP 2 - Enter VT-2 and login as moblab
you should not be prompted for password.
localhost Login: moblab
STEP 3 - Check network configurations
moblab@localhost ~$ ifconfig
eth0 : IP address from your internet network (upper network port)
eth1 : blank, network info for the lower network port
*Note: No meter which network port connects to the internet, it will be assigned as eth0
lo: localhost of Moblab
lxcbr0: IP address (inet) = ***.***.***.*** (this is the DHCP service of Moblab)
STEP 4 - Switch back to Chrome OS UI and browse any web page on internet to make sure internet works.
Before starting this section, you should have:
STEP 1 - Open browser, enter localhost/ or IP of your Moblab
STEP 2 - Click Moblab on top-right corner to enter setup page
STEP 3 - Click Configure to start
Moblab will show its IP as server IP and verify internet connections, if not, click Query to refresh
STEP 4 - Click Next when confirmed.
Please note Moblab doesn't support connections via proxy, please consult with your IT support for solution
STEP 5 - Enter your boto key info
STEP 6 - Enter image storage bucket URL
the format is gs://[YOUR_BUCKET_URL]/
STEP 7 - Click Finish then OK to commit all changes.
Moblab will restart some services in background and show all updated info
You don't need to change anything in Config Settings or Launch Control Key unless you receive instructions from your Google contact.
STEP 1 - Install a test image on DUTs
STEP 2 - Switch to developer mode after re-image completed
STEP 1 - Connect DUT to Moblab by using USB-Ethernet adapter.
Please refer diagram above to connect 1 or more DUTs to Moblab
STEP 2 - Boot up DUT
STEP 3 - Enter VT-2 (Ctrl-Alt-F2) and login as root
STEP 4 - Get the IP address of DUT, if you have multiple DUTs, you'll need to repeat the steps in this section to get all the IP addresses of the DUTs
Attention: IP & MAC address of eth0 on DUT based on USB-Ethernet adapter, swapping adapter may cause unexpected error
STEP 1 - Open browser, enter localhost/ or IP of your Moblab, click Moblab on top-right corner
STEP 2 - Click Manage DUT, you should see DUT listed in the page, IP & MAC address should match the info you found from DUT setup
(Always check IP address, MAC address and labels to ensure you're adding right DUT for autotests)
STEP 3 - Select DUTs and Add Selected DUT's then click Apply, Moblab will add DUT for autotest.
Click Refresh to get DUT labels
If you want to add more labels for DUT, select it again then scroll action column to "Add Labels Selected DUT's", add labels splitted by comma, then click Apply
Checking “Is pool label” will add pool label to selected DUT(s). Pool label can help you to group up different DUTs easily.
STEP 1 - Open browser, enter localhost /or IP of your Moblab, click Moblab on top-right corner
STEP 2 - Click Run Suite
STEP 3 - Select the board that you are testing
( name of the board that's connected to Moblab with your DUT should appear here )
STEP 4 - Select the build that you are going to test with
STEP 5 - Select the suite
Note that you can also have the option to run specific test jobs within the suite package.
Ex. arm.android.bluetooth, arm.android.app.usage
STEP 6 [Optional] - Select only when you have multiple DUT pools
STEP 7 - Click Run Suite
All test jobs can be found at Moblab front page
WMatrix present a better test result viewing UI for Moblab users.
The Test Results UI might look a little overwhelming. It includes all tests that currently existing in Chrome OS autotest.git. But might not be relevant to Partners initially. We recommend to start with 'Suite List" in the "Quick links" section.
Depends on the type of test suites you have run on your Moblab unit, the type of suite in the subsequent screens may differ. You may have faft, storage qual, bvt show up. Here is one example from our faft tests. From here you could drill down further to see aggregated test results.
Aggregated test results of one particular suites across build and devices. Here is a simple example of one device and one build test results. If you have your mouse hover over the "red" section on the right hand side, you will see all the failed tests associated with runs on that build (R52-8350.69.0). Clicking on the red "1" in the screen below will lead you to the detailed failure screen so you can investigate further.
NOTE: If you are running a build that is old, you might see an empty UI when you click on the Suite Name from screen shown earlier. One work around is to add "&days_back=15" (or whatever days you know that are required to get to your build comparing to today's build) to see the result. We are aware of this flaw and working on a fix for next release.
We encourage you use Moblab Admin to run individual test and test suites, if you need more customized configurations for your test, here is the instruction of using AFE running tests. Please only use Moblab Admin for CTS.
Additional information on Autotest: For descriptions of each parameter in the Create Job view or details of how the Autotest scheduler runs jobs, click here.
Running your test from the AFE
STEP 1 - Click the tab to open the Create Job view.
STEP 2 - Type a name for the job. The string should be meaningful when viewing a list of jobs.
STEP 3 -Type the name of the test image to use in the Image URL/Build field. If no image is specified and you are not running a suite job, the latest ToT image is used. For some tests Image URL/Build is a required field. In such cases add the release name here in the following format: <board>-release/RXX-YYYY.ZZ.0 eg. samus-release/R42-6812.88.0.
STEP 4 - Select the tests to run.
Advanced Options can be adjusted for your tests
If you select a suite, the hostless (suite job) attribute will automatically be selected in the “Advanced Options” section.
If you want to run a suite of tests on a pool of hosts you have defined in “Manage DUT” section, specify the pool_name in the Pool: (optional) field.
STEP 5 - Click the names of registered hosts for running the tests.
STEP 6 - Click Submit Job.
In order to test a locally built test image you must first build one in the chroot. Instructions for this can be found in the Chromium OS developer guide.
Staging a custom image on a local Moblab
To stage on a local Moblab that is accessible by your chroot you simply use the ‘cros stage’ command:
cros stage </path/to/board/build/chromiumos-test-image.bin> <Moblab IP address>
In this example I am pushing an image I built in my chroot to the Moblab device.
~/trunk/src/scripts $ cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin 100.XX.XX.XX
The path requires the format of /board/build/chromiumos-test-image.bin.
Once completed ‘cros stage’ will output the name of the image to use when kicking off tests via the run_suite command or the AFE. For this example: peppy-custom/R41-6576.0.2014_12_12_1535
Staging an official image with custom tests on a local Moblab
You can also stage official images from Google Storage with test packages from your chroot onto a local Moblab. The command, cros stage always pulls test packages from the chroot regardless of the image source.
For this workflow, you must have previously built an image for this board at some point. For information about doing custom test development please refer to the Advanced: Custom Client-Side Tests Support.
cros stage gs://<bucket-name>/<board>-<builder type>/<build name>/ <Moblab IP address>
In this example I am pushing an official image with test packages built in my chroot to the Moblab device. I would have already built a squawks test image at some point.
~/trunk/src/scripts $ cros stage gs://chromeos-image-archive/squawks-release/R42-6750.0.0 100.XX.XX.XX
/trunk/src/scripts $ cros stage gs://chromeos-image-archive/squawks-release/R42-6750.0.0 100.XX.XX.XX
Once completed ‘cros stage’ will output the name of the image to use when kicking off tests. For this example: squawks-custom/R42-6570.0.0
Staging a custom image to Google Storage
You can also push images to Google Storage. The advantages to this is that the image will not be deleted for 24 hours, and that remote Moblab’s not in the same local network can access the image as well.
cros stage </path/to/board/build/chromiumos-test-image.bin> <gs_base_path> --boto_file=<boto_file_path>
cros stage ../build/images/peppy/R41-6576.0.2014_12_12_1535-a1/chromiumos_test_image.bin gs://chromeos-moblab-peppy --boto_file=~/peppy.boto
Once completed ‘cros stage’ will output the name of the image to use when kicking off tests just like when staging on a local Moblab.
Testing the custom image
Specify the outputted image name when launching the test:
NOTE: Custom Client-Side Tests Workflow is only supported when using cros stage with a Moblab address and NOT when using an Google Storage URL as the staging destination.
Cros Stage also enables developer to run Custom Client Side Tests on their Moblab. It does this by packaging up the tests that are built and stored in the board’s buildroot (/build/<board>/) of the chroot.
Note you do not need to build a new image. Therefore you can use the same image repeatedly while doing test development.
For more information:
Using command line tools to run tests
You can also use the Autotest command line interface (CLI) to run manual tests. The following Python utilities are available to Moblab users for running tests:
For a more extensive list, click here.
To run a test or test suite from the command line, use the following syntax:
run_suite.py --board=<board_name> --build=<build_name> --suite_name=<suite_name> --pool=””
run_suite.py --board=peppy --build=peppy-release/R37.5847.0.0 --suite_name=smoke --pool=””
To obtain the names of test builds, use gsutil to browse your Google Cloud Storage bucket.
You can check job status and results through the AFE. This page describes the status values for hosts and jobs.
Writing Custom Tests
The Moblab server runs the open source git Autotest framework. The framework supports writing custom tests in Python using the Autotest APIs.
For details and examples of coding to the Autotest APIs, view the general user documentation.
Please note that although Moblab uses the Autotest framework, code refactoring has made it so that many examples do not work with our code base. However, many of these documents are still helpful at a high level.
Booting from a Recovery Image
If the Moblab server hangs and the device can’t be manually rebooted, it is possible to boot from a recovery image. For Chrome OS partners who have deployed Moblab, the recovery image is located in a folder in Google Cloud Storage. It is not generally available on the Chrome OS image release site.
To boot Moblab from a recovery image:
cros flash usb:// remote/stumpy_moblab/latest-dev/recovery
These commands have been added to the Moblab user profile so you can run them anywhere on the system. Please note that this list is valid as of version 6343.
Note: These custom tests in these directories will be overwritten if Moblab is power washed or if a new test with the same name is checked into the source.
Bringing up the test environment
After Moblab is connected to a network and booted, the Moblab server performs the following network configuration tasks:
One or more hosts are connected to the test subnet by either a switch or a hub. Moblab communicates with the test subnet using the lower network port. When the Moblab server boots up, it waits up to 5 minutes to establish an SSH connection through the lower network port and subnet. Then it assigns dynamic IP addresses to all hosts, ranging from 192.168.231.100 to 192.168.231.199.
Access to the partner’s corporate network enables automatic software updates for the Moblab server. It also enables access to Google Cloud Storage for downloading test images and server updates, and uploading logs. The corporate network should provide security features (firewall, VPN, etc.) to protect Moblab and the test subnet from potential attacks from outside the lab.
The Autotest scheduler manages and coordinates build updates. On each Moblab device, it manages two separate types of build images:
The scheduler waits for a time of low or no activity on the Moblab server to update the test image. If tests are running when an image is updated, they continue to run.
The Mob Monitor is a tool available on Moblab devices that provides:
Using the Mob Monitor can help you to:
To access the Mob Monitor web interface, go to http://localhost:9991 on your Moblab (even if your Apache server is not up, e.g. when http://localhost is not functioning, you can still use this address to get to Mob Monitor)
The web interface presents all health information in single table. Each compartment in the table describes the state of a single service running on Moblab. In the above image, we see health information for the following illustrative services:
A service may be described by the monitor as being:
The Mob Monitor colorizes each service based on their health status.
When a service is Quasi-Healthy or Unhealthy the Mob Monitor will display Warnings and Errors for that service.
Looking at the above image, we see that:
Most repair actions will require no user input, you simply have to click the button and wait ~10-15 seconds for the Mob Monitor to execute the action in the background and for the web interface to refresh.
Some repair actions do require input. If that is the case, a dialog will pop up when the run button is clicked.
The dialog will present the following information:
In the event that something terrible happens with your Moblab device (related to the Mob Monitor or not!), the Mob Monitor provides a Collect Logs button in the upper right corner of the UI. Clicking this button will open a separate tab that will provide a package of relevant system logs for download to your local machine.
Note, this operation may be slow if there are many logs on the Moblab.
Some things to keep in mind when using the Mob Monitor:
This process will walk you through how to set up ssh access into Moblab.
Moblab has a public discussion group, email@example.com. Please use this link to join this public group.
Before filing an issue, please confirm you’ve completed actions below:
Please have all screenshots ready and rename to descriptions in bold.
Use Partner Issue Tracker and select Moblab template to report issue.
Yes, you can remote access your Moblab by using browser and Moblab IP you get during setup.
Yes, you can access Moblab from DUT by using browser, however, this is only for debugging, you should never run any test on Moblab from DUT.
Moblab recovery images are publicly shared at gs://chromeos-moblab-public/moblab-images. Please refer to the release notes to find out the latest release.
Please follow this instruction to powerwash your Moblab.
Attention: If you’re trying to immaculate Moblab by re-image, you should always use Moblab recovery image, NOT Chromebox recovery image.
Moblab offloads test results every 24 hours to google cloud storage. If your network connectivity is down, the results will accumulate and if you are running long running tests, the results will build up and eat up your storage space.
Can I purchase Chromebox from somewhere and build Moblab by myself ?
No, Moblab (Wukong) was a customized Chromebox with more powerful computing power and some enhancements on its hardware, we don’t support any self-build Moblab and you should only purchase them from designated channel.
Can I setup Moblab testing environment out of Taiwan or US ?
Yes, as long as the network environment is stable and unblocked, Moblab should work in anywhere.
Please note, we don’t support any Moblab setup in China due to uncertainty network issues.
Yes, you may connect different DUTs to one Moblab unit. When running hostless tests (all "suites" are run hostlessly), during "Create Job" step, if you specify "Image URL/Build" by pointing to a particular type of image for a particular type of DUT, Moblab will automatically only push subtests to corresponding DUT based on "Image URL/Build" type.
Yes, go to VT2 (ctrl+alt+F2), log in as root, and type the following in the command line will silence the DUT:
$> /usr/share/vboot/bin/set_gbb_flags.sh 0x19
Go to the Job List page in AFE and click on Running Jobs. This will give you a list of jobs that are currently running.