Overview

for device farms

Anna Klueva�annaklueva888@gmail.com

Anna Klueva

Position: Automation QA Engineer at Grid Dynamics

Experience: 5 years & 2 years in mobile automation

Email: annaklueva888@gmail.com

Agenda

• Why cloud?
• Overview of device farms:
• Kobiton
• Saucelabs
• AWS Device farm
• Bitbar
• Parallelization
• Summary

3

4

Emulator

Real device

vs.

Why cloud?

5

6

Kobiton: Pros & Cons

Privileged and Confidential

7

• Easy of use
• Easy of setup
• Reporting: no affect on custom reports
• Cheap

Pros

• Supports ONLY Appium
• Appium version can't be changed

Cons

Kobiton: How it works?

1) Upload app to Kobiton

2) Configure Driver and desired capabilities in your project

3) Run test on local machine

Example of configuration:� String APPIUM_URL =� "https://username:apiKey@api.kobiton.com";� driver = new AndroidDriver<MobileElement>(�new URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());�� private static DesiredCapabilities getCapabilities() {� DesiredCapabilities capabilities = new DesiredCapabilities();� capabilities.setCapability("deviceName", “Galaxy Note3”); � capabilities.setCapability("platformVersion", “6”);� capabilities.setCapability("platformName", "Android");� capabilities.setCapability("app", pathToRemoteApp); � return desiredCapabilities;� }�

8

9

Kobiton: Api description

​

Upload App to repo

Get info about devices

Get device status

Get logs info

10

Kobiton: How CI/CD process can be organized?

Kobiton Api

11

Kobiton: Specific feature

1. Health-check for your application
2. Supports different build tools: Maven, Gradle

12

13

Saucelabs: Pros & Cons

14

• Support different frameworks
• Easy of use
• Easy of setup
• Reporting: no affect on custom reports

Pros

• Not cheap

Cons

Saucelabs: How it works?

1. Upload app via Api
2. Configure driver and DesiredCapabilities
3. Run tests

​

​

15

String APPIUM_URL = “https://eu1.appium.testobject.com”;�driver = new AndroidDriver<MobileElement>(new URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());

private static DesiredCapabilities getCapabilities() {� DesiredCapabilities desiredCapabilities = new DesiredCapabilities();� desiredCapabilities.setCapability("platformName","Android");� desiredCapabilities.setCapability("testobject_api_key", SAUCELABS_API_KEY);� desiredCapabilities.setCapability("testobject_app_id", SAUCELABS_UPLOADED_APP_ID);� return desiredCapabilities;�}

​

​

​

16

Saucelabs: Api

Saucelabs (TestObject) provides:

​

17

Storage API (upload app)

Test Reports API

Logs API

Devices API

Appium Suites API

Saucelabs: How to integrate with CI/CD flow?

Two ways:

1. Sauce plugins for Jenkins, TeamCity, Bamboo
2. API

18

Saucelabs: Specific feature

Automated Quality Report

Tests included in the Quality Report are:

• Install and Launch tests
• Stress tests

19

AWS Device farm

20

AWS Device farm: Pros & Cons

21

• Supports different frameworks
• Ability to test specific cases
• Rich test analytics

Pros

• No local run
• No custom reports, just aws reports

Cons

AWS Device farm: How it works?

1) Your project with tests:

1. Configure Driver and DesiredCapabilities�2. Archive project

2) AWS: Upload application to AWS Device farm

3) AWS: Upload zipped project to aws

5) AWS: Select device and run type

6) AWS: Run test on server

22

Example:

String APPIUM_URL = “http://127.0.0.1:4723”;�driver = new AndroidDriver<MobileElement>(new URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());��/*** We do not need to and SHOULD NOT set our own DesiredCapabilities* Device Farm creates custom settings at the server level. Setting your own DesiredCapabilities will result in unexpected results and failures.**/�private static DesiredCapabilities getCapabilities() {� return new DesiredCapabilities(); �}

​

​

23

AWS device farm: API description

Upload app

Upload package with tests

Get info about devices

Get info about user

Run tests on AWS Device cloud

24

AWS Device farm: How to integrate with CI/CD flow?

Two ways:�1) Jenkins plugin

“aws-device-farm”

2) API

​

25

AWS Device farm: Specific feature

1. Build tools: Maven, Gradle
2. AWS Device farm client application

​

26

27

Bitbar: Pros & Cons

28

• Supports different frameworks
• Easy of use
• Rich test analytics
• Integration with third party systems

Pros

• Not easy of setup
• Not cheap

Cons

Bitbar: How it works?

Client-side: run tests locally on remote real devices

Server-side: run tests on Bitbar server on remote real device

29

Bitbar: client-side run

1. Upload app to Bitbar server
2. Configure Driver and DesiredCapabilities
3. Bitbar: create project
4. Run tests

30

Example of configuration:�String APPIUM_URL = “https://appium.bitbar.com”;�driver = new AndroidDriver<MobileElement>(new URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());

private static DesiredCapabilities getCapabilities() {� DesiredCapabilities desiredCapabilities = new DesiredCapabilities();� desiredCapabilities.setCapability("platformName","Android");� desiredCapabilities.setCapability("deviceName", "Android Device");� desiredCapabilities.setCapability("testdroid_apiKey", BITBAR_API_KEY);� desiredCapabilities.setCapability("testdroid_project", BITBAR_PROJECT);� desiredCapabilities.setCapability("testdroid_testrun", "Demo1");� desiredCapabilities.setCapability("testdroid_app",UPLOADED_APP_PATH);� desiredCapabilities.setCapability("testdroid_device","LG Google Nexus 5");� desiredCapabilities.setCapability("testdroid_target", "Android"); � return desiredCapabilities;�}

Bitbar: server-side run

1. Upload app to server
2. Configure Driver and Desired Capabilities
3. Archive and upload your tests to Bitbar server
4. Select devices
5. Run tests on server

31

Example:��String APPIUM_URL = “http://127.0.0.1:4723”;�driver = new AndroidDriver<MobileElement>(new URL(APPIUM_URL+"/wd/hub"), Configuration.getCapabilities());

/** app - don't change path to the .apk and .apk name. Bitbar stores you uploaded app application.apk*/��private static DesiredCapabilities getCapabilities() {� DesiredCapabilities desiredCapabilities = new DesiredCapabilities();� desiredCapabilities.setCapability("platformName","Android");� desiredCapabilities.setCapability("deviceName", "Android Device");� desiredCapabilities.setCapability("app", System.getProperty("user.dir") + "/application.apk");� return desiredCapabilities;�}

​

​

​

32

Bitbar: API description

Bitbar API provides:

​

33

Upload app

Get info about devices

Run tests and get results

Bitbar: How to integrate with CI/CD flow?

Two ways:

1. Jenkins plugin: “Bitbar Run-in-Cloud”
2. Bitbar API

34

Bitbar: Specific feature

• AI Testbot and AppCrawler
• Integration with JIRA, Slack, Hipchat

35

Parallelization

Parallelization in Kobiton and Saucelabs:

Parallelization in Bitbar and AWS:

Set of tests

Test cases

Result

(Runs on real devices)

Test

Test sessions/scripts

Results

(Runs on real devices)

36

Let’s summarize

37

Supported frameworks

38

Tricky cases

1. If you use third party apps (Google Account, Apple ID), the personal data can not be deleted
2. Appium version is updated randomly
3. Documentation is not up-to-date

39

Q & A

Thank you!

www.griddynamics.com