Testing
Hypermedia User Experience Engineering
IM531/SS20
Topics
Testing
Terminology
Types
Automated Tests
Unit Tests
Integration Tests
Functional Tests (End-to-End Tests, GUI Tests,…)
Acceptance Tests
Performance Tests
Smoke Tests
Accessibility Tests
Security Tests
Regression Tests
Manual Tests
Exploratory Tests
Types
Automated Tests
Unit Tests
Integration Tests
Functional Tests (End-to-End Tests, GUI Tests,…)
Acceptance Tests
Performance Tests
Smoke Tests
Accessibility Tests
Security Tests
Regression Tests
Manual Tests
Exploratory Tests
The list goes on …
Types
Automated Tests
Unit Tests
Integration Tests
Functional Tests (End-to-End Tests, GUI Tests,…)
Acceptance Tests
Performance Tests
Smoke Tests
Accessibility Tests
Security Tests
Regression Tests
Manual Tests
Exploratory Tests
The list goes on …
The “Pyramid”
https://watirmelon.blog/testing-pyramids/
The “Cone”
https://watirmelon.blog/testing-pyramids/
Test Coverage
Test coverage is a useful tool for finding untested parts of a codebase. Test coverage is of little use as a numeric statement of how good your tests are.
– Martin Fowler
https://martinfowler.com/bliki/TestCoverage.html
Test Coverage
Sufficiency of testing is much more complicated attribute than coverage can answer. I would say you are doing enough testing if the following is true:
– Martin Fowler
https://martinfowler.com/bliki/TestCoverage.html
The Way of Testivus
Testivus on Test Coverage (1)
Early one morning, a programmer asked the great master:
“I am ready to write some unit tests. What code coverage should I aim for?”
The great master replied:
“Don’t worry about coverage, just write some good tests.”
The programmer smiled, bowed, and left.
https://www.artima.com/weblogs/viewpost.jsp?thread=204677
Testivus on Test Coverage (2)
Later that day, a second programmer asked the same question.
The great master pointed at a pot of boiling water and said:
“How many grains of rice should put in that pot?”
The programmer, looking puzzled, replied:
“How can I possibly tell you? It depends on how many people you need to feed, how hungry they are, what other food you are serving, how much rice you have available, and so on.”
“Exactly,” said the great master.
The second programmer smiled, bowed, and left.
https://www.artima.com/weblogs/viewpost.jsp?thread=204677
Testivus on Test Coverage (3)
Toward the end of the day, a third programmer came and asked the same question about code coverage.
“Eighty percent and no less!” replied the master in a stern voice, pounding his fist on the table.
The third programmer smiled, bowed, and left.
https://www.artima.com/weblogs/viewpost.jsp?thread=204677
Testivus on Test Coverage (4)
After this last reply, a young apprentice approached the great master:
“Great master, today I overheard you answer the same question about code coverage with three different answers. Why?”
The great master stood up from his chair:
“Come get some fresh tea with me and let’s talk about it.”
https://www.artima.com/weblogs/viewpost.jsp?thread=204677
Testivus on Test Coverage (5)
After they filled their cups with smoking hot green tea, the great master began to answer:
“The first programmer is new and just getting started with testing. Right now he has a lot of code and no tests. He has a long way to go; focusing on code coverage at this time would be depressing and quite useless. He’s better off just getting used to writing and running some tests. He can worry about coverage later.”
“The second programmer, on the other hand, is quite experience both at programming and testing. When I replied by asking her how many grains of rice I should put in a pot, I helped her realize that the amount of testing necessary depends on a number of factors, and she knows those factors better than I do – it’s her code after all. There is no single, simple, answer, and she’s smart enough to handle the truth and work with that.”
https://www.artima.com/weblogs/viewpost.jsp?thread=204677
Testivus on Test Coverage (6)
“I see,” said the young apprentice, “but if there is no single simple answer, then why did you answer the third programmer ‘Eighty percent and no less’?”
The great master laughed so hard and loud that his belly, evidence that he drank more than just green tea, flopped up and down.
“The third programmer wants only simple answers – even when there are no simple answers … and then does not follow them anyway.”
The young apprentice and the grizzled great master finished drinking their tea in contemplative silence.
https://www.artima.com/weblogs/viewpost.jsp?thread=204677
Tools
The list goes on …
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Tools
https://medium.com/welldone-software/an-overview-of-javascript-testing-in-2018-f68950900bc3
Testing Structure
describe('calculator', () => {
describe('add', () => {
it('should add 2 numbers', () => {
// Assertions
});
});
});
Assertions
expect(foo).to.be.a('string');
expect(foo).to.equal('bar');
expect(foo).toBeString();
expect(foo).toEqual('bar');
assert.typeOf(foo, 'string');
assert.equal(foo, 'bar');
expect(foo, 'to be a', 'string');
expect(foo, 'to be', 'bar');
Spies
it('should call method once with the argument 3', () => {
const spy = sinon.spy(object, 'method');
object.method(3);
assert(spy.withArgs(3).calledOnce);
});
Stubs
// Sinon�sinon.stub(user, 'isValid').returns(true);
// Jest�jest.spyOn(user, 'isValid').mockImplementation(() => true);
Mocks
// Sinon�sinon.mock(jQuery).expects('ajax').atLeast(2).atMost(5);�jQuery.ajax.verify();
// Jest�const drink = jest.fn();�expect(drink).toHaveBeenCalled();
Spy, Stub or Mock?
There are several definitions of fake objects, called test doubles.
Spies give you information about function calls.
Stubs are like spies, but they completely replace the original function.
Mocks are fake methods (like spies) with pre-programmed behavior (like stubs) as well as pre-programmed expectations.
Fake Timers
const clock = sinon.useFakeTimers({
now: 1483228800000, // 2017-01-01
});
clock.setTimeout(function () {
// …
}, 15);
clock.tick(15);
Fake XMLHttpRequest (1)
it('returns an object containing all users', (done) => {
const server = sinon.createFakeServer();
server.respondWith('GET', '/users', [
200,
{ 'Content-Type': 'application/json' },
'[{ "id": 1, "name": "Gwen" }, { "id": 2, "name": "John" }]',
]);
// ...
});
Fake XMLHttpRequest (2)
it('returns an object containing all users', (done) => {
// ...
Users.all()
.then((collection) => {
const expectedCollection = [{ id: 1, name: 'Gwen' }, { id: 2, name: 'John' }];
expect(collection.toJSON()).to.equal(expectedCollection);
done();
});
// ...
});
Fake XMLHttpRequest (3)
it('returns an object containing all users', (done) => {
// ...
server.respond();
server.restore();
});
Browser Environment
jsdom is a pure-JavaScript implementation of many web standards, notably the WHATWG DOM and HTML Standards, for use with Node.js. In general, the goal of the project is to emulate enough of a subset of a web browser to be useful for testing and scraping real-world web applications.
Unit Testing
Jest
Setup
Writing Tests for Jest
Matchers
Async Tests (1)
test('the data is peanut butter', (done) => {
function callback(data) {
expect(data).toBe('peanut butter');
done();
}
fetchData(callback);
});
Async Tests (2)
test('the data is peanut butter', () => {
expect.assertions(1);
return fetchData().then((data) => {
expect(data).toBe('peanut butter');
});
});
Async Tests (3)
test('the data is peanut butter', () => {
expect.assertions(1);
return expect(fetchData()).resolves.toBe('peanut butter');
});
test('the fetch fails with an error', () => {
expect.assertions(1);
return expect(fetchData()).rejects.toMatch('error');
});
Async Tests (4)
test('the data is peanut butter', async () => {
expect.assertions(1);
const data = await fetchData();
expect(data).toBe('peanut butter');
});
Setup and Teardown
beforeAll(/* ... */);
afterAll(/* ... */);
beforeEach(/* ... */);
afterEach(/* ... */);
test('...', /* ... */);
describe('Scoping', () => {
beforeAll(/* ... */);
// ...
});
Subsets
test.only(), test.skip() describe.only(), descripe.skip()
test.only('this will be the only test that runs', () => {
expect(true).toBe(false);
});
test('this test will not run', () => {
expect('A').toBe('A');
});
// test.only(), test.skip() describe.only(), descripe.skip()
Mock Functions
const myMock = jest.fn();
expect(myMock.mock.calls.length).toBe(1);
expect(myMock.mock.calls[0][0]).toBe('first arg');
expect(myMock.mock.calls[0][1]).toBe('second arg');
expect(myMock.mock.instances.length).toBe(2);
expect(myMock.mock.instances[0].name).toEqual('test');
Custom Matchers
const myMock = jest.fn();
expect(myMock).toBeCalled();
expect(myMock).toBeCalledWith('first arg', 'second arg');
Mock Return Values
const myMock = jest.fn();
console.log(myMock());
// undefined
myMock
.mockReturnValueOnce(10)
.mockReturnValueOnce('x')
.mockReturnValue(true);
Mocking Modules (1)
import axios from 'axios';
class Users {
static all() {
return axios.get('/users.json')
.then(response => response.data);
}
}
export default Users;
Mocking Modules (2)
import axios from 'axios';
import Users from './users';
jest.mock('axios');
test('should fetch users', () => {
const users = [{ name: 'Bob' }];
const response = { data: users };
axios.get.mockResolvedValue(response);
return Users.all().then(response => {
expect(response.data).toEqual(users);
});
});
Mock Implementations
const myMockFn = jest.fn(callback => callback(null, true));
myMockFn((error, value) => console.log(value));
// true
// .mockImplementation(callback => callback(null, true));
// .mockImplementationOnce(callback => callback(null, true));
// .mockImplementationOnce(callback => callback(null, false));
Manual Mocks
Manual Mocks
// __tests__/mocking.test.js
import lodash from 'lodash';
test('if lodash head is mocked', () => {
expect(lodash.head([2, 3])).toEqual(5);
});
// __mocks__/lodash.js
const lodash = jest.genMockFromModule('lodash');
lodash.head = arr => 5;
export default lodash;
Snapshot Testing (1)
it('renders correctly', () => {
const tree = renderer
.create(<Link page="https://www.fh-ooe.at/">FH OÖ</Link>)
.toJSON();
expect(tree).toMatchSnapshot();
});
Snapshot Testing (2)
exports[`renders correctly 1`] = `
<a
className="normal"
href="https://www.fh-ooe.at/"
onMouseEnter={[Function]}
onMouseLeave={[Function]}
>
FH OÖ
</a>
`;
Snapshot Testing
Writing Tests for Vue.js
Vue Test Utils
Vue Test Utils
Vue Test Utils tests Vue components by mounting them in isolation, mocking the necessary inputs (props, injections and user events) and asserting the outputs (render result, emitted custom events).
Mounted components are returned inside a wrapper, which exposes many convenience methods for manipulating, traversing and querying the underlying Vue component instance.
Vue Test Utils
Vue Test Utils
<template>
<div>
<div class="message">{{ message }}</div>
Enter your username: <input v-model="username">
<div v-if="error" class="error">
Please enter a username with at least seven letters.
</div>
</div>
</template>
Vue Test Utils
export default {
name: 'Foo',
data() {
return { message: 'Howdy!', username: '' };
},
computed: {
error() {
return this.username.trim().length < 7;
},
},
};
Vue Test Utils
describe('Foo', () => {
it('renders a message and responds correctly to user input', () => {
const wrapper = shallowMount(Foo, {
data: { message: 'Hello World', username: '' },
});
expect(wrapper.find('.message').text()).toEqual('Hello World');
expect(wrapper.find('.error').exists()).toBeTruthy();
wrapper.setData({ username: 'Gandalf' });
expect(wrapper.find('.error').exists()).toBeFalsy();
});
});
Vue Test Utils
import { shallowMount } from '@vue/test-utils';
import Foo from './Foo';
const factory = (values = {}) => shallowMount(Foo, {
data: { ...values },
});
describe('Foo', () => {
// ...
});
Vue Test Utils
// ...
describe('Foo', () => {
it('renders a welcome message', () => {
const wrapper = factory();
expect(wrapper.find('.message').text()).toEqual('Howdy!');
});
it('renders an error when username is less than 7 characters', () => {
const wrapper = factory({ username: '' });
expect(wrapper.find('.error').exists()).toBeTruthy();
});
// ...
});
Vue Test Utils
// ...
describe('Foo', () => {
// ...
it('renders an error when username is whitespace', () => {
const wrapper = factory({ username: ' '.repeat(7) });
expect(wrapper.find('.error').exists()).toBeTruthy();
});
it('does not render an error when username is 7 characters or more', () => {
const wrapper = factory({ username: 'Lachlan' });
expect(wrapper.find('.error').exists()).toBeFalsy();
});
});
Vue Test Utils
End-to-End Testing
Nightwatch
Theory of Operation
GeckoDriver
ChromeDriver
SafariDriver
Microsoft�WebDriver
Browsers
POST /session
{ "sessionId": 123 }
Setup (+ Examples)
Tests
Page Objects
The purpose of the page object pattern is to allow a software client to do anything and see anything that a human can by abstracting away the underlying HTML actions needed to access and manipulate the page.
Page objects allow you to
Page Objects
// google.js (1)�const googleCommands = {
submit() {
this.api.pause(1000);
return this.waitForElementVisible('@submitButton', 1000)
.click('@submitButton')
.waitForElementNotPresent('@submitButton');
}
};
Page Objects
// google.js (2)
module.exports = {� url: 'https://google.com',
commands: [googleCommands],
elements: {
searchBar: { selector: 'input[type=text]' },
submitButton: { selector: 'input[name=btnK]' }
}
};
Page Objects
module.exports = {
'Test': function (browser) {
const google = browser.page.google();
google� .setValue('@searchBar', 'FH OÖ Campus Hagenberg')
.submit();
browser.end();
}
};
Extending Nightwatch
Most of the time you will need to extend the Nightwatch commands to suit your own application needs. Doing that is only a matter of creating a separate folder and defining your own commands inside there, each one inside its own file.
Nightwatch also allows you to define your own assertions �(by extending the available assert object) and reporters.
Cucumber
Cucumber�Minimal Nightwatch Config w/ Chrome Driver�Using Nightwatch w/ Cucumber
Expands on Behaviour-Driven Development (BDD), as Cucumber aims to keep requirements specifications, tests and documentation in sync.
Cucumber executes executable specifications written in plain language and produces reports indicating whether the software behaves according to the specification or not.