Principles of Software Construction: Objects, Design, and Concurrency��Test case design��Jonathan Aldrich, Charlie Garrod, Jeremy Lacomis
1
17-214/514
Learning Goals
Explain different coverage metrics and when they are appropriate
Write test cases to achieve line and branch coverage for a function
Deliberately decide when and how to use or combine structural and specification-based testing
2
17-214/514
Today
3
17-214/514
Specifications and testing are closely related
Q: What exactly do you test when given a method?
4
17-214/514
Structural Testing
5
17-214/514
Structural Testing: a closer look
Takes into account the internal mechanism of a system (IEEE, 1990).
6
17-214/514
Case Study
Assume various Wallets
public interface Wallet {
boolean pay(int cost);
int getValue();
}
7
17-214/514
DebitWallet.pay()
What should we test in this code?
public boolean pay(int cost) {
if (cost <= this.money) {
this.money -= cost;
return true;
}
return false;
}
8
17-214/514
DebitWallet.pay()
public boolean pay(int cost) {
if (cost <= this.money) {
this.money -= cost;
return true;
}
return false;
}
new DebitWallet(100).pay(10);
9
17-214/514
DebitWallet.pay()
public boolean pay(int cost) {
if (cost <= this.money) {
this.money -= cost;
return true;
}
return false;
}
new DebitWallet(0).pay(10);
10
17-214/514
CreditWallet.pay()
How about now?
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (this.credit + cost <= this.maxCredit) {
this.credit += cost;
return true;
}
}
if (cost <= this.cash) {
this.cash -= cost;
return true;
}
return false;
}
11
17-214/514
CreditWallet.pay()
Exercise: think about as many test scenarios as you can
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
12
17-214/514
CreditWallet.pay()
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
Test case | useCredit | enough Credit | enough Cash | Result | Coverage |
1 | T | T | - | Pass | -- |
13
17-214/514
CreditWallet.pay()
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
Test case | useCredit | enough Credit | enough Cash | Result | Coverage |
1 | T | T | - | Pass | -- |
2 | F | - | T | Pass | -- |
3 | F | - | F | Fails | Statement |
14
17-214/514
Coverage
We have tested every statement; are we done?�Depends on desired coverage:
15
17-214/514
Structures in Code
16
17-214/514
Control-Flow of CreditCard.pay()
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
17
17-214/514
Control-Flow of CreditCard.pay()
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
18
17-214/514
Control-Flow of CreditCard.pay()
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
Test case | useCredit | enough Credit | enough Cash | Result | Coverage |
1 | T | T | - | Pass | -- |
2 | F | - | T | Pass | -- |
3 | F | - | F | Fails | Statement |
19
17-214/514
Control-Flow of CreditCard.pay()
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
Test case | useCredit | enough Credit | enough Cash | Result | Coverage |
1 | T | T | - | Pass | -- |
2 | F | - | T | Pass | -- |
3 | F | - | F | Fails | Statement |
20
17-214/514
Control-Flow of CreditCard.pay()
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
Test case | useCredit | enough Credit | enough Cash | Result | Coverage |
1 | T | T | - | Pass | -- |
2 | F | - | T | Pass | -- |
3 | F | - | F | Fails | Statement |
21
17-214/514
CreditWallet.pay()
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
Test case | useCredit | enough Credit | enough Cash | Result | Coverage | |
1 | T | T | - | Pass | -- | |
2 | F | - | T | Pass | -- | |
3 | F | - | F | Fails | Statement | |
4 | T | F | T | Pass | Branch | |
22
17-214/514
Path Coverage
We have seen every condition … what else is missing?
23
17-214/514
Path Coverage
We have seen every condition … but not every path.
24
17-214/514
Control-Flow of CreditCard.pay()
Paths:
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
25
17-214/514
Control-Flow of CreditCard.pay()
Paths:
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
26
17-214/514
Control-Flow of CreditCard.pay()
Paths:
useCredit
enough Credit
pay w/credit
true
true
enough
Cash
pay w/cash
fail
false
false
false
true
27
17-214/514
CreditWallet.pay()
public boolean pay(int cost, boolean useCredit) {
if (useCredit) {
if (enoughCredit) {
return true;
}
}
if (enoughCash) {
return true;
}
return false;
}
Test case | useCredit | enough Credit | enough Cash | Result | Coverage | ||
1 | T | T | - | Pass | -- | | |
2 | F | - | T | Pass | -- | | |
3 | F | - | F | Fails | Statement | | |
4 | T | F | T | Pass | Branch | | |
5 | T | F | F | Fails | (Basis) paths | ||
28
17-214/514
BitCoinWallet.pay()
public boolean pay(int cost) {
int currValue;
while ((currValue = getValue()) < cost) {
// Just wait.
}
this.btc -= cost / currValue;
return true;
}
public int getValue() {
return (int)
(this.btc * Math.pow(2, 20*Math.random()));
}
29
17-214/514
Control-flow of BitCoinWallet.pay()
What are all the paths?
BTC value�enough?
pay w/btc
true
false
30
17-214/514
Control-flow of BitCoinWallet.pay()
What are all the paths?
BTC value�enough?
pay w/btc
true
false
31
17-214/514
Control-flow of BitCoinWallet.pay()
Perfect “general” path coverage is elusive
But “adequate” coverage criteria exist:
BTC value�enough?
pay w/btc
true
false
32
17-214/514
More Coverage
Many more criteria exist:
33
17-214/514
Test Coverage?
34
17-214/514
Coverage and Quality
if a ≤ 1
x = a - 1
y = z / x
else
x = 5
Question 1: Is there a defect?
then
35
17-214/514
Coverage and Quality
if a ≤ 1
x = a - 1
y = z / x
else
x = 5
Question 2: Can we achieve 100% statement coverage and miss the defect?
then
36
17-214/514
Coverage and Quality
if a ≤ 1
x = a - 1
y = z / x
then
else
x = 5
Question 3: Can we achieve 100% branch coverage and miss the defect?
37
17-214/514
Test your Understanding: Old Midterm Question
38
17-214/514
Outline
39
17-214/514
Testability
40
17-214/514
Coding like the tour the france
public boolean foo() {
try {
synchronized () {
if () {
} else {
}
for () {
if () {
if () {
if () {
if ()
{
if () {
for () {
}
}
}
} else {
if () {
for () {
if () {
} else {
}
if () {
} else {
if () {
}
}
if () {
if () {
if () {
for () {
}
}
}
} else {
}
}
} else {
}
}
}
}
}
if () {
}
41
17-214/514
Writing Testable Code
What is the problem with this?
public boolean hasHeader(String path) throws IOException {
List<String> lines = Files.readAllLines(Path.of(path));
return !lines.get(0).isEmpty()
}
// complete control-flow coverage!
hasHeader(“cards.csv”) // true
42
17-214/514
Writing Testable Code
What is the problem with this?
public boolean hasHeader(String path) throws IOException {
List<String> lines = Files.readAllLines(Path.of(path));
return !lines.get(0).isEmpty()
}
// to achieve a ‘false’ output without having a test input file:
try {
Path tempFile = Files.createTempFile(null, null);
Files.write(tempFile,"\n".getBytes(StandardCharsets.UTF_8));
hasHeader(tempFile.toFile().getAbsolutePath()); // false
} catch (IOException e) {
e.printStackTrace();
}
43
17-214/514
Writing Testable Code
Exercise: rewrite to make this easier
public boolean hasHeader(String path) throws IOException {
List<String> lines = Files.readAllLines(Path.of(path));
return !lines.get(0).isEmpty()
}
44
17-214/514
Writing Testable Code
Aim to write easily testable code
public List<String> getLines(String path) throws IOException {
return Files.readAllLines(Path.of(path));
}
public boolean hasHeader(List<String> lines) {
return !lines.get(0).isEmpty()
}
// Test:
// - hasHeader with empty, non-empty first line
// - getLines (if you must) with null, real path
45
17-214/514
Writing Testable Code
What is the problem with this?
public String[] getHeaderParts(List<String> lines) {
if (!lines.isEmpty()) {
String header = lines.get(0);
if (header.contains(",")) {
return header.split(",");
} else {
return new String[0];
}
} else {
return null;
}
}
46
17-214/514
Writing Testable Code
Split functionality into easily testable units
public String getFirstLine(List<String> lines) {
if (!lines.isEmpty()) {
return lines.get(0);
} else {
return null;
}
}
public String[] getHeaderParts(String header) {
if (header.contains(",")) {
return header.split(",");
} else {
return new String[0];
}
}
47
17-214/514
Coding like the tour the france
public boolean foo() {
try {
synchronized () {
if () {
} else {
}
for () {
if () {
if () {
if () {
if ()
{
if () {
for () {
}
}
}
} else {
if () {
for () {
if () {
} else {
}
if () {
} else {
if () {
}
}
if () {
if () {
if () {
for () {
}
}
}
} else {
}
}
} else {
}
}
}
}
}
if () {
}
48
17-214/514
Testing Coverage Practices
Coverage is useful, but no substitute for your insight
49
17-214/514
Good Testing Practices
50
17-214/514
Good Testing Practices
51
17-214/514
Quiz: Which Good Testing Practices does this Test Violate?
public String[] getHeaderParts(String header) {
if (header.contains(",")) {
return header.split(",");
} else {
return null;
}
}
@Test
public void testGetHeaderParts() {
for (String header : List.of("line", "", "one,two")) {
String[] parts = getHeaderParts(header);
if (header.contains(",")) assertNull(parts);
else assertEqual(header.split(","), parts.length);
}
}
bit.ly/214f23q5
52
17-214/514
How to Improve?
public String[] getHeaderParts(String header) {
if (header.contains(",")) {
return header.split(",");
} else {
return null;
}
}
@Test
public void testGetHeaderParts() {
for (String header : List.of("line", "", "one,two")) {
String[] parts = getHeaderParts(header);
if (header.contains(",")) assertNull(parts);
else assertEqual(header.split(","), parts.length);
}
}
53
17-214/514
Keep tests simple, small
public String[] getHeaderParts(String header) {
if (header.contains(",")) {
return header.split(",");
} else {
return null;
}
}
@Test
public void testGetHeaderPartsNoComma() {
String[] parts = getHeaderParts("line");
assertNull(parts);
}
@Test
...
54
17-214/514
Avoid repeating implementation
public String[] getHeaderParts(String header) {
if (header.contains(",")) {
return header.split(",");
} else {
return null;
}
}
@Test
public void testGetHeaderPartsSplit() {
String line = "one,two";
assertEqual(line.split(","), getHeaderParts(line))
}
@Test
...
55
17-214/514
Good Testing Practices
56
17-214/514
Let’s critique some tests
https://github.com/yargs/yargs/tree/main/test
https://github.com/JabRef/jabref/tree/main/src/test/java/org/jabref
https://github.com/Ironclad/rivet/tree/main/packages/core/test/model/nodes
57
17-214/514
Outline
58
17-214/514
Specification-Based Testing
59
17-214/514
Back to Specification Testing
What would you test differently in this situation?
/** Pays with credit if useCredit is set and enough
* credit is available; otherwise, pays with cash if
* enough cash is available; otherwise, returns false.
*/
public boolean pay(int cost, boolean useCredit);
60
17-214/514
Back to Specification Testing
What would you test differently in this situation?
/** Pays with credit if useCredit is set and enough
* credit is available; otherwise, pays with cash if
* enough cash is available; otherwise, returns false.
*/
public boolean pay(int cost, boolean useCredit);
61
17-214/514
Specification Testing
We need a strategy to identify plausible mistakes
62
17-214/514
Specification Testing
We need a strategy to identify plausible mistakes
63
17-214/514
Boundary Value Testing
We need a strategy to identify plausible mistakes
/** Returns true and subtracts cost if enough � * money is available, false otherwise.
*/
public boolean pay(int cost) {
if (cost < this.money) {
this.money -= cost;
return true;
}
return false;
}
64
17-214/514
Boundary Value Testing
We need a strategy to identify plausible mistakes
/** Returns true and subtracts cost if enough � * money is available, false otherwise.
*/
public boolean pay(int cost) {
if (cost < this.money) {
this.money -= cost;
return true;
}
return false;
}
65
17-214/514
Boundary Value Testing
We need a strategy to identify plausible mistakes
/** Pays with credit if useCredit is set and enough
* credit is available; otherwise, pays with cash if
* enough cash is available; otherwise, returns false.
*/
public boolean pay(int cost, boolean useCredit);
66
17-214/514
Equivalence Class Testing
Cannot try every single value -> group them where we expect similarities, select one representative each
cost: <0, 0, 1, >1, MAX_INT
useCredit: true, false
available credit: 0, >1, > 100, MAXINT
available cash: 0, >1, > 100, MAXINT
/** Pays with credit if useCredit is set and enough
* credit is available; otherwise, pays with cash if
* enough cash is available; otherwise, returns false.
*/
public boolean pay(int cost, boolean useCredit);
67
17-214/514
Combining Test Inputs
Works for both both boundary values and equivalence classes
Weak: Create a test to cover every class at least once
Strong: Create a test for every combination of classes for different values
Combinatorial: Cover only pairwise/threewise interactions
/** Pays with credit if useCredit is set and enough
* credit is available; otherwise, pays with cash if
* enough cash is available; otherwise, returns false.
*/
public boolean pay(int cost, boolean useCredit);
68
17-214/514
Decision Tables
We need a strategy to identify plausible mistakes
Test case | useCredit | enough Credit | enough Cash | Result |
1 | T | T | - | Pass |
2 | F | - | T | Pass |
3 | F | - | F | Fails |
4 | T | F | T | Pass |
5 | T | F | F | Fails |
69
17-214/514
Specification Tests
So what is the right granularity?
70
17-214/514
Structural Testing vs. Specification Testing
You will typically have both code & (prose) specification
71
17-214/514
Further Testing Strategies
Many more aspects, some later in this course:
72
17-214/514
Summary
Testing comprehensively is hard
73
17-214/514