1 of 42

Unit-5�Efficient Test Suite Management

2 of 42

WHY DOES A TEST SUITE GROW?

  • A testing criterion is a rule or collection of rules that imposes requirements on a set of test cases.
  • Test requirements are specific things that must be satisfied or covered e.g. for statement coverage, each statement is a requirement.
  • Test engineers measure the extent to which a criterion is satisfied in terms of coverage;
  • A test set achieves 100% coverage if it completely satisfies the criterion.

3 of 42

WHY DOES A TEST SUITE GROW?

  • Coverage is used as a stopping point to decide when a program is sufficiently tested.
  • For example, to achieve statement coverage adequacy for a program, one would add additional test cases to the test suite until each statement in that program is executed by at least one test case.

4 of 42

WHY DOES A TEST SUITE GROW?

  • Test cases in an existing test suite can often be used to test a modified program.
  • However, if the test suite is inadequate for retesting, new test cases may be developed and added to the test suite. Thus, the size of a test suite grows as the software evolves.

5 of 42

WHY DOES A TEST SUITE GROW?

  • A change in a program causes a test case to become obsolete by removing the reason for the test case’s inclusion in the test suite.
  • A test case is redundant if other test cases in the test suite provide the same coverage of the program.
  • Thus, due to obsolete and redundant test cases, the size of a test suite continues to grow unnecessarily.

​

6 of 42

MINIMIZING THE TEST SUITE AND ITS BENEFITS

  • A test suite can sometimes grow to an extent that it is nearly impossible to execute.
  • In this case, it becomes necessary to minimize the test cases.
  • Following are the reasons why minimization is important:
  • Release date of the product is near.
  • Limited staff to execute all the test cases.
  • Limited test equipments or unavailability of testing tools.

7 of 42

MINIMIZING THE TEST SUITE AND ITS BENEFITS

Minimizing a test suite has the following benefits:

  1. Through minimization, these redundant test cases will be eliminated.
  2. Test suite minimization techniques lower costs by reducing a test suite to a minimal subset.
  3. Reducing the size of a test suite decreases both the overhead of maintaining the test suite and the number of test cases that must be rerun after changes are made to the software, thereby reducing the cost of regression testing.

8 of 42

TEST SUITE PRIORITIZATION

  • The purpose of prioritization is to reduce the set of test cases based on some criteria, while aiming to select the most appropriate tests.
  • Priority 1 The test cases must be executed
  • Priority 2 The test cases may be executed, if time permits.
  • Priority 3 The test case is not important prior to the current release.
  • Priority 4 The test case is never important, as its impact is nearly negligible.

9 of 42

TEST SUITE PRIORITIZATION

Goals of prioritization

  • Testers or customers may want to get some critical features tested and presented in the first version of the software. Thus, the important features become the criteria for prioritizing the test cases.
  • Prioritization can be on the basis of the functionality advertised in the market. It becomes important to test those functionalities on a priority basis, which the company has promised to its customers.

​

10 of 42

TEST SUITE PRIORITIZATION

  • Goals of prioritization
  • Increase the likelihood of revealing faults related to specific code changes, earlier in the regression testing process.

11 of 42

TYPES OF TEST CASE PRIORITIZATION

General Test Case Prioritization

  • In this prioritization, we prioritize the test cases that will be useful over a succession of subsequent modified versions of P, without any knowledge of the modified versions.

Version-Specific Test Case Prioritization

  • Here, we prioritize the test cases such that they will be useful on a specific version P I of P.

12 of 42

PRIORITIZATION TECHNIQUES

  • Prioritization techniques schedule the execution of test cases in an order that attempts to increase their effectiveness at meeting some performance goal.
  • Prioritization can be done at two levels
  • Prioritization for regression test suite
  • Since regression testing is performed whenever there is a change in the software, we need to identify the test cases corresponding to modified and affected modules.

13 of 42

PRIORITIZATION TECHNIQUES

Prioritization for system test suite

  • This category prioritizes the test suite of system testing.
  • Here, the consideration is not the change in the modules.
  • The test cases of system testing are prioritized based on several criteria: risk analysis, user feedback, fault-detection rate, etc.

14 of 42

PRIORITIZATION TECHNIQUES

COVERAGE-BASED TEST CASE PRIORITIZATION:

  • This type of prioritization is based on the coverage of codes, such as statement coverage, branch coverage, etc. and the fault exposing capability of the test cases.
  • Total Statement Coverage Prioritization
  • This prioritization orders the test cases based on the total number of statements covered by the test cases and orders them in a descending order.
  • The test case that covers the highest number of statements will be executed first.

15 of 42

PRIORITIZATION TECHNIQUES

  • For example, if T1 covers 5 statements, T2 covers 3, and T3 covers 12 statements; then according to this prioritization, the order will be T3, T1, T2.

Additional Statement Coverage Prioritization

  • Additional statement coverage prioritization iteratively selects a test case T1, that yields the greatest statement coverage, then selects a test case which covers a statement uncovered by T1. Repeat this process until all statements covered by at least one test case have been covered.

16 of 42

PRIORITIZATION TECHNIQUES

17 of 42

PRIORITIZATION TECHNIQUES

  • According to total statement coverage criteria, the order is 2, 1, 3. But additional statement coverage selects test case 2 first and next, it selects test case 3, as it covers statement 4 which has not been covered by test case 2. Thus, the order according to addition coverage criteria is 2,3,1.

18 of 42

PRIORITIZATION TECHNIQUES

  • Total Branch Coverage Prioritization
  • In this prioritization, the criterion to order is to consider condition branches in a program instead of statements.
  • The test case which will cover maximum branch outcomes will be ordered first.
  • For example, Here the order will be 1, 2, 3.

19 of 42

PRIORITIZATION TECHNIQUES

For example, Here the order will be 1, 2, 3.

​

20 of 42

PRIORITIZATION TECHNIQUES

Total Fault-Exposing-Potential ( FEP) Prioritization

  • The ability of a test case to expose a fault is called the fault exposing potential.
  • To perform total FEP prioritization, given these (FEP)(s, t) values, calculate an award value for each test case ti , by summing the (FEP)(sj, ti) values for all statements sj in Program.
  • Given these award values, we prioritize the test cases by sorting them in order of descending award value.

21 of 42

PRIORITIZATION TECHNIQUES

Total FEP prioritization outputs the test case order as (2, 3, 1).

22 of 42

PRIORITIZATION TECHNIQUES

RISK-BASED PRIORITIZATION

  • Risk-based technique is to prioritize the test cases based on some potential problems which may occur during the project.
  • The testers use this risk analysis to select the most crucial tests.

​

23 of 42

PRIORITIZATION TECHNIQUES

  • Probability of occurrence
  • It indicates the probability of occurrence of a problem.
  • Severity of impact/ failure impact
  • If the problem has occurred, how much impact does it have on the software.
  • Risk analysis uses these two components by first listing the potential problems and then, assigning a probability and severity value for each identified problem, as shown in Table .

24 of 42

PRIORITIZATION TECHNIQUES

25 of 42

PRIORITIZATION TECHNIQUES

  • By ranking the results in the form of risk exposure, testers can identify the potential problems against which the software needs to be tested and executed first.
  • For example, the problems in the given table can be prioritized in the order of P5, P4, P2, P3, P1.

26 of 42

PRIORITIZATION TECHNIQUES

PRIORITIZATION BASED ON REQUIREMENTS

  • This technique is used for prioritizing the system test cases.
  • Since system test cases are largely dependent on the requirements, the requirements can be analysed to prioritize the test cases.
  • Test cases corresponding to important and critical requirements are given more weight as compared to others,

27 of 42

PRIORITIZATION TECHNIQUES

  • four factors for analysing and measuring the criticality of requirements:
  • Customer-assigned priority of requirements
  • customer assigns a weight (on a scale of 1 to 10) to each requirement. Higher the number, higher is the priority of the requirement.
  • Requirement volatility
  • This is a rating based on the frequency of change of a requirement. The requirement with a higher change frequency is assigned a higher weight as compared to the stable requirements.

28 of 42

PRIORITIZATION TECHNIQUES

  • Developer-perceived implementation complexity
  • All the requirements are not equal on a implementation level. The developer gives more weight to a requirement which he thinks is more difficult to implement.
  • Fault proneness of requirements
  • If a requirement in an earlier version of the system has more bugs, i.e. it is error-prone, then this requirement in the current version is given more weight. This factor cannot be considered for a new software.

29 of 42

PRIORITIZATION TECHNIQUES

  • Based on these four factor values, a prioritization factor value (PFV) is computed
  • PFVi = Σ(FVi j X FWj)
  • where FVij = value of factor j corresponding to requirement i
  • FWj = Factor weight is the weight given to factor j
  • PFV is then used to produce a prioritized list of system test cases.

​

30 of 42

PRIORITIZATION TECHNIQUES

PRIORITIZATION USING RELEVANT SLICES

  • Re-running the test suite for every change in the software makes regression testing a time-consuming process.
  • If we can find the portion of the software which has been affected with the change in software, then we can prioritize the test cases based on this information. This is called the slicing technique.

31 of 42

PRIORITIZATION TECHNIQUES

  • Execution Slice
  • The set of statements executed under a test case is called the execution slice of the program.
  • For example, the following program

​

32 of 42

PRIORITIZATION TECHNIQUES

  • Begin
  • S1: read (basic, empid);
  • S2: gross = 0;
  • S3: if (basic > 5000 || empid > 0)
  • {
  • S4: da = (basic*30)/100;
  • S5: gross = basic + da;
  • }
  • S6: else
  • {
  • S7: da = (basic*15)/100;
  • S8: gross = basic + da;
  • }
  • S9: print (gross, empid);
  • End

​

33 of 42

PRIORITIZATION TECHNIQUES

34 of 42

PRIORITIZATION TECHNIQUES

  • Begin
  • S1: read (basic, empid);
  • S2: gross=0;
  • S3: if(basic > 5000 || empid > 0)
  • {
  • S4: da = (basic*30)/100;
  • S5: gross = basic + da;
  • }
  • S6: else
  • {
  • S7: da = (basic*15)/100;
  • S8: gross = basic + da;
  • }
  • S9: print(gross, empid);
  • End

execution slice is highlighted in the given code segment for T2.

​

35 of 42

PRIORITIZATION TECHNIQUES

  • if there is a change in S7 statement

[da = (basic*25)/100;

instead of da = (basic*15)/100;],

then only T 2 will be rerun. So in the execution slice, we will have less number of statements.

​

​

36 of 42

  • Dynamic Slice
  • The set of statements executed under a test case having an effect on the program output is called the dynamic slice of the program with respect to the output variables.

37 of 42

PRIORITIZATION TECHNIQUES

  • Begin
  • S1: read (a,b);
  • S2: sum=0;
  • S2.1: I=0;
  • S3: if (a==0)
  • {
  • S4: print(b);
  • S5: sum+=b;
  • }
  • S6: else if(b==0)
  • {
  • S7: print(a);

​

  • S8: sum+=a;
  • }
  • S9: else
  • {
  • S10: sum=a+b+sum;
  • S10.1 I=25;
  • S10.2 print(I);
  • }
  • S11:endif
  • S12:print(sum);
  • End

​

38 of 42

PRIORITIZATION TECHNIQUES

39 of 42

PRIORITIZATION TECHNIQUES

  • Begin
  • S1: read (a,b);
  • S2: sum = 0;
  • S2.1: I=0;
  • S3: if (a==0)
  • {
  • S4: print(b);
  • S5: sum+=b;
  • }
  • S6: elseif(b==0)
  • {
  • S7: print(a);

​

  • S8: sum+=a;
  • }
  • S9: else
  • {
  • S10: sum=a+b+sum;
  • S10.1 I=25;
  • S10.2 print(I);
  • }
  • S11: endif
  • S12: print(sum);
  • End

40 of 42

PRIORITIZATION TECHNIQUES

  • if S 10 is changed [say, sum=a*b+sum], then this change will affect the output variable ‘sum’, so there is a need to rerun T 3.

​

  • T 1, T 2, and T 3 will run correctly but if some modification is done in S 10.1 [say I = 50], then this change will not affect the output variable.
  • Therefore, there is no need to rerun any of the test cases.

​

41 of 42

PRIORITIZATION TECHNIQUES

  • Relevant Slice
  • The set of statements that were executed under a test case and did not affect the output of program, but have the potential to affect the output produced by a test case, is known as the relevant slice of the program.
  • It contains the dynamic slice and other statements which, if corrected, may modify the output variables of the program.

42 of 42

PRIORITIZATION TECHNIQUES

  • If there is a change in any statement in the relevant slice, we need to rerun the modified software on only those test cases whose relevant slices contain a modified statement.
  • Thus, on the basis of relevant slices, we can prioritize the test cases.