�MCA : III Sem �Professional Elective 4�Course code : 22MCA344�Course Title : Software Testing

Test Books : ��1. “Foundations of Software Testing- Fundamental Algorithm and Techniques “ � by Adithya P Mathur.��2. “Software testing and Analysis Process, Principals and Techniques”� by Paul C Jorgensen.

Module 1��Basic of Software Testing, Basic Principles, Test case selection and Adequacy

Software Testing : � * Software Testing is the process of evaluating and verifying that a software product or application does what it is supposed to do.� * Software Testing is a method to check whether the actual software product matches expected requirements and to ensure that software product is Defect free. 

Errors are a part of our daily life. �

Humans make errors in their thoughts, actions, and in the products that might result from their actions.

​

Errors occur wherever humans are involved in taking actions and making decisions.��Error: When make mistakes while coding, we call these mistakes bugs.�

Error: Mistake done in the programmer while writing the code.�Failure : At the customer side, if they are facing some issue.�Defect : Difference between actual behavior and expected behavior.�Bug: Mis-matched output after test by the tester between actual and expected result.

Errors: Examples

Errors, Fault and Failures in the process of programming and testing

Test Automation

​

• Testing of complex systems, embedded and otherwise, can be a human intensive task.
• Most software development organizations, automate test-related tasks such as regression testing, graphical user interface testing, and I/O device driver testing.
• The process of test automation cannot be generalized.

Developers and Testers as two Roles:

Developer is one who writes code & tester is one who tests code. Developer & Tester roles are different and complementary roles. Thus, the same individual could be a developer and a tester. It is hard to imagine an individual who assumes the role of a developer but never that of a tester, and viceversa.

Certainly, within a software development organization, the primary role of a individual might be to test and hence an individual assumes the role of a tester. Similarly, the primary role of an individual who designs applications and writes code is that of a developer.

SOFTWARE QUALITY

​

Software quality is a multidimensional quantity and is measurable.

Quality Attributes

These can be divided to static and dynamic quality attributes.

​

Static Quality Attributes : � It refers to the actual code and related documents.�Dynamic Quality Attributes : � Reliability � Correctness � Completeness � Consistency � Usability � Performance

Reliability: � It refers to the probability of failure free operation. Correctness: �Refers to the correct operation.�For a Tester, correctness is w.r.t to the requirements.�For a user, correctness is w.r.t the user manual .�Completeness: � Refers to the availability of all the features listed in the requirements or in the user manual. � An incomplete software is one that does not fully implement all features required.

Consistency: �Refers to adherence to a common set of conventions and assumptions. �Ex: All buttons in the user interface might follow a common-color coding convention.�Usability: �Refer to case with which an application can be used. This is an area in itself and there exist techniques for usability testing. � Psychology plays an important role in the design of techniques for usability testing. � Usability testing is a testing done by its potential users.

Performance: � Refers to the time the application takes to perform a requested task. Performance is considered as a non functional requirement.

Requirements, Behavior and Correctness�� Requirements leading to two different programs: ��Requirement 1: It is required to write a program that inputs two numbers and outputs the maximum of these.��Requirement 2: It is required to write a program that inputs a�sequence of integers and outputs the sorted version of this sequence.�

Requirements: Incompleteness� � Suppose that program max is developed to satisfy Requirement 1. The expected output of max when the input integers are 13 and 19 can be easily determined to be 19. �� Suppose now that the tester wants to know if the two integers are to be input to the program on one line followed by a carriage return, or on two separate lines with a carriage return typed in after each number. The requirement as stated above fails to provide an answer to this question. ��

Requirements: Ambiguity

​

Requirement 2 is ambiguous. It is not clear whether the input sequence is to sorted in ascending or in descending order. The behavior of sort program, written to satisfy this requirement, will depend on the decision taken by the programmer while writing sort.

​

Input domain (Input space)

​

• The set of all possible inputs to a program P is known as the input domain or input space of P.
• Using Requirement 1 (integer )above we find the input domain of max to be the set of all pairs of integers where each element in the pair integers is in the range -32,768 till 32,767.
• Using Requirement 2 it is not possible to find the input domain for the sort program.

​

Input domain (Continued)

Modified Requirement 2:

• It is required to write a program that inputs a sequence of integers and outputs the integers in this sequence sorted in either ascending or descending order. The order of the output sequence is determined by an input request character which should be ``A'' when an ascending sequence is desired, and ``D'' otherwise.
• While providing input to the program, the request character is input first followed by the sequence of integers to be sorted, the sequence is terminated with a period.

​

Valid/Invalid Inputs

​

• The modified requirement for sort mentions that the

request characters can be ``A'' and ``D'', but fails to answer the question ``What if the user types a different character ?’’

• When using sort it is certainly possible for the user to type a

character other than ``A'' and ``D''. Any character other than ``A'’ and ``D'' is considered as invalid input to sort. The requirement for sort does not specify what action it should take when an invalid input is encountered.

​

Correctness and Testing

​

​

• While correctness attempts to establish that the program is error free, testing attempts to find if there are any errors in it.
• Thus, completeness of testing does not necessarily demonstrate that a program is error free.

​

Testing and debugging�

​

​

• Testing is the process of determining if a program has any errors.

​

• When testing reveals an error, the process used to determine the cause of this error and to remove it, is known as debugging.

​

Test and debug Cycle

���

A sample Test plan for Sort Program

1. Execute the program on at least two input sequences, one with “A” and the other with “D” as request characters.

2. Execute the program on an empty input sequence.

3. Test the program for robustness against error inputs such as “R'' typed in as the request character.

4. All failures of the test program should be recorded in a suitable file using the Company Failure Report Form.

​

​

Test case / data

A test case is a pair consisting of test data to be input to the program and the expected output. The test data is a set of values, one for each input variable.

A test set is a collection of zero or more test cases.

Sample test case for sort:

Test case 1:

Test data: <''A” 12 -29 32. >

Expected output: -29 12 32

​

Test case 2:� Test data: <“D” 12 -29 32. >� Expected output: 32 12 -29��Test case 3:� Test data: <“A”.>� Expected output: No input to be sorted in ascending order.��Test case 4: � Test data : <“D”.>� Excepted output: No input to be sorted in Descending order.�

Test case 5:� Test data: <'‘R” 3 17. >� Expected output: Invalid request character.

Valid characters: “A” and “D”.

�Test case 6:� Test data: <“A” c 17.>� Expected output: Invalid number.

Executing the Program

Execution of a program under test is the next significant step in the testing.

​

Program behavior�

Can be specified in several ways: plain natural language, a state diagram, formal mathematical specification, etc.

​

A state diagram specifies program states and how the program changes its state on an input sequence.

​

A program that input two integers X and Y, compares two values, sets the value of Z to the larger of 2.

Integer X,Y,Z

Input(X, Y)

If(X < Y)

Z=Y

else

Z=X

endif

Output (Z)

end

​

​

Program behavior: Example

Consider a menu driven application.

Program behavior: Example

Behavior: Observation and analysis

​

• In the first step, observe the behavior.
• In the second step, analyzes the observed behavior to check if it is correct or not. Both these steps could be quite complex for large commercial programs.
• The entity that performs the task of checking the correctness of the observed behavior is known as an oracle.

​

Oracle: Example

Relationship between the program under test and the oracle.

Oracle: Programs

​

​

• Oracles can also be programs designed to check the behavior of other programs.
• For example, one might use a matrix multiplication program to check if a matrix inversion program has produced the correct output. In this case, the matrix inversion program inverts a given matrix A and generates B as the output matrix.

Test Oracles can be manual or automated.��

• Manual test oracles relay on human experts to evaluate the correctness of software application behavior.
• Automated test oracles use software tools and algorithms to compare the actual output with the excepted output of the test cases.

Oracle: Construction

​

• Construction of automated oracles, such as the one to check a matrix multiplication program or a sort program, requires the determination of input-output relationship.

​

• In general, the construction of automated oracles is a complex undertaking.

​

Test Matrix

• The term matrix refers to a standard of measurement.
• Matrix can be computed at the organizational, process and project levels. Each set of measurements has its value in monitoring , planning and control.

Organizational Metrics

Organizational level metrics allow senior management to monitor the overall strength of the organization and points to area of weakness.

Project Metrics

• Project metrics relate to a specific project.
• Another project metric is the ratio of the number of successful tests to the total number of tests in the system test phase.

​

Process Metrics � � Every project uses some test process. The big bang approach is one process some times used in relatively small single –person projects.�The goal of a process metrics is to access the goodness of the process.�

Testability�� According to IEEE, testability is the “degree to which a system or component facilitates the establishment of test criteria and the performance of tests to determine whether those criteria have been met”. �Two types:� 1. Static testability metrics � 2. Dynamic testability metrics.

Static testability metric: � Static metrics are those computed without having to execute the product code.�Dynamic metrics: � Dynamic metrics requires code execution. �Ex: when it is difficult to generate tests that satisfy the statement coverage criterion is considered to have low testability them one for which it is easier to construct such tests.

Software and Hardware testing

Two input NAND Gate

Testing and verification

• Program verification aims at proving the correctness of programs by showing that it contains no errors. This is very different from testing that aims at uncovering errors in a program.
• Program verification and testing are best considered as complementary techniques. In practice, program verification is often avoided, and the focus is on testing.

​

DEFECT MANAGEMENT� Defect Management is an integral part of a development and test process in many software development organizations. It is a sub process of a the development process. �It includes the following: � Detect prevention � Discovery � Recording and reporting � Classification � Resolution � Production

Defect Prevention: It is achieved through a variety of process and tools.

They are

Good coding techniques.

Unit test plans.

Code Inspections.

Defect Discovery:

Defect discovery is the identification of defects in response to failures observed during dynamic testing or found during static testing. It involves debugging the code under test.

​

​

​

​

Defect Classification : Defects found are classified and recorded in a database. Classification becomes important in dealing with the defects. Classified into

High severity-to be attended first by developer

Low severity.

Resolution : Each defect, when recorded, is marked as ‘open’ indicating that it needs to be resolved. It required careful scrutiny of the defects, identifying a fix if needed, implementing the fix, testing the fix, and finally closing the defect indicating that every recorded defect is resolved prior to release.

Defect Prediction

Organizations often do source code Analysis to predict how many defects an application might contain before it enters the testing the phase.

Advanced statistical techniques are used to predict defects during the test process.

Tools are existing for Recording defects, and computing and reporting defect related statistics.

BugZilla - Open source

Fog-Buzz - commercially available tools.

EXECUTION HISTORY

​

Execution history of a program, also known as execution trace, is an organized collection of information about various elements of a program during a given execution. An execution slice is an executable subsequence of execution history. There are several ways to represent an execution history.

• Sequence in which the functions in a given program are executed against a given test input.
• Sequence in which program blocks are executed.
• Sequence of objects and the corresponding methods accessed for object oriented languages such as Java.
• An execution history may also included values of program variables.

TEST GENERATION STRATEGIES

​

• Model based test generation (Subset of requirements are used and represent in terms of graphs)

​

• Specification based test generation (Use mathematical notation)

​

• Code based test generation (direct source code is used)

TEST GENERATION STRATEGIES

STATIC TESTING

• Static testing is carried out without executing the application under test.
• This is in contrast to dynamic testing that requires one or more executions of the application under test.
• It is useful in that it may lead to the discovery of faults in the application, ambiguities and errors in the requirements and other application-related document, at a relatively low cost, This is especially so when dynamic testing expensive.

Static testing is complementary to dynamic testing. This is carried out by an individual who did not write the code or by a team of individuals.

The test team responsible for static testing has access to requirements document, application, and all associated documents such as design document and user manual. Team also has access to one or more static testing tools. A static testing tool takes the application code as input and generates a variety of data useful in the test process.

WALKTHROUGHS

​

Walkthroughs and inspections are an integral part of static testing.

Walkthroughs is an informal process to review an application related document.

Walkthrough are an integral part of static testing.

Walkthrough are two types

​

1. Requirement walkthrough.
1. Review plan
2. Based on plan, testing
3. After testing preparation of detailed report

2. Code walkthrough.

Inspection

• Inspection is a more formally defined process than a walkthrough. This term is usually associated with code.
• Several organizations consider formal code inspections as a tool to improve code quality at a lower cost than incurred when dynamic testing is used.

Inspection plan:

​

• Statement of purpose
• Work product to be inspected (this includes code and associated documents needed for inspection.)
• Team formation, roles, and tasks to be performed.
• Rate at which the inspection task is to be completed.
• Data collection and time spent in each task.

Members of inspection team :

1. Moderator: in charge of the process and leads the review.
2. Leader: actual code is read by the reader, perhaps with help of a code browser and with monitors for all in the team to view the code.
3. Recorder: records any errors discovered or issues to be looked into.
4. Author: actual developer of the code.

�Use of static code analysis tools in static testing �

• Static code analysis tools can be provide control flow and data flow information.
• Control flow information presented in terms of a CFG, is helpful to the inspection team in that it allows the determination of the flow of control under different conditions.

• A CFG can be converted with data flow information to make a data flow graph.

​

• This information is valuable to the inspection team in understanding the code as well as pointing out possible defect.

Test Generation from Predicates

A techniques for generating tests that are guaranteed to detect certain faults in the coding of condition.

A condition is represented formally as a predicates.

Ex: “If the printer is ON and has paper then send the document for printing “

Predicate is denoted by p_r, represents the condition part of the statement.

P_r: (printer_status=ON) ^ (printer_tray=¬empty)

​

• We are interested in generating the test cases from predicates such that any fault that belongs to a class of faults is guaranteed to be detected during testing.
• A condition can be represented as simple predicate or a compound predicate.

A Simple predicate is a Boolean variable or a relational expression where one or more variable might be negated.�Ex: � q� q^r� a+b<c��A Compound predicate is a expression formed by joining two or more simple predicates.�Ex: (a+b<c)^(¬p)�

Fault model for predicate testing

• An incorrect Boolean operator is used.
• An incorrect Boolean variable is used.
• Missing or extra Boolean variables are added.
• An incorrect relational operators are used.
• Parentheses are used incorrectly.

Path predicate

The predicate which is associated with the path is knows as path predicates.

Path predicate expression is a set of Boolean expression which will give direction to specific path.

Path can be formed from number of goto statement, jump statements, switch statement etc.

​

​

Test Adequacy Criteria�� Adequacy criterion: A test adequacy criterion is a predicate that is true (satisfied) or false.� Usually a test adequacy criterion is expressed in the form of a rule for deriving a set of test rules from another program or specification.�

The goal here is to ensure internal components of the programs are working properly.

Coverages :

• Path coverage
• Statement coverage
• Branch /Decision coverage

​

Path coverage

Covers all the paths.

Path coverage refers to designing test cases such that all lineally independent path are executed at least once.

​

Statement coverage

Statement coverage is a white box testing technique, which involves execution of all the statements at least once in a source code.

Branch coverage

Branch or decision coverage which aims to ensure that each one of the possible branch from each decision point is executed at least once.

Sensitivity

• Sensitivity principle states that it is better to fail every time than sometimes.
• Cost of detecting and repairing the software fault immediately, then the cost of correction is very small.
• If a fault is detected in unit testing, the cost is still relatively small. If it detects at integration testing, the cost of correction is much greater.

The sensitivity principle says that we should try to make faults easier to detect by making them cause failure more often.�� It can be applied in 3 ways :� 1. At design Level� 2. Analysis and testing level� 3. Environment level�

Redundancy

• Redundancy is the opposite of independence. In communication, redundancy can be introduced into messages in the form of error detecting and error correcting code to guard against transmission errors.

Restriction

• When there are no acceptably cheap and effective ways to check a property, sometimes one can change the problem by checking a different, more restrictive property or by limiting the check to a smaller, more restrictive class of programs.

​

Restriction principle is useful mainly during design and specification. It can be applied not only to solve a single problem but can also used in architectural level to simplify a whole set of analysis problems.

Partition

• Partition is often known as “ Divide and Conquer “ .
• Dividing the complex problem into sub problems and solve independently.
• The partition principle is widely used and exploited.
• Partitioning can be applied both at process and technique level.

Visibility �.

• Visibility means the ability to measure progress or status against goals

Feedback

• Feedback is another classic principle that applies to analysis and testing. Feedback applies both to the process itself and to individual techniques.
• Systematic inspection and walkthrough derive part of their success from feedback.
• Participants in inspection are guided by checklists, and checklists are revised and refined based on experience.

​

Sensitivity: better to fail every time than sometimes.�Redundancy : making intentions explicit .�Restriction: making the problem easier.�Partition: divide and conquer.�Visibility : measure progress Vs goal.�Feedback :applying lessons from experience in process and techniques.