UNIT-6

Software quality

• We use the term quality very often in our daily life.
• But how do we define quality?
• Crosby defines quality as conformance to requirements.
• This means that the product should be developed according to the pre-specified requirements.
• If the requirements are misunderstood and the product is developed with incorrect requirements, then the product lacks quality.

Software quality

• Juran & Gryna defines quality as fitness for use.
• It means whether the product is actually usable by the user or not.
• If the user is satisfied with the product, it is of good quality.
• Quality is not a single idea; rather it is a multi-dimensional concept,

Software quality

On the basis of the multi-dimensional concept, quality can be defined as ‘ the degree to which a product or service possesses a desired combination of attributes ’.

​

Software quality

• The concept of software quality also differs from physical products.
• Software has many possible quality characteristics.
• It is commonly recognized as lack of bugs in the software.
• If a software has functional bugs, then it may be possible that the software is not able to perform the functions.

BROADENING THE CONCEPT OF QUALITY

• Software has been considered a product and in quality sense, we expect all its attributes to improve.
• The concept of quality can be extended to the process adopted for development of the product.
• There are two views of quality.

Product Quality

• Quality measures are taken to improve the end-product.
• product quality is according to the customer’s point of view.

BROADENING THE CONCEPT OF QUALITY

• Process Quality
• If the quality of development process is not good enough, then it will certainly affect the quality of the end-product.
• If each stage of the development process is developed according to the requirement of its intermediate user, then the final end product will be of high quality.

BROADENING THE CONCEPT OF QUALITY

BROADENING THE CONCEPT OF QUALITY

• process quality is according to the intermediate user of the next stage.
• For example, the design person will use SRS.
• All the intermediate stages of development process are being developed for a high-quality end product.

​

​

BENEFITS OF INVESTMENT ON QUALITY

• Costs incurred for quality is decomposed into three major categories:
• Prevention costs
• Appraisal costs
• Failure costs
• If we invest in prevention and appraisal activities at an early stage, then failures and failure-costs reduce.

​

BENEFITS OF INVESTMENT ON QUALITY

By implementing quality evaluation programs, the following benefits are achieved:

1. Customer is satisfied, as the end-product is of high quality.
2. Productivity increases due to shorter SDLC cycle.
3. Failures and failure costs reduce.
4. Rework and cost of quality reduce.

QUALITY CONTROL AND QUALITY ASSURANCE

• It is clear that for a high-quality software, the removal of defects is necessary.
• This removal of defects takes place when quality control activities are adopted.
• According to Pankaj Jalote , quality control focuses on finding and removing defects, whereas the main purpose of quality assurance is to verify that applicable procedures and standards are being followed.

​

QUALITY CONTROL AND QUALITY ASSURANCE

• Quality control is basically related to software product
• Variation according to desired one is checked at each step of development.
• Quality control may include the following activities: reviews, testing using manual techniques or with automated tools ( V&V).
• Quality control involves a series of static and dynamic testing throughout the process, such that all the stages produce outputs according to the desired requirements.

​

QUALITY CONTROL AND QUALITY ASSURANCE

• Quality assurance is largely related to the process.
• Auditing and reporting of quality, based on quantitative measurements, are performed.
• The goal is to inform the management about quality-related issues and supply relevant data.
• Thus, software quality assurance ( SQA) is defined as a planned and systematic approach to evaluate quality and maintain software product standards, and procedures

QUALITY CONTROL AND QUALITY ASSURANCE

• SQA includes the process of assuring that standards and procedures are established and are followed throughout the software acquisition life cycle.

QUALITY CONTROL AND QUALITY ASSURANCE

• Quality assurance may include the following activities:
• Measuring the products and processes.
• Auditing the quality of every step in SDLC and the end-product.
• Verifying that the adopted procedures and standards are being followed.

QUALITY FACTORS

• Quality is not a single idea; rather it is a combination of many factors.
• These factors are based on the project and the environment in which the software is to be developed.
• Functionality The major visible factor for software quality is its functionality such that all the intended features are working.
• Correctness Software should be correct satisfying specifications and fulfilling user’s objectives. Another measure of correctness may be the extent to which a software is fault-free.

QUALITY FACTORS

• Completeness A software product should be complete to the extent that all its parts are present and fully developed.
• Efficiency
• The software should be efficient in its working, i.e. utilizing all resources.
• Portability
• A software product should be operated on all well-known platforms and configurations.

QUALITY FACTORS

• Testability
• A software should be testable such that it is easy to debug and maintain.
• Usability
• The software should be easy to use, operate, and understand in every manner that its user wants.

QUALITY FACTORS

• Reliability
• A software product should be reliable in the sense that all its desired features must run forever without bugs; it should not stop after working for some years.
• This implies a time factor in which a reliable product is expected to perform correctly over a period of time.

QUALITY FACTORS

• Integrity
• It defines the extent to which an unauthorized access or a modification can be controlled in a computer system (either manually or with automated tools).
• Maintainability
• The software should have the scope of maintaining it, if a bug appears or a new requirement needs to be incorporated.

SOFTWARE QUALITY METRICS

• Software quality metrics focus on the quality aspects of the product, process, and project.
• In general, software quality metrics are more closely associated with process and product metrics than with project metrics.
• software quality metrics can be grouped into the following three categories
• product quality metrics,
• in-process quality metrics
• metrics for software maintenance.

SOFTWARE QUALITY METRICS

Product Quality Metrics

1 . Mean-time to failure ( MTTF)

• MTTF metric is an estimate of the average or mean time until a product’s first failure occurs.
• It is most often used with safety critical systems such as the airline traffic control system.
• It is calculated by dividing the total operating time of the units tested by the total number of failures encountered.

SOFTWARE QUALITY METRICS

2. Defect density metrics

• It measures the defects relative to the software size.
• Defect density = Number of defects / Size of product
• Defect rate metrics or the volume of defects generally used for commercial software systems.
• It is important for cost and resource estimate of the maintenance phase.

SOFTWARE QUALITY METRICS

3.Customer satisfaction metrics

• Customer satisfaction is usually measured through various methods of customer surveys via the five-point scale.

1. Very satisfied

2. Satisfied

3. Neutral

4. Dissatisfied

5. Very dissatisfied

SOFTWARE QUALITY METRICS

4 . Customer problem metrics

• This metric measures the problems which customers face while using the product. The problems may be valid defects or usability problems, unclear documentation, etc.
• The problem metrics is usually expressed in terms of problems per user month ( PUM), as given below:

​

• PUM = Total problems reported by the customer for a time period

Total number of licensed months of the software during the period

SOFTWARE QUALITY METRICS

• In-process Quality Metrics
• In-process quality metrics are less formally defined than the end-product metrics.

1. Defect-density during testing

• Higher defect rates found during testing is an indicator that the software has experienced higher error injection during its development process.
• This metric is a good indicator of quality while the software is still being tested.

SOFTWARE QUALITY METRICS

2. Defect-arrival pattern during testing

• The pattern of defect arrivals or the time between consecutive failures gives more information.
• Different patterns of defect arrivals indicate different quality levels in the field.
• The objective is to have defect arrivals that stabilize at a very low level or times between failures is far apart.

SOFTWARE QUALITY METRICS

3. Defect-removal efficiency

​

SOFTWARE QUALITY METRICS

Metrics for Software Maintenance

1. Fix backlog and backlog management index

• Fix backlog metrics is the count of reported problems that remain open at the end of a month or a week.
• Backlog management index ( BMI) is also the metric to manage the backlog of unresolved problems.

​

SOFTWARE QUALITY METRICS

SOFTWARE QUALITY METRICS

​

• If BMI is larger than 100, it means the backlog is reduced;
• if BMI is less than 100, the backlog is increased. Therefore, the goal is to have a BMI larger than 100.

​

SOFTWARE QUALITY METRICS

2. Fix response time

• The fix response time metric calculated as mean-time of high severity defects from open to closed.
• Thus, a short fix response time leads to customer satisfaction.

SOFTWARE QUALITY METRICS

3. Fix quality

• It is the metric to measure the number of fixes that turn out to be defective.
• A fix is defective if it did not fix the reported problems or injected a new defect as a consequence of fixing.
• Fix quality is measured as a percent defective fixes.
• Percent defective fixes is the percentage of all fixes in a time interval that are defective.

NEED FOR AUTOMATION

• The testing techniques, either static or dynamic, can be automated with the help of software tools which can greatly enhance the software testing process.
• Benefits of automation
• Reduction of testing effort
• Test cases for a complete software may be of thousands or more in number. Executing all of them manually takes a lot of testing effort and time. Thus, execution of test suites through software tools greatly reduces the amount of time required.

​

NEED FOR AUTOMATION

• Facilitates regression testing
• Regression testing is the most time consuming process. If we automate the process of regression testing, then testing effort as well as the time taken will reduce as compared to manual testing.
• Avoids human mistakes
• Manually executing the test cases may incorporate errors in the process. Testing tools will not cause these problems which are introduced due to manual testing.

NEED FOR AUTOMATION

• Reduces the testers’ involvement in executing tests
• Sometimes executing the test cases takes a long time. Automating this process of executing the test suit will relieve the testers to do some other work.
• Reduces overall cost of the software
• If testing time increases, cost of the software also increases. But due to testing tools, time and cost can be reduced as they ease the burden of the test case production and execution.

NEED FOR AUTOMATION

• Simulated testing
• Load performance testing is an example of testing where the real-life situation needs to be simulated in the test environment.
• Sometimes, it may not be possible to create the load of number of concurrent users or large amount of data in a project.
• Automated tools, can create millions of concurrent virtual users/data and effectively test the project in the test environment before releasing the product.

NEED FOR AUTOMATION

• Test enablers
• While development is not complete, some modules for testing are not ready. At that time, stubs or drivers are needed to prepare data, simulate environment, make calls, and then verify results.
• Automation reduces the effort required in this case and becomes essential.
• Test case design
• Automated tools can be used to design test cases also.

CATEGORIZATION OF TESTING TOOLS

• A variety of testing tools are available according to different needs.

​

• STATIC TESTING TOOLS
• DYNAMIC TESTING TOOLS
• TESTING ACTIVITY TOOLS

​

STATIC TESTING TOOLS

• STATIC TESTING TOOLS
• For static testing, there are static program analysers which scan the source program and detect possible faults and anomalies.
• Static tools perform the following types of static analysis:
• Control flow analysis
• This analysis detects loops with multiple exits and entry points and unreachable code.

STATIC TESTING TOOLS

• Data use analysis
• It detects all types of data faults.
• Interface analysis
• It also detects functions which are never declared and never called or function results that are never used.
• Path analysis
• It identifies all possible paths through the program.

DYNAMIC TESTING TOOLS

• Supports Dynamic testing activities.
• Automated test tools enable the test team to capture the state of events during the execution of a program by preserving a snapshot of the conditions. These tools are called program monitors.
• The monitors perform the following functions:
• List the number of times a component is called or line of code is executed. i.e used for statement or path coverage of test cases

​

DYNAMIC TESTING TOOLS

• Report on whether a decision point has branched in all directions, thereby providing information about branch coverage.
• Report summary statistics providing a high-level view of the percentage of statements, paths, and branches that have been covered by the collective set of test cases run.

TESTING ACTIVITY TOOLS

• These tools are based on the testing activities or tasks in a particular phase of the SDLC.
• Testing activities can be categorized as :
• Reviews and inspections
• Test planning
• Test design and development
• Test execution and evaluation

TESTING ACTIVITY TOOLS

Tools for review and inspections

• Complexity analysis tools
• Complexity analysis tools analyse the areas of complexity and provide indication to testers so that testing time and resources can be estimated.
• Code comprehension
• These tools help in tracing program logic, viewing graphical representations of the program, and identifying the dead code.

TESTING ACTIVITY TOOLS

Tools for test planning

• tools required for test planning are:

1. Templates for test plan documentation.

2. Test schedule and staffing estimates.

3. Complexity analyser.

TESTING ACTIVITY TOOLS

Tools for test design and development

• Test data generator
• generation of test data based on a user defined format.
• Test case generator : uses the information provided by the requirement management tool and creates the test cases.
• The test cases can also be generated with the information provided by the test engineer regarding the previous failures discovered by him.

TESTING ACTIVITY TOOLS

Test execution and evaluation tools

• Capture/playback tools
• These tools record events (including keystrokes, mouse activity, and display output) at the time of running the system and place the information into a script. The tool can then replay the script to test the system.

TESTING ACTIVITY TOOLS

Memory testing tools

• These tools verify that an application is properly using its memory resources. They check whether an application is:
• Not releasing memory allocated to it
• Overwriting/overreading array bounds
• Reading and using uninitialized memory

TESTING ACTIVITY TOOLS

Network-testing tools

• There are various applications running in the client server Environments.
• These tools monitor, measure, test, and diagnose performance across an entire network including the following:
• Cover the performance of the server and the network
• Overall system performance
• Functionality across server, client, and the network

SELECTION OF TESTING TOOLS

• The big question is how to select a testing tool.
• It may depend on several factors
• What are the needs of the organization;
• what is the project environment;
• what is the current testing methodology; all these factors should be considered when choosing testing tools.

SELECTION OF TESTING TOOLS

• Guidelines to be followed for selecting a testing tool are given below.

1. Match the tool to its appropriate use

• Before selecting the tool, it is necessary to know its use.
• Most of the tools are meant for specific tasks.
• Therefore, the tester needs to be familiar with both the tool and its uses in order to make a proper selection.

SELECTION OF TESTING TOOLS

2. Select the tool to its appropriate SDLC phase

• Testing methods change according to the SDLC phase. Therefore, it is necessary to choose the tool according to the SDLC phase, in which testing is to be done.

3. Select the tool to the skill of the tester

• The individual performing the test must select a tool that conforms to his skill level.
• For example, it would be inappropriate for a user to select a tool that requires programming skills when the user does not possess those skills.

SELECTION OF TESTING TOOLS

4. Select a tool which is affordable

• Tools are always costly and increase the cost of the project.
• Therefore, choose the tool within the budget of the project.
• If the tool is under utilization, then added cost will have no benefits to the project.
• Once you are sure that a particular tool will really help the project, then only go for it.

SELECTION OF TESTING TOOLS

5. Determine how many tools are required for testing the system

• A single tool generally cannot satisfy all test requirements.
• Therefore, assess the tool as per the test requirements and determine the number and type of tools required.

SELECTION OF TESTING TOOLS

6. Select the tool after examining the schedule of testing

• First, get an idea of the entire schedule of testing activities and then decide whether there is enough time for learning the testing tool and then performing automation with that tool.
• If there is not enough time to provide training on the tool, then there is no use of automation.

COMMERCIAL TESTING TOOLS

1. Mercury Interactive’s WinRunner

• It is a tool used for performing functional / regression testing.
• It automatically creates the test scripts by recording the user interactions on GUI of the software.
• The test scripts can also be modified if required using Test Script language ( TSL) with a ‘C’ like syntax.

COMMERCIAL TESTING TOOLS

2. IBM Rational SQA Robot

• It is another powerful tool for functional/regression testing.
• Synchronization of test cases with a default delay of 20 seconds is also available.

3. Apache’s JMeter

• This is an open-source software tool used for performance and load testing

COMMERCIAL TESTING TOOLS

4. Segue Software’s SilkTest

• This tool is also for functional/regression testing.
• It supports Test as a scripting language which is an object-oriented scripting language.
• Silk Test has in-built recovery system which helps in continuing the automated testing even if there is some failure in between.

COMMERCIAL TESTING TOOLS

5. Mercury Interactive’s LoadRunner

• This tool is used for performance and load testing of a system.
• Generally, the tool is helpful for client/server applications of various parameters like response time, the number of users, etc.
• The major benefit of using this tool is that it creates virtual users on a single machine and tests the system on various parameters.