Software Risk Identification

1

Project Risks

2

What can go wrong?

​

What is the likelihood?

​

What will the damage be?

​

What can we do about it?

Reactive Risk Management

• project team reacts to risks when they occur
• mitigation—plan for additional resources in anticipation of fire fighting
• fix on failure—resource are found and applied when the risk strikes
• crisis management—failure does not respond to applied resources and project is in jeopardy

3

Proactive Risk Management

• formal risk analysis is performed
• organization corrects the root causes of risk
• TQM concepts and statistical SQA
• examining risk sources that lie beyond the bounds of the software
• developing the skill to manage change

4

Seven Principles

• Maintain a global perspective—view software risks within the context of system and the business problem
• Take a forward-looking view—think about the risks that may arise in the future; establish contingency plans
• Encourage open communication—if someone states a potential risk, don’t discount it.
• Integrate—a consideration of risk must be integrated into the software process
• Emphasize a continuous process—the team must be vigilant throughout the software process, modifying identified risks as more information is known and adding new ones as better insight is achieved.
• Develop a shared product vision—if all stakeholders share the same vision of the software, it likely that better risk identification and assessment will occur.
• Encourage teamwork—the talents, skills and knowledge of all stakeholder should be pooled

5

Risk Management Paradigm

6

RISK

control

identify

analyze

plan

track

Risk Identification

• Product size—risks associated with the overall size of the software to be built or modified.
• Business impact—risks associated with constraints imposed by management or the marketplace.
• Customer characteristics—risks associated with the sophistication of the customer and the developer's ability to communicate with the customer in a timely manner.
• Process definition—risks associated with the degree to which the software process has been defined and is followed by the development organization.
• Development environment—risks associated with the availability and quality of the tools to be used to build the product.
• Technology to be built—risks associated with the complexity of the system to be built and the "newness" of the technology that is packaged by the system.
• Staff size and experience—risks associated with the overall technical and project experience of the software engineers who will do the work.

7

Assessing Project Risk-I

• Have top software and customer managers formally committed to support the project?
• Are end-users enthusiastically committed to the project and the system/product to be built?
• Are requirements fully understood by the software engineering team and their customers?
• Have customers been involved fully in the definition of requirements?
• Do end-users have realistic expectations?

8

Assessing Project Risk-II

• Is project scope stable?
• Does the software engineering team have the right mix of skills?
• Are project requirements stable?
• Does the project team have experience with the technology to be implemented?
• Is the number of people on the project team adequate to do the job?
• Do all customer/user constituencies agree on the importance of the project and on the requirements for the system/product to be built?

9

Risk Components

• performance risk—the degree of uncertainty that the product will meet its requirements and be fit for its intended use.
• cost risk—the degree of uncertainty that the project budget will be maintained.
• support risk—the degree of uncertainty that the resultant software will be easy to correct, adapt, and enhance.
• schedule risk—the degree of uncertainty that the project schedule will be maintained and that the product will be delivered on time.

10

Risk Projection

• Risk projection, also called risk estimation, attempts to rate each risk in two ways
• the likelihood or probability that the risk is real
• the consequences of the problems associated with the risk, should it occur.
• The are four risk projection steps:
• establish a scale that reflects the perceived likelihood of a risk
• delineate the consequences of the risk
• estimate the impact of the risk on the project and the product,
• note the overall accuracy of the risk projection so that there will be no misunderstandings.

11

Building a Risk Table

12

Risk

Probability

Impact

RMMM

Risk

Mitigation

Monitoring

&

Management

Building the Risk Table

• Estimate the probability of occurrence
• Estimate the impact on the project on a scale of 1 to 5, where
• 1 = low impact on project success
• 5 = catastrophic impact on project success
• sort the table by probability and impact

13

Risk Exposure (Impact)

14

The overall risk exposure, RE, is determined using the following relationship [Hal98]:

RE = P x C

where

P is the probability of occurrence for a risk, and

C is the cost to the project should the risk occur.

​

Risk Exposure Example

• Risk identification. Only 70 percent of the software components scheduled for reuse will, in fact, be integrated into the application. The remaining functionality will have to be custom developed.
• Risk probability. 80% (likely).
• Risk impact. 60 reusable software components were planned. If only 70 percent can be used, 18 components would have to be developed from scratch (in addition to other custom software that has been scheduled for development). Since the average component is 100 LOC and local data indicate that the software engineering cost for each LOC is $14.00, the overall cost (impact) to develop the components would be 18 x 100 x 14 = $25,200.
• Risk exposure. RE = 0.80 x 25,200 ~ $20,200.

15

Risk Mitigation, Monitoring,�and Management

• mitigation—how can we avoid the risk?
• monitoring—what factors can we track that will enable us to determine if the risk is becoming more or less likely?
• management—what contingency plans do we have if the risk becomes a reality?

16

Risk Due to Product Size

17

• estimated size of the product in LOC or FP?

​

​

​

• estimated size of product in number of programs,

files, transactions?

​

​

​

• percentage deviation in size of product from

average for previous products?

​

​

​

• size of database created or used by the product?

​

​

​

• number of users of the product?

​

​

​

• number of projected changes to the requirements

for the product? before delivery? after delivery?

​

​

​

• amount of reused software?

​

Attributes that affect risk:

Risk Due to Business Impact

18

• affect of this product on company revenue?

​

• visibility of this product by senior management?

​

• reasonableness of delivery deadline?

​

• number of customers who will use this product

• interoperability constraints

• sophistication of end users?

​

• amount and quality of product documentation that

must be produced and delivered to the customer?

​

• governmental constraints

​

​

• costs associated with late delivery?

​

• costs associated with a defective product?

Attributes that affect risk:

Risks Due to the Customer

19

• Have you worked with the customer in the past?

​

• Does the customer have a solid idea of requirements?

• Has the customer agreed to spend time with you?

• Is the customer willing to participate in reviews?

​

• Is the customer technically sophisticated?

• Is the customer willing to let your people do their

job—that is, will the customer resist looking over your

shoulder during technically detailed work?

​

• Does the customer understand the software

engineering process?

Questions that must be answered:

Risks Due to Process Maturity

20

• Have you established a common process framework?

• Is it followed by project teams?

​

• Do you have management support for

software engineering

• Do you have a proactive approach to SQA?

​

• Do you conduct formal technical reviews?

​

• Are CASE tools used for analysis, design and

testing?

​

• Are the tools integrated with one another?

​

• Have document formats been established?

Questions that must be answered:

Technology Risks

21

• Is the technology new to your organization?

​

• Are new algorithms, I/O technology required?

• Is new or unproven hardware involved?

​

• Does the application interface with new software?

• Is a specialized user interface required?

​

• Is the application radically different?

• Are you using new software engineering methods?

• Are you using unconventional software development

methods, such as formal methods, AI-based approaches,

artificial neural networks?

​

• Are there significant performance constraints?

​

• Is there doubt the functionality requested is "do-able?"

Questions that must be answered:

Staff/People Risks

22

• Are the best people available?

​

• Does staff have the right skills?

​

• Are enough people available?

​

• Are staff committed for entire duration?

​

• Will some people work part time?

• Do staff have the right expectations?

• Have staff received necessary training?

​

• Will turnover among staff be low?

Questions that must be answered:

Recording Risk Information

23

Project: Embedded software for XYZ system

Risk type: schedule risk

Priority (1 low ... 5 critical): 4

Risk factor: Project completion will depend on tests which require

hardware component under development. Hardware component

delivery may be delayed

Probability: 60 %

Impact: Project completion will be delayed for each day that

hardware is unavailable for use in software testing

Monitoring approach:

Scheduled milestone reviews with hardware group

Contingency plan:

Modification of testing strategy to accommodate delay using

software simulation

Estimated resources: 6 additional person months beginning in July

​

​

Software Configuration Management

24

The “First Law”

25

No matter where you are in the system life cycle, the system will change, and the desire to change it will persist throughout the life cycle.

​

Bersoff, et al, 1980

What Are These Changes?

26

data

other

documents

code

Test

Project

​

Plan

changes in

​

technical requirements

changes in

​

business requirements

changes in

​

user requirements

software models

The Software Configuration

27

programs

documents

data

The pieces

Baselines

• The IEEE (IEEE Std. No. 610.12-1990) defines a baseline as:
• A specification or product that has been formally reviewed and agreed upon, that thereafter serves as the basis for further development, and that can be changed only through formal change control procedures.
• a baseline is a milestone in the development of software that is marked by the delivery of one or more software configuration items and the approval of these SCIs that is obtained through a formal technical review

28

Baselines

29

Software Configuration Objects

30

SCM Repository

• The SCM repository is the set of mechanisms and data structures that allow a software team to manage change in an effective manner
• The repository performs or precipitates the following functions [For89]:
• Data integrity
• Information sharing
• Tool integration
• Data integration
• Methodology enforcement
• Document standardization

31

Repository Content

32

Repository Features

• Versioning.
• saves all of these versions to enable effective management of product releases and to permit developers to go back to previous versions
• Dependency tracking and change management.
• The repository manages a wide variety of relationships among the data elements stored in it.
• Requirements tracing.
• Provides the ability to track all the design and construction components and deliverables that result from a specific requirement specification
• Configuration management.
• Keeps track of a series of configurations representing specific project milestones or production releases. Version management provides the needed versions, and link management keeps track of interdependencies.
• Audit trails.
• establishes additional information about when, why, and by whom changes are made.

33

SCM Elements

• Component elements—a set of tools coupled within a file management system (e.g., a database) that enables access to and management of each software configuration item.
• Process elements—a collection of procedures and tasks that define an effective approach to change management (and related activities) for all constituencies involved in the management, engineering and use of computer software.
• Construction elements—a set of tools that automate the construction of software by ensuring that the proper set of validated components (i.e., the correct version) have been assembled.
• Human elements—to implement effective SCM, the software team uses a set of tools and process features (encompassing other CM elements)

34

The SCM Process

• How does a software team identify the discrete elements of a software configuration?
• How does an organization manage the many existing versions of a program (and its documentation) in a manner that will enable change to be accommodated efficiently?
• How does an organization control changes before and after software is released to a customer?
• Who has responsibility for approving and ranking changes?
• How can we ensure that changes have been made properly?
• What mechanism is used to appraise others of changes that are made?

​

35

Addresses the following questions …

The SCM Process

36

Version Control

• Version control combines procedures and tools to manage different versions of configuration objects that are created during the software process
• A version control system implements or is directly integrated with four major capabilities:
• a project database (repository) that stores all relevant configuration objects
• a version management capability that stores all versions of a configuration object (or enables any version to be constructed using differences from past versions);
• a make facility that enables the software engineer to collect all relevant configuration objects and construct a specific version of the software.
• an issues tracking (also called bug tracking) capability that enables the team to record and track the status of all outstanding issues associated with each configuration object.

37

Change Control

38

Change Control Process—I

39

Change Control Process-II

40

assign people to SCIs

check-out SCIs

make the change

review/audit the change

establish a “baseline” for testing

change control process—III

Change Control Process-III

41

perform SQA and testing activities

promote SCI for inclusion in next release

rebuild appropriate version

review/audit the change

include all changes in release

check-in the changed SCIs

Auditing

42

SCIs

Change

Requests

SQA

Plan

SCM Audit

Status Accounting

43

SCIs

Change

Requests

Change

Reports

ECOs

Status Accounting

Reporting

SCM for Web and Mobile Engineering-I

• Content.
• A typical Web or Mobile App contains a vast array of content—text, graphics, applets, scripts, audio/video files, forms, active page elements, tables, streaming data, and many others.
• The challenge is to organize this sea of content into a rational set of configuration objects (Section 29.2.1) and then establish appropriate configuration control mechanisms for these objects.
• People.
• Because a significant percentage of Web and Mobile App development continues to be conducted in an ad hoc manner, any person involved in the App can (and often does) create content.

44

SCM for Web and Mobile Engineering-II

• Scalability.
• As size and complexity grow, small changes can have far-reaching and unintended affects that can be problematic. Therefore, the rigor of configuration control mechanisms should be directly proportional to application scale.
• Politics.
• Who ‘owns’ a App?
• Who assumes responsibility for the accuracy of the information displayed by the App?
• Who assures that quality control processes have been followed before information is published to the site?
• Who is responsible for making changes?
• Who assumes the cost of change?

45

Content Management-I

• The collection subsystem encompasses all actions required to create and/or acquire content, and the technical functions that are necessary to
• convert content into a form that can be represented by a mark-up language (e.g., HTML, XML)
• organize content into packets that can be displayed effectively on the client-side.
• The management subsystem implements a repository that encompasses the following elements:
• Content database—the information structure that has been established to store all content objects
• Database capabilities—functions that enable the CMS to search for specific content objects (or categories of objects), store and retrieve objects, and manage the file structure that has been established for the content
• Configuration management functions—the functional elements and associated workflow that support content object identification, version control, change management, change auditing, and reporting.

46

Content Management-II

• The publishing subsystem extracts from the repository, converts it to a form that is amenable to publication, and formats it so that it can be transmitted to client-side browsers. The publishing subsystem accomplishes these tasks using a series of templates.
• Each template is a function that builds a publication using one of three different components [BOI02]:
• Static elements—text, graphics, media, and scripts that require no further processing are transmitted directly to the client-side
• Publication services—function calls to specific retrieval and formatting services that personalize content (using predefined rules), perform data conversion, and build appropriate navigation links.
• External services—provide access to external corporate information infrastructure such as enterprise data or “back-room” applications.

47

Content Management - III

48

Change Management for Web and Mobile Apps-I

49

Change Management for Web and Mobile Apps-II

50

Software Quality Assurance

51

Comment on Quality

• Phil Crosby once said:
• The problem of quality management is not what people don't know about it. The problem is what they think they do know . .

​

• Everybody is for it. (Under certain conditions, of course.)
• Everyone feels they understand it. (Even though they wouldn't want to explain it.)
• Everyone thinks execution is only a matter of following natural inclinations. (After all, we do get along somehow.)
• And, of course, most people feel that problems in these areas are caused by other people. (If only they would take the time to do things right.)

52

Elements of SQA

• Standards
• Reviews and Audits
• Testing
• Error/defect collection and analysis
• Change management
• Education
• Vendor management
• Security management
• Safety
• Risk management

53

Role of the SQA Group-I

• Prepares an SQA plan for a project.
• The plan identifies
• evaluations to be performed
• audits and reviews to be performed
• standards that are applicable to the project
• procedures for error reporting and tracking
• documents to be produced by the SQA group
• amount of feedback provided to the software project team
• Participates in the development of the project’s software process description.
• The SQA group reviews the process description for compliance with organizational policy, internal software standards, externally imposed standards (e.g., ISO-9001), and other parts of the software project plan.

54

Role of the SQA Group-II

• Reviews software engineering activities to verify compliance with the defined software process.
• identifies, documents, and tracks deviations from the process and verifies that corrections have been made.
• Audits designated software work products to verify compliance with those defined as part of the software process.
• reviews selected work products; identifies, documents, and tracks deviations; verifies that corrections have been made
• periodically reports the results of its work to the project manager.
• Ensures that deviations in software work and work products are documented and handled according to a documented procedure.
• Records any noncompliance and reports to senior management.
• Noncompliance items are tracked until they are resolved.

55

SQA Goals

• Requirements quality. The correctness, completeness, and consistency of the requirements model will have a strong influence on the quality of all work products that follow.

​

• Design quality. Every element of the design model should be assessed by the software team to ensure that it exhibits high quality and that the design itself conforms to requirements.

​

• Code quality. Source code and related work products (e.g., other descriptive information) must conform to local coding standards and exhibit characteristics that will facilitate maintainability.

​

• Quality control effectiveness. A software team should apply limited resources in a way that has the highest likelihood of achieving a high quality result.

56

Formal SQA

• Assumes that a rigorous syntax and semantics can be defined for every programming language

​

• Allows the use of a rigorous approach to the specification of software requirements

​

• Applies mathematical proof of correctness techniques to demonstrate that a program conforms to its specification

​

57

Statistical SQA

58

Product

& Process

measurement

... an understanding of how

to improve quality ...

Collect information on all defects

Find the causes of the defects

Move to provide fixes for the process

Statistical SQA

• Information about software errors and defects is collected and categorized.

​

• An attempt is made to trace each error and defect to its underlying cause (e.g., non-conformance to specifications, design error, violation of standards, poor communication with the customer).

​

• Using the Pareto principle (80 percent of the defects can be traced to 20 percent of all possible causes), isolate the 20 percent (the vital few).

​

• Once the vital few causes have been identified, move to correct the problems that have caused the errors and defects.

59

Six-Sigma for Software Engineering

• The term “six sigma” is derived from six standard deviations—3.4 instances (defects) per million occurrences—implying an extremely high quality standard.
• The Six Sigma methodology defines three core steps:
• Define customer requirements and deliverables and project goals via well-defined methods of customer communication
• Measure the existing process and its output to determine current quality performance (collect defect metrics)
• Analyze defect metrics and determine the vital few causes.
• Improve the process by eliminating the root causes of defects.
• Control the process to ensure that future work does not reintroduce the causes of defects.

60

Software Reliability

• A simple measure of reliability is mean-time-between-failure (MTBF), where

MTBF = MTTF + MTTR

• The acronyms MTTF and MTTR are mean-time-to-failure and mean-time-to-repair, respectively.
• Software availability is the probability that a program is operating according to requirements at a given point in time and is defined as

Availability = [MTTF/(MTTF + MTTR)] x 100%

61

Software Safety

• Software safety is a software quality assurance activity that focuses on the identification and assessment of potential hazards that may affect software negatively and cause an entire system to fail.
• If hazards can be identified early in the software process, software design features can be specified that will either eliminate or control potential hazards.

62

ISO 9001:2008 Standard

• ISO 9001:2008 is the quality assurance standard that applies to software engineering.
• The standard contains 20 requirements that must be present for an effective quality assurance system.
• The requirements delineated by ISO 9001:2008 address topics such as
• management responsibility, quality system, contract review, design control, document and data control, product identification and traceability, process control, inspection and testing, corrective and preventive action, control of quality records, internal quality audits, training, servicing, and statistical techniques.

63

Review Techniques

64

Reviews

65

... there is no particular reason

why your friend and colleague

cannot also be your sternest critic.

Jerry Weinberg

What Are Reviews?

• a meeting conducted by technical people for technical people
• a technical assessment of a work product created during the software engineering process
• a software quality assurance mechanism
• a training ground

66

What Reviews Are Not

• A project summary or progress assessment
• A meeting intended solely to impart information
• A mechanism for political or personal reprisal!

67

What Do We Look For?

• Errors and defects
• Error—a quality problem found before the software is released to end users
• Defect—a quality problem found only after the software has been released to end-users
• We make this distinction because errors and defects have very different economic, business, psychological, and human impact
• However, the temporal distinction made between errors and defects in this book is not mainstream thinking

68

Defect Amplification

• A defect amplification model [IBM81] can be used to illustrate the generation and detection of errors during the design and code generation actions of a software process.

69

Errors passed through

Amplified errors 1:x

Newly generated errors

Development step

Errors from

Previous step

Errors passed

To next step

Defects

Detection

Percent

Efficiency

Defect Amplification

• In the example provided in SEPA, Section 20.2,
• a software process that does NOT include reviews,
• yields 94 errors at the beginning of testing and
• Releases 12 latent defects to the field
• a software process that does include reviews,
• yields 24 errors at the beginning of testing and
• releases 3 latent defects to the field
• A cost analysis indicates that the process with NO reviews costs approximately 3 times more than the process with reviews, taking the cost of correcting the latent defects into account

70

Metrics

• The total review effort and the total number of errors discovered are defined as:
• E_review = E_p + E_a + E_r
• Err_tot = Err_minor + Err_major
• Defect density represents the errors found per unit of work product reviewed.
• Defect density = Err_tot / WPS
• where …

71

Metrics

• Preparation effort, E_p — the effort (in person-hours) required to review a work product prior to the actual review meeting
• Assessment effort, E_a — the effort (in person-hours) that is expending during the actual review
• Rework effort, E_r — the effort (in person-hours) that is dedicated to the correction of those errors uncovered during the review
• Work product size, WPS — a measure of the size of the work product that has been reviewed (e.g., the number of UML models, or the number of document pages, or the number of lines of code)
• Minor errors found, Err_minor — the number of errors found that can be categorized as minor (requiring less than some pre-specified effort to correct)
• Major errors found, Err_major — the number of errors found that can be categorized as major (requiring more than some pre-specified effort to correct)

72

An Example—I

• If past history indicates that
• the average defect density for a requirements model is 0.6 errors per page, and a new requirement model is 32 pages long,
• a rough estimate suggests that your software team will find about 19 or 20 errors during the review of the document.
• If you find only 6 errors, you’ve done an extremely good job in developing the requirements model or your review approach was not thorough enough.

73

An Example—II

• The effort required to correct a minor model error (immediately after the review) was found to require 4 person-hours.
• The effort required for a major requirement error was found to be 18 person-hours.
• Examining the review data collected, you find that minor errors occur about 6 times more frequently than major errors. Therefore, you can estimate that the average effort to find and correct a requirements error during review is about 6 person-hours.
• Requirements related errors uncovered during testing require an average of 45 person-hours to find and correct. Using the averages noted, we get:
• Effort saved per error = E_testing – E_reviews
• 45 – 6 = 30 person-hours/error
• Since 22 errors were found during the review of the requirements model, a saving of about 660 person-hours of testing effort would be achieved. And that’s just for requirements-related errors.

74

Overall

• Effort expended with and without reviews

75

with reviews

Reference Model

76

Informal Reviews

• Informal reviews include:
• a simple desk check of a software engineering work product with a colleague
• a casual meeting (involving more than 2 people) for the purpose of reviewing a work product, or
• the review-oriented aspects of pair programming
• pair programming encourages continuous review as a work product (design or code) is created.
• The benefit is immediate discovery of errors and better work product quality as a consequence.

77

Formal Technical Reviews

• The objectives of an FTR are:
• to uncover errors in function, logic, or implementation for any representation of the software
• to verify that the software under review meets its requirements
• to ensure that the software has been represented according to predefined standards
• to achieve software that is developed in a uniform manner
• to make projects more manageable
• The FTR is actually a class of reviews that includes walkthroughs and inspections.

78

The Review Meeting

• Between three and five people (typically) should be involved in the review.
• Advance preparation should occur but should require no more than two hours of work for each person.
• The duration of the review meeting should be less than two hours.
• Focus is on a work product (e.g., a portion of a requirements model, a detailed component design, source code for a component)

79

The Players

80

review

​

leader

producer

recorder

reviewer

standards bearer (SQA)

maintenance

oracle

user rep

The Players

• Producer—the individual who has developed the work product
• informs the project leader that the work product is complete and that a review is required
• Review leader—evaluates the product for readiness, generates copies of product materials, and distributes them to two or three reviewers for advance preparation.
• Reviewer(s)—expected to spend between one and two hours reviewing the product, making notes, and otherwise becoming familiar with the work.
• Recorder—reviewer who records (in writing) all important issues raised during the review.

81

Conducting the Review

• Review the product, not the producer.
• Set an agenda and maintain it.
• Limit debate and rebuttal.
• Enunciate problem areas, but don't attempt to solve every problem noted.
• Take written notes.
• Limit the number of participants and insist upon advance preparation.
• Develop a checklist for each product that is likely to be reviewed.
• Allocate resources and schedule time for FTRs.
• Conduct meaningful training for all reviewers.
• Review your early reviews.

82

Review Options Matrix

83

trained leader

agenda established

reviewers prepare in advance

producer presents product

“reader” presents product

recorder takes notes

checklists used to find errors

errors categorized as found

issues list created

team must sign-off on result

​

​

IPR—informal peer review WT—Walkthrough

IN—Inspection RRR—round robin review

IPR

WT

IN

RRR

no

maybe

maybe

maybe

no

maybe

no

no

no

no

yes

yes

yes

yes

no

yes

no

no

yes

yes

yes

yes

yes

no

yes

yes

yes

yes

yes

yes

yes

yes

yes

no

no

yes

no

no

yes

maybe

*

*

Sample-Driven Reviews (SDRs)

• SDRs attempt to quantify those work products that are primary targets for full FTRs.

To accomplish this …

• Inspect a fraction a_i of each software work product, i. Record the number of faults, f_i found within a_i.
• Develop a gross estimate of the number of faults within work product i by multiplying f_i by 1/a_i.
• Sort the work products in descending order according to the gross estimate of the number of faults in each.
• Focus available review resources on those work products that have the highest estimated number of faults.

84

Metrics Derived from Reviews

85

inspection time per page of documentation

​

inspection time per KLOC or FP

​

errors uncovered per reviewer hour

​

errors uncovered per preparation hour

​

errors uncovered per SE task (e.g., design)

​

number of minor errors (e.g., typos)

​

​

number of errors found during preparation

number of major errors

(e.g., nonconformance to req.)

inspection effort per KLOC or FP