Does Distributed Development Affect Software Quality?

An Empirical Case Study of Windows Vista

Authors:-

Christian Bird

Nachiappan Nagappan

Premkumar Devanbu

Harald Gall

Brendan Murphy

Presented by:-

Prasad Pai(MT2011106)

What is this paper about:-

• Comparison of post-release failures of components that were developed in a distributed fashion with those that were developed by collocated teams.
• Empirical study of overall development of Windows Vista.
• The authors are the researchers and developers in Microsoft.
• What different approach has Microsoft taken in global software development from others.

Point of view of Authors:-

1. Authors examined distributed development at multiple levels of separation (building, campus, continent, etc.).
2. Authors examined a large scale software development effort, composed of thousands of executables, libraries and thousands developers.
3. Authors examined complexity and maintenance characteristics of the distributed and collocated executables, libraries to check for inherent differences that might influence post-release quality.
4. Authors have examined a project in which all sites involved are part of the same company and have been using the same process and tools for years.

The inability of a system or component to perform its required functions within specified performance requirements.

IEEE definition of a failure:-

Effects of post-release failure:-

• Reputation
• Marketshare
• Time

Binaries:-

• Binaries are individual executables and libraries.
• Binaries are classified into

(a) Distributed

(b) Collocated

• Binaries are classified as developed in a distributed manner if at least 25% of the commits came from locations other than where binary’s owner resides.

​

Hypothesis:-

1. H1: Binaries that are developed by teams of engineers that are distributed will have more post-release failures than those developed by collocated engineers.
2. H2: Binaries that are distributed will be less complex, experience less code churn, and have fewer dependencies than collocated binaries.

Related Work:-

• There are a number of experience reports for globally distributed software development projects at various companies like Siemens, Alcatel, Motorola, Lucent and Philips.
• Their work has been classified into three categories:-
• Effects on bug resolution
• Effects on quality and productivity
• Issues and solutions

1. Effects on bug resolution:-

• Herbsleb and Mockus examined the time to resolution of Modification Requests (MRs) in two departments of Lucent working on distinct network elements for a telecommunication system.
• The average time needed to complete a single-site MR was 5 days versus 12.7 for distributed.
• When controlling for other factors such as number of people working on an MR, how diffused the changes are across the code base, size of the change, and severity, the effect of being distributed was no longer significant.

• The paper concluded claiming

(a) People who are assigned work from many sources have lower productivity.

(b) MRs that require work in multiple modules have a longer cycle time than those which require changes to just one.

• But this paper focuses on the effect of distributed development on defect occurrence, rather than on defect resolution time.

b) Effects on quality and productivity:-

• Ramasubbu and Balan examined the relationship between the dispersion of a project and its development productivity and conformance quality.
• They gathered information from 42 projects over 2 years.
• They found that projects that had more dispersion also had lower levels of productivity and conformance quality.
• But this effects were strongly mitigated through quality management approaches.
• But this study examines characteristics of components within one large software project, which arguably provides better control over possibly confounding project-specific factors.

c) Issues and solutions:-

Carmel categorizes project risk factors into four categories that act as centrifugal forces that pull global projects apart. These are

1. Loss of communication richness
2. Coordination breakdowns
3. Geographic dispersion
4. Cultural differences

Solutions to overcome this issues were:-

1. Reduce intensive collaboration.
2. Reduce national and organizational cultural distance.
3. Reduce temporal distance.

About Windows Vista:-

• Thousands of skilled Microsoft developers were involved.
• Project comprised of thousands of binaries.
• Source code base was of tens of millions of lines of code.
• Developers were distributed across 59 buildings and 21 campuses in Asia, Europe, and North America.

Data collection:-

Data collected focused on three properties

1. Code quality
2. Code ownership
3. Geographical location

Geographical location:-

1. Building:-
1. A binary classified at the building level may have been worked on by developers on different floors of the same building.
2. Developers who work in the same building will enjoy more face to face and informal contact.

2. Cafeteria:-
1. In Microsoft, a cafeteria can be seen between one and five nearby buildings.
2. Developers in different, but nearby buildings, may share meals together and meet by chance during meal times.

3. Campus:-
1. A campus represents a group of buildings in one location.
2. It is easy to travel between buildings on the same campus by foot.

4. Locality:-

• Locality is used to represent groups of geographically proximate campuses.
• All sites in a particular locality operate in the same time zone, making coordination and communication within a locality easier than between localities.

5. Continent:-

• All of the locations on a given continent fall into this category.
• Authors chose to group at the continent level rather than the country level because Microsoft has many offices across different countries which are less than 3 hours to each other by road.

5. World:-
1. Binaries developed by engineers on different continents are placed in this category.
2. Face to face meetings are rare and synchronous communication such as phone calls or online chats are hindered by time differences.
3. Cultural and language differences are more likely.

• Each binary is assigned the lowest level in the hierarchy from which at least 75% of the commits were made.
• Commit is made after significant change is done or at least one function point is added in code.

Binaries distribution:-

Post-release failures:-

• Mann Whitney test was used to quantitatively measure which is having more defects on an average terms.
• It was found that on an average distributed binaries were 8% more in average terms.

Mathematical modeling:-

• Linear regression was used to examine the effect of distributed development on number of failures.
• Four separate models were generated.
• Two models studied effect of having distributed development.
• Two models studied effect of having distributed development with variable number of developers in each side.
• F-test was used to test the validity of this models.

Results of the models:-

Model 1 result:-

Model 2 result:-

Model 4 result:-

Model 3 result:-

What does results say:-

• A binary with 4 failures if collocated would have 4.24 failures if distributed.
• If the number of people working on a development task is controlled, distribution does not have a large effect.
• Based on these results, hypothesis H1 is not confirmed.

How results are so good?

• Is it that the distributed binaries are smaller, less complex and have fewer dependencies.
• There is a need to understand the characteristics of the binaries.

Different characteristics of binaries:-

a) Size and complexity:-

• The code size and complexity measures include number of independent paths through the code, number of functions, classes, parameters, blocks, lines, local and global variables and cyclomatic complexity.

b) Code churn:

• In this, change in size of the binary, the frequency of edits and the churn size in terms of lines removed, added, and modified from the beginning of Vista development until release to manufacturing was examined.

c) Test coverage:-

• The number of blocks and branches as well as the block coverage and branch coverage are recorded during the testing cycle for each binary.

d) Dependencies:-

• In this direct incoming and outgoing dependencies as well as the transitive closure of these dependencies was examined.

e) People:-

• We include a number of statistics on the people and organizations that worked on the binaries.

The results on binaries characteristics:-

About Microsoft:-

What did Microsoft do on its part to overcome the challenges of Global Software Development.

a) Relationship between sites:-

• Engineers at different sites may feel competitive or may for other reasons be less likely to help each other.
• In Microsoft, all sites have existed and worked together on software for many years.
• There is no threat that if one site performs better, the other will be shut down.
• The pay scale and benefits are equivalent at all sites in the company.

b) Cultural Barriers:-

• A large part of development was done in Redmond in U.S.A. and in Hyderabad in India.
• To promote cultural ties, the teams from Redmond had visited Hyderabad teams.

c) Communication:-

• The Vista developers made heavy use of synchronous communication daily.
• Employees took on the responsibility of staying at work late or arriving early for a status conference call on a rotating basis, changing the site that needed to keep odd hours every week.
• Keeping in close and frequent contact increases the level of awareness and the feeling of team spirit.

d) Consistent use of tools:-

• Microsoft employs the use of one configuration management and builds system throughout all of its sites.
• Every engineer is familiar with the same source code management tools, development environment, documentation method, defect tracking system, and integration process.

e) End to end ownership:-

• When an entity fails, needs testing, or requires a modification, it may not be clear who is responsible for performing the task or assigning the work.
• One developer is clearly in control of a particular piece of code from design, through implementation, and into testing and maintenance.
• Effort is made to minimize the number of ownership changes.

f) Common schedules:-

• The project was not made up of distributed modules that shipped separately.
• Vista had a fixed release date for all parties and mile-stones were shared across all sites.
• All engineers have a strong interest in working together to accomplish their tasks within common time frames.

g) Organizational Integration:-

• All organizations operating in multiple sites have different reporting managers.
• But in Microsoft, there is only one division of reporting managers across the globe.
• This causes geographically dispersed developers to be more integrated into the company and the project.

Threats to validity:-

1. Construct Validity: -
1. The data collection in Windows Vista project is automated.
2. Internal Validity:-
1. The points discussed about Microsoft has not been mathematically modeled and there is no concrete evidence that all points mentioned by them holds true in all the cases.
3. External Validity:-
1. It is unclear how well the results can be generalized to other situations.
2. There are many ways in which distributed software projects may vary and the particular characteristics must be taken into account.

Conclusion:-

1. If good software engineering practices are employed, distributed development is as good as single-site development.
2. An organizationally compact but geographically distributed project would be better than a geographically local, organizationally distributed project.
3. Work that is distributed should be less complex, should experience less code churn, and should have fewer dependencies than collocated work.

​