Predicting open source contributor turnover from �value-related discussions: An analysis of GitHub issues
jack@jackjamieson.net
Jack Jamieson�NTT Kyoto, Japan | Naomi Yamashita NTT Kyoto, Japan | Eureka Foong�Tokyo College, �University of Tokyo, Japan |
Full paper:
jackjamieson.net/icse2024/
Values and software engineering
Positive social impact
Build software that
Values: “Guiding principles of what people consider important in life” (Cheng & Fleischmann, 2010)
Effect on software contributors’ motivation
2
Copyright NTT CORPORATION
Limitation: Prior research is based on surveys and interviews
Effect on software contributors’ motivation
In sum, engaging with values during software development seems to affect contributor turnover
3
Copyright NTT CORPORATION
To what extent do changes in the percentage of discussions that are related to values �predict significant changes in contributor turnover?
4
Copyright NTT CORPORATION
Research site: �DWeb projects on GitHub
Community of companies, teams, and individuals building Decentralized Web infrastructures.
Key benefits for this study:
https://getdweb.net
5
Copyright NTT CORPORATION
Research model
Define dictionary of keywords representing core DWeb values:
Detect value-related discussions using�keyword-matching
Quantify % of discussions each month related to each value
Quantify turnover rates (per project per month)
Incoming turnover:
% of contributors who made their first contribution �each month
Outgoing turnover:
% of contributors who made their final contribution �each month
Collect data (GitHub API)
Copyright NTT CORPORATION
Compare two timelines (by project, by month):
X: Percentage of discussions related to {value}
Y: Turnover rates (incoming, outgoing)
Illustration of perfect Granger Causality:
Impulse Response Function
🡨 One month later
Granger causality analysis: Does an increase in X predict a change in Y?
7
Copyright NTT CORPORATION
Highlighted findings
8
Copyright NTT CORPORATION
“Respectfulness”
Top terms: code of conduct; polite; rude
64% discussion-focused
Generally, complaints about others’ behaviour:
”failure to respond here, while continuing to post in other groups, is rude” ��“This is the second time you opened a new issue specifically to circumvent a prior issue’s closure. Continuing to do so is in violation of the code of conduct.”
9
Copyright NTT CORPORATION
After an increase in“respectfulness” discussions…
🡪 More people quit
🡪 Fewer people join
10
Copyright NTT CORPORATION
“Freedom” issues
Top terms: freedom; user choice; sovereign
100% Product-focused
About the role of “freedom” in the software
“[This feature allows] the developer to have more freedom to structure the implementation of the user module instance.”
“[Opposing] Social power” issues
Top terms: central authority; gatekeeper; monopoly
~99% Product-focused
About opposing concentrations of “social power” in the software
“what DIDs do is allow one to find cryptographic material proven to be associated with a given identifier, without a central authority involved.”
11
Copyright NTT CORPORATION
“Freedom” and “social power”
12
Copyright NTT CORPORATION
Summary of findings
“Respectfulness” discussions
(Generally) interpersonal conflict about disrespectful collaborator behaviour. ��Predicts... more people leaving and fewer people joining.
“Freedom” and “Social power” discussions
(Generally) about designing the software to reflect shared values.
Predicts... fewer people leaving.
13
Copyright NTT CORPORATION
Implications
Monitoring conflicts about values could help gauge project health
(especially “respectfulness”)
Open source project dashboards could monitor value-related discussions.��e.g., CHAOSS [1], Climate Coach [2]
[1] Sean P. Goggins, Matt Germonprez, and Kevin Lumbard. 2021. Making Open Source Project Health Transparent. Computer 54, 8 (2021), 104–111.
[2] Huilian Sophie Qiu et al. 2023. Climate Coach: A Dashboard for Open-Source Maintainers to Overview Community Dynamics. CHI ’23
14
Copyright NTT CORPORATION
Implications
Freedom and social power:
Discussions that integrate �widely-agreed-upon values with technical work may motivate sustained contributions.
Maintainers and managers may increase motivation by highlighting relationships between shared values and technical decisions.
15
Copyright NTT CORPORATION
Implications
Toxicity detection �is very difficult in software engineering contexts, especially for subtle toxicity.
”Respectfulness” discussions identified non-verbal and subtly toxic behavior that would be missed by most NLP approaches.
Identifying reactions (e.g., accusations of disrespectfulness) �may improve detection of subtle toxicity.
16
Copyright NTT CORPORATION
Thank you!
jack@jackjamieson.net
Jack Jamieson�NTT Kyoto, Japan | Naomi Yamashita NTT Kyoto, Japan | Eureka Foong�Tokyo College, �University of Tokyo, Japan |
Full paper:
jackjamieson.net/icse2024/
18
Copyright NTT CORPORATION
Discussion – What do we mean by “predicts”?
Disrespectful behavior
Discussions about "respectfulness"
Increase in project turnover
Q: Do discussions about respectfulness cause turnover?
19
Copyright NTT CORPORATION
Discussion – What do we mean by “predicts”?
Disrespectful behavior
Discussions about "respectfulness"
Increase in project turnover
Q: Do discussions about respectfulness cause turnover?
20
Copyright NTT CORPORATION