1 of 26

Repository Cohorts

How OSPOs Can Programmatically Categorize All Their Repositories

Remy DeCausemaker, Justin Gosses, Natalia Luzuriaga,

Isaac Milarsky, James Siri

#ossummit

@

2 of 26

Live Demos of Repository Cohorts to Try During Talk or After

Demo that takes any GitHub organization but only has partial data https://aka.ms/RepositoryCohorts/Demo/YourGitHubOrg

�Demo using Microsoft repository metadata, no data gaps:

https://aka.ms/RepositoryCohorts/Demo/Microsoft

3 of 26

What Are Repository Cohorts

The concept of programmatic manipulation of raw repository metadata into higher abstract forms

Allows for quickly understanding of the defining characteristics of a repository, its community, and its underlying usage

Easily customizable with repeatable usage for minimal additional investment

4 of 26

Repository Cohorts Definition

Repository cohorts are reusable, standardized, programmatically determined group labels.

Enable OSPOs to treat repositories based on their important characteristics rather than treating every repository as the median repository

5 of 26

Repository Cohorts - In Practice

For each repository, each cohort has a value of either true or false. A continuous dimension like “age” will be split into a number of cohorts. In a group of cohorts, like age, only one cohort will be true. Other cohorts are singular like “does repo use GitHub Actions”.

	cohort_age_ Baby 30d	cohort_age_ Toddler 30to90d	cohort_age_ Teen 90to365d	cohort_age_ Adult 365to1095d	cohort_age_ senior More1095d	Cohort_ githubActions
Repo A	False	False	False	True	False	True
Repo B	True	False	False	False	False	True

6 of 26

What Problems Do Repository Cohorts Solve?

OSPOs sometimes have more repositories than they could realistically read and remember

Some problems require applying rules/policies/guidance to only a subset of repositories

Collecting and analyzing raw repository metadata values takes considerable time, so it is only done when time and expected value seems like a good investment

what problems do repository cohorts solve?

OSPOs have hundreds or even thousands of repositories under their control which makes it hard to understand the particular needs and circumstances of every individual repository.

People don’t have time or patience to read every README or look at issues and pull requests to understand each individual community or how a repository is being used out in the wild.

As a result, all repositories get treated the same. A single set of rules. A single set of guidance. A single set of “best practices” to follow.

This becomes less than ideal when some of your repositories are actively being developed by teams of twenty and have a user base in the thousands while other repositories are intended to be mostly static examples of an implementation used in a published paper or target a specific audience.

Raw repository metadata analysis takes huge effort to do manually and takes considerable time.

These are just three examples, but there is a variety of different use cases for repositories that have different needs and different risks to be managed.

But they all clearly demonstrate that creating policy or guidance that accommodates the various needs become increasingly difficult as repository types diverge.

7 of 26

How Do Repository Cohorts Help?

Instead of…	Repository Cohorts...
..manually reading repositories to understand what they’re about	..uses metadata to work at any scale
..creating queries against raw metadata for each project	..creates ability to identify cohorts once, then reuses in many projects
..having to think about combinations of raw values	..are easy to remember higher order representations

Repository cohorts improve time to insights when working with repository metadata by reducing number of steps required as well as cognitive load.

So, how can repository cohorts help?

First, with repository cohorts, we’re trying to gain an ability to describe important characteristics of repositories with metadata such that we can work with a number of repositories too great to individually analyze.

OSPOs with a large open source presence can have repositories numbering in the tens of thousands but even a few hundred is often too great for people to reliably and repeatedly process. A metadata driven approach solves for this and allows OSPOs to function at whatever scale they need.��Second, there’s a tendency to write queries that work with raw metadata directly. This is fine, but it provides an easy path for projects to be analyzed with differing metrics bounds which causes confusion and uncertainty over time.

It also means you’re writing a lot of queries from scratch which increases the time and cognitive load every time you try to use repository metadata manually.

When using repository cohorts we’re using queries defined in code that is generalizable and repeatable, meaning it can be reused from project to project without additional time investment.

Third, you can really only reason about maybe two or three groups of raw metadata values in your head at a time. With repository cohorts, we’ll be making higher order representations, which makes it easier to think through that data. This point will make more sense once you see some samples.

8 of 26

Example Microsoft Repository Cohort Groups

Age

Cohorts based on thresholds for how long that repository has existed.

Activity

Cohorts are based on amounts of activity in a repository within time periods relative to project creation and current day.

Community

Cohorts that describe a repos community based on number of contributors, percentage of non-Microsoft contributors, clone amounts, and ratios of stars, forks, contributors, etc.

Content

Does the repo use GitHub Actions?
Does the repo publish a package used internally
Do strings in the title or README that suggest a repo is likely a sample or demo, etc.?

Here are some of the examples of different dimensions for which Microsoft uses repository cohorts.

For the most part, we are basing our cohort labels on Nadia labels. More info on these are coming later in this presentation.

We define our age cohort of baby, toddler, teen, adult and senior based on number of days since a repository was created.

We have defined an activity cohorts based on the days since an update occurred in a repository.

There are a variety of community cohorts based on

cutoffs for the size of a community
different ratios for

contributors
stars
forks
and others that give an indication of the shape of a community

Content based cohorts which covers things like:

does the repository use github actions
does the repository publishes a package or binary artifact
or does it have the words “sample”, “demo”, or “example” in the name or description suggesting the repository is intended to be used in a particular manner

These are important as these labels can be helpful to understand if a repository is created by one person with engagement of thousands or by ten people with the engagement of twenty.

These also simplify identification of different projects with different needs and different capabilities to respond with certain best practice guidance.

9 of 26

Two Microsoft Repositories, Only True Cohorts

*this was true at one point in time

Now, let’s apply some of these groups and see what we can learn using some sample repositories.

Here we have two different repositories and a selection of the cohorts for which they fall into.

By looking at just the cohort columns that are flagged, we can start to tell a lot about repositories very quickly.

Typescript is a repository that is

fairly old
developed by a large group of contributors
highly cloned
has a significant number of non-Microsoft contributors
was updated recently
and uses CI/CD tooling in the form of GitHub Actions

In contrast, the AzureOpenAI-Docs-Sample repository is

younger in age
a toy repository meaning there is a small number of contributors and engaged users as reflected in the stars ratio
has zero non-Microsoft contributors
and is declared by name to be used as a sample

By describing the repository with its identified cohorts, we can quickly gain insight into how a repository is being used by its community.

I will also note that repository cohorts are true at the time they are calculated.

As a repository ages, it will move age cohorts. As the community changes, it might grow into a federation or stadium type community.

Now that you have seen Microsoft’s approach to repository cohorts, I will hand things over to my partners at CMS.gov

10 of 26

CMS Repository Cohort Pipeline

Open Source Repository Maturity Models

Where is our project on our Open Source Journey?

Repository Templates

What files are required/recommended for good repository hygiene?

Outbound Checklists

What steps should our project take to release the repository publicly?

Cookiecutter

How do we know what Maturity Model Tier our new project should be in? What files are required in that Tier?

Repolinter Configs

What files or information is missing from our repo?

Establishing a Baseline: Repo Metrics, Maturity Models, Templates, and Checklists

@ 4:10-4:50pm RM447-448 https://ossna2024.sched.com/event/1aBPh

11 of 26

CMS Repository Cohort Definitions: Maturity Model Tiers

Level	Name	Purpose	Description
Tier 0	Private Repository	Experimental, Historical	Project is private, usually with a single developer. Typically working projects, example code, and early prototypes.
Tier 1	One-Time Release	Publication for Informational, Accountability, Transparency Purposes	Project released publicly, but without planned future activity or maintenance from original author(s).
Tier 2	Close Collaboration	Collaboration with smaller, mostly internal teams	Project within a team or Operational Division (OpDiv), Internal Repo for Innersource-style work
Tier 3	Working in Public	Collaboration in the open with smaller, semi-open teams	Project developed Open Source by CMS or a CMS contractor, public website hosted on GitHub, tool or utility used in CMS official business by the public. Limited external contribution, CMS-led (by choice or by statute.)
Tier 4	Community Governance	Collaboration broadly in public	Project donated to or stewarded by an external community, open standard that welcomes public input, mature open source project that purposefully develops an open governance structure.

12 of 26

CMS Repository Cohort Definitions: Maturity Model Tiers

Public Repo on GitHub!

https://github.com/dsacms/repo-scaffolder

cookiecutter cmdline tool

13 of 26

CMS Repository Cohort Definitions: Nadia Labels

In her book “Working in Public” by Nadia Asparouhova classifies repositories into four cohorts:

Toys
Clubs
Federations
Stadiums

https://project-types.github.io/

14 of 26

CMS Repository Cohort Definitions: Nadia Labels in Augur!

See Augur API Documentation for nadia_project_labeling_badge:�https://github.com/chaoss/augur/blob/main/augur/api/metrics/repo_meta.py#L205

15 of 26

How CMS Creates Repository Cohorts (1/3): Repo Metrics

Cohort Reuse

DSACMS Metrics scripts pulls Augur.io API endpoint to get Project Label and Color, and uses Shields.io to create a project badge on a Repo Report

Metadata Collection

When we can, we ‘hack upstream’ and use existing open source tools. ��In this case, we are using CHAOSS’ Augur project

Augur.io Ingests public repository data and history from GitHub.com

Cohort Creation

Augur.io uses previously mentioned nadia_project_category() to categorize projects based on methodology identified by CHAOSS OSPO Metrics WG members (Shout-out Justin Gosses!)

16 of 26

How CMS Creates Repository Cohorts (3/3): Repolinter

https://github.com/todogroup/repolinter

Repolinter is a tool maintained by the TODOGroup for checking repositories for common open source issues, using pre-defined rulesets. This can be run stand-alone as a script, pre-commit in your IDE, or post-commit or within CI/CD systems!

✔ = Pass

✖ = Fail

⚠ = Warn

Thanks to Chan and Satwic at the Comcast OSPO, we now have repolinter.json configs and rules that map to each Tier of our Open Source Repository Maturity Model!

https://github.com/DSACMS/repo-scaffolder/blob/main/tier3/%7B%7Bcookiecutter.project_slug%7D%7D/repolinter.json

17 of 26

Demo: CMS.gov Open Source Repository Metrics

https://dsacms.github.io/metrics

18 of 26

How Microsoft Creates Repository Cohorts

Cohort Reuse

Repository cohorts get reused as a Kusto query that gets combined with other queries as well as in PowerBI dashboards and Jupyter notebooks.

Metadata Collection

A centralized data team is responsible for metadata collection across all public & internal code platforms.

They make available upon internal request multiple tables of metadata for public GitHub repos as part of their Kusto cluster, a type of cloud database.

Cohort Creation

Microsoft OSPO has an internally visible catalog of Kusto queries, including a query for generating repository cohorts that uses the collected GitHub metadata as well as an export of ClearlyDefined.io to determine which repos build packages.

19 of 26

Microsoft Examples of Using Repository Cohorts

Demographics: How many of each repository cohort?

Representative sample: Select repositories for interviews or manual investigations.

Snapshot: When a potential new policy only applies to a subset of repositories, we can use cohorts to produce a snapshot description of the range of activity, community shape, etc. of that subset to help design effective policy and outreach.

Repository cohorts have been useful to the Microsoft OSPO in a number of different ways:

Understanding the distribution of different repository types and use cases across our large organizations.

This has been great at providing hard data for the number of projects that use their repository in a way that is different than the median repository that we might see across the organization
This allows us to craft policies and best practices that target specific cohorts without negatively impacting others outside that cohort

Identify representative repositories for in depth interviews or manual investigation.

For example, if we are looking for some group of repositories that meet some condition of interest but 25% look like stadium and 75% look like toys.

We can then design questions or policy to make sure not to miss the 25% or the 75% groups
This results in precisely targeted requirements and policies that have a smaller blast radius if things go wrong

Designing compliance measurements effectively. If there’s a rule that only applies to a subset of cohorts, we can hold them to requirements that might not fit other cohorts.

for example, we can ask if a cohort is filled with repositories that are actively being developed or are 85% of the repositories ones that haven’t had a commit in the last six months?
understanding this can change the way we approach repositories in that cohort and dictate a better course of action or policy

20 of 26

Demo A: Repository Cohorts Applied to Microsoft repositories

https://aka.ms/RepositoryCohorts/Demo/Microsoft

10k+ repos
No data gaps
GitHub pages
JavaScript

Count of Repos

21 of 26

Demo B: See Repository Cohorts for your GitHub Organization

Input your own GitHub Org and see results immediately.
Observable page
JavaScript
Run live SQL queries
Has missing fields and missing repos.

https://aka.ms/RepositoryCohorts/Demo/YourGitHubOrg

22 of 26

Demo: All Cohort Definitions are in a JSON.

Please submit PRs

To the cohort.js file in the Microsoft demo @

https://aka.ms/RepositoryCohorts/Demo/Cohort.js

23 of 26

Conclusions

Repository Cohorts can help OSPOs deliver fit-for-purpose guidance and compliance rather than one-size-fits-all.

Repository cohorts lower the time and cognitive burdens for using repository metadata.

There are multiple ways to collect the raw repository metadata, each suited to your own organization. The cohort definitions are sharable and reusable. �
There are organizations and experts outside of your OSPO (shoutout to CHAOSS.community, TODOgroup.org, and others) that have developed open standards that create a shared understanding across projects, organizations, and communities.

24 of 26

Live Demos of Repository Cohorts

Questions & Pull Requests

https://aka.ms/RepositoryCohorts/Demo/Cohort.js

Demo that takes any GitHub organization but only has partial data https://aka.ms/RepositoryCohorts/Demo/YourGitHubOrg

�Demo using Microsoft repository metadata, no data gaps:

https://aka.ms/RepositoryCohorts/Demo/Microsoft

25 of 26

Any Repository Cohorts Questions?

We are here to answer!

opensource@microsoft.com

opensource@cms.hhs.gov

Dog Tax

1 of 26

2 of 26

3 of 26

4 of 26

5 of 26

6 of 26

7 of 26

8 of 26

9 of 26

10 of 26

11 of 26

12 of 26

13 of 26

14 of 26

15 of 26

16 of 26

17 of 26

18 of 26

19 of 26

20 of 26

21 of 26

22 of 26

23 of 26

24 of 26

25 of 26

26 of 26