1 of 71

Drupal.org: Updating the tools we use for collaboration

2 of 71

hestenet

Tim Lehnen

drumm

Neil Drumm

mixologic

Ryan Aslett

b_man

Brendan Blaine

nnewton

Narayan Newton

mlhess

Michael Hess

fjgarlin

Fran Garcia-Linares

fmitchell

Frederic Mitchell

crawleyhost

Caleb Crawley

Infrastructure Team

Drupal Association Staff

Consultants

irinaz

Irina Zaks

Engineering Team

3 of 71

Infrastructure Partners

4 of 71

What does the DA invest

in engineering?

4

5 of 71

$375,000 in direct costs

+ More than $500,000 of in-kind trade

+ Salaries for personnel�

About half of the net revenue from DA programs is invested directly in engineering initiatives.

6 of 71

Drupal 7

End of Life

6

7 of 71

… We are announcing that moving forward, the scheduled Drupal 7 End-of-Life date will be re-evaluated annually. As of today, we are extending the end-of-life by one year to November 1, 2023.

- PSA-2022-02-23�drupal.org/security

“

8 of 71

Drupal 6 LTS vendor-provided support will end on October 22, 2022.

On February 24th, 2016, Drupal 6 was marked end-of-life (EOL). The Drupal 6 Long-Term-Support (LTS) program added more than 6 years of additional coverage for program participants and the community.

On behalf of the community the Drupal Security Team would like to thank all the vendors that participated in this program: Tag1, Acquia, and myDropWizard.

- PSA-2022-03-09�drupal.org/security

“

9 of 71

Community

Groups

9

10 of 71

11 of 71

12 of 71

13 of 71

Release Pages�Overhauled

13

14 of 71

Contrib Releases

15 of 71

Core Releases

16 of 71

Organization Profile

Refresh

16

17 of 71

18 of 71

19 of 71

Distributions�&�Starter Kits

19

20 of 71

Composer Template

Support Module

Distributions

=

+

drupal.org/i/3264926

A new model for distributions

21 of 71

drupal.org/i/3252534

22 of 71

GDPR Updates

22

23 of 71

Moving to a first-party analytics solution.�

Providing more fine-grained control of cookie acceptance.

24 of 71

Drupal 9 Upgrades

api.drupal.org

events.drupal.org

www.drupal.org

9

24

25 of 71

api.drupal.org

26 of 71

27 of 71

28 of 71

29 of 71

www.drupal.org�project endpoints

30 of 71

31 of 71

32 of 71

Bluecheese

What's left before we start deploying our Drupal 9 sites?

Auth/SSO

Theme

Production

Infrastructure

33 of 71

DA + Core Initiatives

Supply Chain Security�Automatic Updates

Project Browser

33

34 of 71

35 of 71

36 of 71

TED BOWMAN

drupal.org/u/tedbow

Getting Ready for Automatic Updates in Drupal Core

Wednesday, 27 April | 11:00 PDT

Principal Software Engineer

Acquia

USA

Learn More

37 of 71

Authentication/SSO

KeyCloak via Cloud-IAM

37

38 of 71

drupal.org/project/openid_connect

OpenID Connect

9

7

Camp & Community Sites?

39 of 71

40 of 71

41 of 71

42 of 71

Gitlab Acceleration Initiative

Big picture and details of implementation

42

43 of 71

Goals�

Make Code contribution to the Drupal project easier. �

Adopt standard tools and adapt for Drupal ecosystem.

�Preserve the collaborative nature of Drupal.

44 of 71

Remains on Drupal.org

Migrates to git.drupalcode.org

Discovery
Project Pages
Releases
Contribution Credit
Composer Endpoints
User Documentation� (D.O Guides and Pages)

Repositories - completed
Testing (CI)
Issues
New: Kanban Boards
New: Developer Documentation GitLab Pages

45 of 71

Drupal Association

Configure collaboration �workflow & permissions
Provide helpful defaults
Coordinate migration
Host and maintain �git.drupalcode.org

GitLab

Support our migration
Innovate with new collaboration tools
Documentation

Community

Brainstorming and issue participation
“Trail-blazers” projects explore and experiment
Users-help-users to overcome�the learning curve.

This is a major collaboration between drupal community , association and gitlab

Clearly defined roles and responsibilities allow us to move forward efficiently.

We want to thank all contributors to this project, and especially DrupalSpoons project for expertise, brainstorming and insights that you provided and continue to provide !!! Thank you!!

Important part of our collaboration with gitlab is that we are using standard features and not hacking gitlab core. Using basic principles of Open source we identify missing features, define how gitlab core can be improved and contribute back to gitlab via issues and discussions rather then building features on our end.

One of examples of such collaboration is a feature request that Gitlab implement recently, which allows collaboration by default in a shared namespace - drupal community is a beneficiary of this change but every project on gitlab can now also use this feature.

46 of 71

The big picture

In progress	Single Sign-on
No big change	Projects		Releases
In progress	Simplifying Drupal.org (commits pages, profiles, credit system)
In progress	GitLab CI
Planning	Issue Collaboration	Issue Credit		Issue Migration

Helps Drupal.org update past Drupal 7

drupalcamp.be/en/drupal-dev-days-2022/session/drupalorg-gitlab-acceleration-update

47 of 71

GitLab Quick Wins &

Simplifying Drupal.org

48 of 71

Features enabled in GitLab

SSH Key management

GPG for commit signing

�Notifications

Authentication/Audit Log

Dark mode

.. and more.

49 of 71

Full code search

50 of 71

Project pages

Removed commit information.�

Replace our listings with�links to git.drupalcode.org.

�Documentation link simplification.�

When issues migrate,�their block will be�replaced or removed.

Most of the changes you’ll see are in the sidebar.

We no longer have listings of commits to projects on www.drupal.org. The maintainers block was redone. There were longstanding requests to change maintainers to be less-code-focussed anyway, not all contribution is code. We also wanted to show profile pictures, not just names.

GitLab has information about Git activity and trends built in, we don’t need to replicate it.

Documentation link changes are upcoming. The existing system favors entity references to documentation on Drupal.org. GitLab has more documentation options, like GitLab pages and wikis. While those won’t have the features and searchability of documentation on Drupal.org, we don’t need to maintain the currently over-complicated links.

And of course issues will go away when we migrate issues, we’ll come back to that later.

51 of 71

Profile pages

Commit counts on profile pages�will be replaced with link to GitLab activity.�

Adding link to your�git.drupalcode.org profile.

Profile pages are another place where commits are shown. As far as I know, this sort of summary data isn’t something we can get out of GitLab’s API with an efficient number of HTTP requests. And if we pull it via the API, we’re back to the same level of complexity to maintain as what we’re trying to get away from.

This will be replaced by something, I’m not sure what exactly yet. This is the number one place we’ve gotten support requests over the years, people like seeing their numbers of commits go up. But it’s just not something that is worthwhile to maintain. We do have issue credit, which is in many ways more fair. We could have a simple listing of projects someone is a maintainer of.

And since people have a profile on git.drupalcode.org, that will become increasingly important, we should link to it.

52 of 71

GitLabCI

53 of 71

GitLabCI

DrupalCI was centralized.��GitLab CI is decentralized.�

GitLab CI replaces parts of DrupalCI that were cumbersome to maintain.

Default testing is owned by DA

Module / core can write their tests and be responsible for them

��Drumm's speaker notes from DevDays:�Testing and continuous integration has been hugely important to the Drupal project. What started as community members getting together a testing stack custom built for Drupal has become an essential part of the core development process and for many contrib projects too. In many ways DrupalCI was innovative and ahead of its time. But now parts of it are cumbersome to maintain, and Drupal Association resources can be a bottleneck to using it in new ways.

DrupalCI has been centralized. We could do things like add coding the same standards check to every test run. While that helped ensure some consistency across projects with testing, improvements were limited by the time of a few people. GitLab CI is configured separately for every project, every maintainer is responsible for setting up their own project, and even more responsible for deciding what standards are right for their project.

GitLab CI lets us replace the parts of the DrupalCI that are a pain to maintain.

dispatcher.drupalci.org is a public Jenkins server. Jenkins isn’t really meant to be on the public internet. And has had some occasional maintenance challenges.
Project Issue File Test, PIFT, we haven’t always had the time to fix rough edges in the user interface, or do the best job of displaying test results.
The DrupalCI testbot, which sets up the codebase, and gathers results, is a bit of a bespoke system, was built before many now-common testing solutions

54 of 71

GitLab CI: Actually running tests

Every maintainer will have to set up their own .gitlab-ci.yml file, if they want testing.�

Replaces both the scheduling on Drupal.org and configuration that was implicit in DrupalCI.

�We want to find ways to make this�transition easy for maintainers.

�Drumm's speaker notes from DevDays:�Unlike DrupalCI, GitLab CI expects every maintainer set up their own tests, mostly from scratch. This makes sense for GitLab.com, there’s no one way to test every codebase. But we have thousands of Drupal projects and DrupalCI provided a consistent baseline of testing. The same PHP/database options for everyone, running Drupal in a subdirectory to catch everyone who doesn’t think of that off-guard, the same coding standards for everyone, linting to catch syntax errors without spending the time to run a full test. This also came with limitations, like not having containers outside of the ones we provide and us at the Drupal Association being a bottleneck for making improvements.

GitLab CI is going to require every maintainer to do a bit more work to set up testing, but we do want to smooth out the transition so it’s easy for projects can move. GitLab CI has some templating functionality, so some parts can be set up once for everyone. Bits and pieces of this have been done in the past, such as Moshe Weitzman’s DrupalSpoons project, but nothing comprehensive and designed to support every project.

We’ve enabled GitLab CI for a handful of projects to get started using it, so they can help us architect how the Drupal project can best leverage GitLab CI. Once we’ve got something that works, we’ll open it up for more projects. We’re also working with core maintainers to help get core’s testing up and running with GitLab CI.

55 of 71

GitLab CI: Discontinuing .patch files

GitLab CI tests merge requests,�not patches.�

When a project switches from DrupalCI to GitLab CI,�patches won’t be tested.

56 of 71

What about including .patch files in deployments?

Bad: Hot-linking to remotely hosted patch files is a really common habit in the Drupal community, but it's one we need to break.

Better: Commit the patch file to an artifacts directory on your infra.

Best: Help the patch get committed

- contribute your reviews.

57 of 71

Translating DrupalCI to GitLabCI

58 of 71

Thank you to these Contrib Projects

Search API Open Search

Decoupled Menus Initiative

Recurring Dates Field

BPMN.io

KeyCDN

Project Browser

Upgrade Status

Simple OAUTH

Scheduled Transitions

Views Filter Summary

Using Drupal Spoons .gitlab-ci.yml

Using own .gitlab-ci.yml

Thank you, DrupalSpoons!

59 of 71

Contribution Credit

60 of 71

Contribution is more than just issues

61 of 71

Mockup

62 of 71

Contributor Attribution

Maintainer

Credit Assignment

Mockup

63 of 71

Issue Migration

64 of 71

Drupal Flow: Collaborative by default

65 of 71

Issues: Collaboration

Issues on Drupal.org�are collaborative.�

We want to keep that�for GitLab issues.

Issues fixed in the last year,�that had people credited�As of 2022-04-02

People credited	Issues	Project
5.8	1826	Drupal core
2.2	525	Cloud
2.2	289	Open Social
1.8	276	Webform
2.3	225	Automatic Updates
1.1	220	Block Class
2.5	155	Commerce Core
2.5	148	CKEditor 5
2.7	148	Florida DrupalCamp
2	141	ECA: Event - Condition - Action
2.3	132	Gin Admin Theme
7.5	98	Project Browser
1.9	97	simplytest.me
1.5	96	Search API Solr
1.6	95	farmOS
1.3	86	Geolocation Field
3.9	85	Views (for Drupal 7)
1.4	83	Web Experience Toolkit
1.5	81	Smart Date

Drumm's speaker notes from DevDays: ��With patch files in issues on Drupal.org, collaboration was the default. Anyone can apply a patch, work on it, and upload a new patch. Merge & pull requests are not quite like that. They are built for one person to drive an issue to completion, maybe allowing commit access to a couple of collaborators. We want to ensure anyone can pick up a merge request and push code for it in the same way.

Issue forks on Drupal.org got around this since the integration we wrote for this on Drupal.org can create forks and grant access to them. We want to set this up in a way that doesn’t heavily rely on custom tooling. GitLab is a product after all, our customization options are limited.

So we’ll be setting GitLab permissions up in a way that allows people to make forks at any time, not necessarily attached to issues, and allows people to contribute to forks and merge requests they didn’t start.

We also want to see if we can allow anyone to edit an issue or merge request summary, like we expect from working on Drupal.org. This isn’t something GitLab provides out of the box.

66 of 71

Fork-Merge vs. Drupal Flow

PROJECT

Fork a

Fork b

Fork-of-fork b

MR a

MR b

MR c

Personal�Namespace

Collaboration by maintainer invite only

PROJECT

Fork

MR

Shared

Namespace

Collaboration by default

67 of 71

Issues: Preparation for Migration

References to issues on www.drupal.org will go away as we move issues away.�

This will impact change records, documentation, and more.

68 of 71

Issues: Migration

Project by project.�

All the metadata, preserve as much as practical.�

Aim for a smooth transition.

69 of 71

How you can help with the GitLab Acceleration

Meta issue at drupal.org/i/3227737
#gitlab on Drupal Slack
Help test things as they become available.
Use merge requests.
Look for projects with GitLab CI enabled.�If you are a maintainer, it will be available soon.

70 of 71

Question & Answer

?

70

71 of 71

Join us in Contribution!

General Contribution | Every Day | Oregon Ballroom 204

Mentored Contribution | Wed & Thu | 14:30 to 17:30 | C120-121-122

Migration Assistance | Wed & Thu | 14:30 to 17:30 | B114-115

Photo credit: Josef Jerabek

Photo credit: Amazee Labs & Roy Segall