1 of 162

Setting up a Scientific Software Project using GitLab: From “Zero” to “Hero”

US-RSE Education & Training Seminar Series

Marshall McDonnell (mcdonnellmt@ornl.gov)

Oak Ridge National Laboratory

ORNL is managed by UT-Battelle LLC for the US Department of Energy

2 of 162

Introduction

3 of 162

What do I do at Oak Ridge National Laboratory (ORNL)?

Previously, worked in neutron scattering

Now, I am a Research Software Engineer

Help make quality software for science!

What scientific software do I do?

Materials Science
Neutron Scattering
Data Management
Supercomputing

4 of 162

Why talk about GitLab?

ORNL has multiple instances of GitLab (external, internal,...)
Really awesome resource for entire national lab!

5 of 162

Why talk about GitLab?

(Arguably) GitHub is better known

Previously, I only worked with GitHub

Sharing my experience now that I mostly use GitLab on a daily basis

Nice to diversify experience with different platforms

6 of 162

DISCLAIMER

I’m definitely not an expert on GitLab or GitHub!

In my daily work, use a “premium” version of GitLab with additional features above the free tier (implies bias)

Created this tutorial from a previous one using ORNL GitLab so remove / edit most of the content due to more recent version + features not available in free tier on gitlab.com

I have more material than we have time; feel free to ask questions when needed and TAs or myself will try and answer

7 of 162

Help during tutorial

In creating the materials, I have a reference GitLab group with most everything complete

https://gitlab.com/my-new-group8733610
Feel free to use as needed, yet…

Suggestion: try to follow along, ask questions, and use TAs

US-RSE Slack:

Channel: #et-seminar-gitlab-tutorial-march-2023
Wanna join? It is free! https://us-rse.org/join/

8 of 162

Legend for presentation

Red arrow => bring attention to something or click on a button in the web page

Green + white arrow => navigate to a different page

Typing keyboard => hands-on parts of tutorial

Keyboard typing image from from Flaticon.com

9 of 162

Introduction to GitLab

10 of 162

GitLab

GitLab is primarily a source code management platform

version control; based on git
https://gitlab.com

Good reference:

https://gitlab.com/help

Similar to:

GitHub, BitBucket, Gitea, Cgit

12 of 162

GitLab vs. GitHub

GitLab is Open Source; GitHub is not

GitLab group: https://gitlab.com/gitlab-org/

GitLab and GitHub feature sets are almost identical; some small variations

Example: GitLab allows groups inside of groups: GitHub has a “flatter” project / repository structure

GitLab can be self-hosted / self-managed (for free); GitHub Enterprise is a product you must purchase

GitLab docs on self-hosting: https://about.gitlab.com/install/
GitHub Enterprise: https://github.com/enterprise

13 of 162

GitLab vs. GitHub

Image from: https://about.gitlab.com/blog/2016/01/27/comparing-terms-gitlab-github-bitbucket/

14 of 162

Creating a GitLab Account

15 of 162

Create a GitLab Account

https://gitlab.com/users

16 of 162

Create a GitLab Account

Difference between creating a GitLab Account and using the buttons at the bottom?

Username and password: simply creates a new GitLab account.
Buttons: re-use an account on another platform to setup your GitLab account

Called Single-Sign On (SSO) since you can login once and reuse that login in a browser session without signing in to multiple accounts (i.e. Google for GMail in one tab, Google Drive in another tab, GitLab in another tab…)

Yet, realize with SSO you will “share” info with GitLab!...

17 of 162

Create a GitLab Account using Google SSO

Will share following info w/ GitLab

Name
Email
Language preference
Profile picture

18 of 162

Create a GitLab Account using GitHub SSO

Will share following info w/ GitLab

…nothing???
…or just don’t state what is shared

19 of 162

Create a GitLab Account using Twitter SSO

Will share following info w/ GitLab

Tweets (all of them)
Profile info
Account settings
Twitter accounts you follow

20 of 162

Create a GitLab Account using Bitbucket SSO

Will share following info w/ GitLab

…nothing???
…or just don’t state what is shared

Bitbucket also has SSO options so could potentially use those for Bitbucket and use Bitbucket for GitLab!

21 of 162

Create a GitLab Account using Salesforce SSO

Will share following info w/ GitLab

…nothing???
…or just don’t state what is shared

22 of 162

Which “Create a GitLab Account” option to use?

Which one do you use???

Daily, I personally use GitHub SSO option
I’m on both of these platforms already
Reduces my number of logins I have to remember / save in something like LastPass, 1Password, etc.

Okay… but what do you suggest today?

If this is your first time, I would just make an account (will go over in hands-on momentarily)
You can always use the SSO options later and transfer projects to that account later.

23 of 162

Which “Create a GitLab Account” option to use?

If you do chose SSO, one disclaimer:

What does this mean?

When you use HTTPS for git “push and pull”, you enter a password.
YET, you used SSO so there is not “GitLab password” for your identity.
So you either set a password afterwards or generate a token

If you use SSH for git “push and pull”, disregard

24 of 162

Let’s create our GitLab account…

HANDS-ON:

Go to https://gitlab.com/users
Create a GitLab account (username + password)
Get to the page that looks something like this…

25 of 162

Create a GitLab Account using Username + Password

Click “Register”
Enter information requested

26 of 162

Create a GitLab Account using Username + Password

Confirm Email
Login using password
Pick reasonable options for GitLab set up

27 of 162

Create a GitLab Account using Username + Password

Will create a default group + project (will go over this again in next section)
Can change this later if needed

28 of 162

Create a GitLab Account using Username + Password

29 of 162

Working with Projects and Groups in GitLab

30 of 162

Yet, we have to get started somewhere…

Objectives:

Create new projects in GitLab for your software
Create groups to associated multiple projects together in GitLab
Move projects around groups for future changes in structure

HANDS-ON:

Get to your GitLab dashboard
Make a project in GitLab
Make a group in GitLab
Move project into the group

31 of 162

Get to your GitLab dashboard

32 of 162

Make a project in GitLab

33 of 162

Make a project in GitLab

34 of 162

Make a project in GitLab

35 of 162

Make a project in GitLab

36 of 162

You just created a project on GitLab!
Instructions provided to get started working locally if you chose without README

Without README initialized

With README initialized

37 of 162

Make a group in GitLab

38 of 162

Make a group in GitLab

39 of 162

Make a group in GitLab

NOTE: Most of the options are to help personalize the group view. Examples:

Role
Who will be using this group?
What will you use this group for?

You can invite members now or later on

40 of 162

Make a group in GitLab

You just created a group on GitLab!
Can create a new project (what we just did previously) or a new sub-group (nested groups)

41 of 162

Groups / Subgroups / Projects example from GitLab

Group

Subgroup

Project

42 of 162

…vs. GitHub

Organization

Repository

GitLab Group == GitHub Organization
GitLab Project == GitHub Repository
GitHub doesn’t have a “subgroup” equivalent; “flatter” repository grouping structure

43 of 162

Move project into the group

44 of 162

Conclusion

Now able to…

…make projects to work across a team of developers
…make groups to organize multiple software projects
…move projects around groups for changes in structure

Moving on, you might want to bring in pre-existing projects

Going to go over forking and cloning in GitLab

45 of 162

Fork / Clone Existing Projects in GitLab

46 of 162

Bring in pre-existing projects…

Objective:

Add pre-existing projects to your own software project and groups

HANDS-ON / DEMO:

Fork an existing GitLab project into your namespace
(Demo only) Clone an existing GitLab project into your namespace

47 of 162

Fork an existing GitLab project into your group

Example to fork: https://gitlab.com/my-new-group8733610/my-new-repo

Click “Fork”

48 of 162

Fork an existing GitLab project into your group

Mainly, need to select a namespace / group
Optionally, can change: project name, “slug” (the /my-repo part in the URL), visibility, and description

49 of 162

Fork an existing GitLab project into your group

New namespace

Same commits

Maintains “upstream” of fork repo

No issues carried over to fork repo

50 of 162

Fork an existing GitLab project into your group

Another example: https://gitlab.com/ase/ase

Importance of forking:

Modify existing projects for your own purpose

Provide fixes to dependencies / projects you depend on (usually must fork to contribute)

51 of 162

Clone an existing GitLab project into your group

DEMO ONLY

Unless you have SSH keys setup and want to follow along

Use when, for some reason you can’t fork…

Clone repository locally. Example:

git clone git@gitlab.com:my-new-group/my-new-repo.git

52 of 162

Clone an existing GitLab project into your group

Create new, blank project

53 of 162

Clone an existing GitLab project into your group

Get to cloned repo and check the “origin:

cd /path/to/clone/of/repo
git remote -v

origin git@gitlab.com:my-new-group/my-new-repo.git (fetch)

origin git@gitlab.com:my-new-group/my-new-repo.git (push)

Change origin to new repository

git remote set-url origin git@gitlab.com:my-new-group/my-new-repo-clone.git

“Push” clone repository to this new repository

git push -u origin main

54 of 162

Clone an existing GitLab project into your group

55 of 162

Clone an existing GitLab project into your group

Why did you show us this?!?

This is a strategy that works to move projects between any of the “git” platforms (i.e. GitHub, GitLab, Bitbucket)

Fork only works on the same instance and platform

Have come across it enough times that I think it is useful to include

56 of 162

Conclusion

Now able to…

…add pre-existing projects to your own software project and groups

Next, we assume your project work is growing. You will want to effectively manage the work on your software project.

Going to go over creating issues / tasks and associating work that addresses these issues / tasks.

57 of 162

Working with Issues to Organize Project Work in GitLab

58 of 162

How GitLab can help you manage your work

Objectives:

Create issues to track software development
Identify if an issue is a task, defect, etc.
Submit work to close out an issue

HANDS-ON:

Create an issue
Create a label
Create a group label
Make a feature branch from this issue
Use Web IDE
Create a Merge Request to address the issue

59 of 162

Create an issue

60 of 162

Create an issue

61 of 162

Create a label

62 of 162

Create a label

63 of 162

Create a label

64 of 162

Create a label

65 of 162

Create a group label

66 of 162

Create a group label

versus…

67 of 162

Create a group label

68 of 162

Create a group label

Must re-assign the label to the issue since we deleted the repository-level issue for a group-level issue

69 of 162

Make a feature branch from this issue

70 of 162

Use Web IDE

Web IDE: useful for small changes directly in the browser!

71 of 162

Use Web IDE

Recently (for me) switched to VSCode! So apologies, I’m learning the new Web IDE layout…

72 of 162

Use Web IDE

73 of 162

Create a Merge Request to address the issue

74 of 162

Create a Merge Request to address the issue

NOTE: These would be done by a reviewer

75 of 162

Create a Merge Request to address the issue

76 of 162

Create a Merge Request to address the issue

77 of 162

Conclusion

Now able to…

…create issues to track work on tasks, defects, new features, etc.
…create labels to identify an issue is a task, defect, etc.
…create a branch to address the issue directly
…use the Web IDE
…submit a merge request (pull request in GitHub) using the branch to close out the work for the task

Next, assume your team is growing (since you project work is growing)! Need to effectively manage your software project even more.

Going to go over creating issues templates, issue boards, and milestones to match issues to deadlines

78 of 162

Issues Templates, Issue Boards, and Milestones in GitLab

79 of 162

How GitLab can help you manage your changes

Objectives:

Streamline creating types of issues with structured templates
Manage issue boards to visualize + track project work
Manage milestones to track work required by deadlines

HANDS-ON:

Add an issue templates
Create a few issues using the templates + add labels
Create an issue board
Add a milestone

80 of 162

Add an issue template

81 of 162

Add an issue template

Click “New Folder”
Specifically name “.gitlab/issue_templates”

82 of 162

Add an issue template

Create markdown files

Name of file will be in drop-down for new issues

Templates courtesy of John Hetrick, Principal Architect in Software Engineering Group

83 of 162

Add an issue template

84 of 162

Add an issue template

85 of 162

Create an issue board

Create the following labels for the group

86 of 162

Create an issue board

Add labels to the previous issue created

87 of 162

Create an issue board

Add a few more issues and vary the labels

88 of 162

Create an issue board

89 of 162

Create an issue board

Create following board layout

90 of 162

Add a milestone

91 of 162

Add a milestone

92 of 162

Add a milestone

Assign a few of the issues to the milestone

93 of 162

Add a milestone

Look at milestone chart

94 of 162

Advice on Issue Board vs. Milestones + extra

You need both!

Issue board is “global” view of issues
Milestones group issues for delivery dates
Can move issues across milestones in issue board

Things I haven’t covered…

Iterations - cycles within milestones; come from Agile practices
Epics - at group-level; for large feature changes that share a theme across projects and milestones; use for high-level discussion
Epic Boards - one level higher than Issue boards

95 of 162

Conclusion

Now able to…

…create issue templates to structure issue layouts
…create issue boards to visualize + track project work
…create milestone to map issues to deadlines

Next, noticing the number of bugs is increasing and code not looking “clean. Need to automate code checks your software project.

Going to go over setting up continuous integration, or the automation of integrating new code changes (i.e. build, lint, test)

96 of 162

Continuous Integration in GitLab: Automating Code Checks

97 of 162

Continuous Integration (CI)

Image from: https://docs.gitlab.com/ee/ci/introduction/index.html#gitlab-cicd-workflow

98 of 162

GitLab CI vs. GitHub Actions

GitLab CI == GitHub Actions

GitLab CI: https://docs.gitlab.com/ee/ci/
GitHub Actions: https://docs.github.com/en/actions

Both are pretty similar, more overlap than differences

Migrating from GitLab CI -> Github Actions: https://docs.github.com/en/actions/migrating-to-github-actions/manual-migrations/migrating-from-gitlab-cicd-to-github-actions

99 of 162

How GitLab can help you automate code checks

Objectives:

Set up automated code checks via continuous integration
Find GitLab-provided templates so you don’t start from scratch every time

HANDS-ON:

Setup CI pipeline
Find CI pipeline after setup

100 of 162

Setup CI pipeline

100

101 of 162

Setup CI pipeline

By default, CI pipeline is defined by the file .gitlab-ci.yml

101

102 of 162

Setup CI pipeline

Browse the templates!

102

103 of 162

Setup CI pipeline

With the CI Editor open…

Copy and paste the Bash template provided by GitLab as your .gitlab-ci.yml file

https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml

Commit the .gitlab-ci.yml file

103

104 of 162

Find CI pipeline after setup

104

105 of 162

What do I need to run CI?

To run continuous integration and/or continuous delivery / deployment pipelines, you need “runners”

GitLab CI Runner
“GitLab Runner is an application that works with GitLab CI/CD to run jobs in a pipeline.”
Gitlab.com provides runners by default

many-to-many relationship

Runner

Project

105

106 of 162

What do I need to run CI?

Self-managed Runner: need to install the GitLab CI Runner software on a server that acts as an agent to run CI jobs

Can provide your own unique hardware for a runner:

GPU, *Cloud compute resources, ARM, etc.

*Maybe look into infrastructure-as-code to manage…

GitHub: self-hosted runners

Runner

Project

Compute images from from Flaticon.com

106

107 of 162

Conclusion

Now able to…

…create file that sets up CI pipeline
…find GitLab-provided templates for certain code bases
…run and find CI pipeline results

Next, you might need a place for your software artifacts for Users to download or find the software service deployment is very manual. Need to automate the final step to “operations” of your software project.

Going to go over software artifacts available on GitLab,

107

108 of 162

Software Artifacts and Continuous Delivery / Deployment in GitLab

108

109 of 162

What about my software artifacts?

GitLab has artifact registries for:

Packages
Container Images
Infrastructure-as-Code modules

CD: continuous delivery or continuous deployment

automation of publishing artifacts and / or deploying

109

110 of 162

Continuous Delivery / Deployment (CD)

Image from: https://docs.gitlab.com/ee/ci/introduction/index.html#gitlab-cicd-workflow

110

111 of 162

Wait… continuous delivery and continuous deployment… what is the difference???

My favorite reference

Atlassian docs on continuous delivery vs. continuous deployment

Figure from: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment

111

112 of 162

Extend CI to publish artifacts

Objective:

Set up automated artifact publishing via continuous delivery / deployment

HANDS-ON / DEMO:

Add a packaging step to .gitlab-ci.yml
Add a container build + publish step to .gitlab-ci.yml

NOTE: This part will be less hands-on and more demonstration

112

113 of 162

Add a packaging step to .gitlab-ci.yml

Reference project:

MCPI (Monte Carlo to calculate 𝜋)
https://gitlab.com/my-new-group8733610/mcpi

Simple Python project to show packaging and publishing via GitLab CI

Publish Python package to GitLab PyPi Package Repository

GitLab reference docs

Package registries: https://docs.gitlab.com/ee/user/packages/package_registry/
PyPI package registry: https://docs.gitlab.com/ee/user/packages/pypi_repository

113

114 of 162

Add a packaging step to .gitlab-ci.yml

Packaging job in .gitlab-ci.yml

Link to lines-of-code

package:

stage: deploy

script:

- pip install build twine

- python -m build

- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

only:

- tags

114

115 of 162

Add a packaging step to .gitlab-ci.yml

package:

stage: deploy

script:

- pip install build twine

- python -m build

- TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*

only:

- tags

Using PyPA’s build to build the package

generating a source-distribution and wheel in dist/

Using PyPA’s twine to upload the package

Tool to interact with PyPI

Using pre-defined GitLab CI variables for URL and authentication

115

116 of 162

Add a packaging step to .gitlab-ci.yml

Create a tag to “kick off” publishing a new package version

116

117 of 162

Add a packaging step to .gitlab-ci.yml

Create a tag to “kick off” publishing a new package version

117

118 of 162

Add a packaging step to .gitlab-ci.yml

118

119 of 162

Add a packaging step to .gitlab-ci.yml

119

120 of 162

Add a container build + publish step to .gitlab-ci.yml

(Same) reference project:

MCPI (Monte Carlo to calculate 𝜋)
https://gitlab.com/my-new-group8733610/mcpi

Simple Python project to show working with building and publishing container images via GitLab CI

GitLab reference docs

Container registries: https://docs.gitlab.com/ee/user/packages/container_registry/
Using Docker to build docker images: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

120

121 of 162

Add a container build + publish step to .gitlab-ci.yml

Building job in .gitlab-ci.yml

Link to lines-of-code

image: python:latest

services:

- docker:dind

…

container-build:

image: docker:latest

before_script:

- docker --version

script:

- docker build -t mcpi .

121

122 of 162

Add a container build + publish step to .gitlab-ci.yml

image: python:latest

services:

- docker:dind

…

container-build:

image: docker:latest

before_script:

- docker --version

script:

- docker build -t mcpi .

Using docker image provides all the docker tools

Overriding the “global” python:latest image

Using Docker-in-docker (dind) service to allow GitLab CI runners to access docker daemon (required to spin up other docker containers)

122

123 of 162

Add a container build + publish step to .gitlab-ci.yml

123

124 of 162

Add a container build + publish step to .gitlab-ci.yml

Publishing job in .gitlab-ci.yml

Link to lines-of-code

container-package:

stage: deploy

image: docker:latest

before_script:

- docker --version

- docker login --username=$CI_REGISTRY_USER --password=$CI_REGISTRY_PASSWORD $CI_REGISTRY

script:

- docker build -t mcpi .

- docker tag mcpi $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA

- docker tag mcpi $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest

- docker push $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA

- docker push $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest

124

125 of 162

Add a container build + publish step to .gitlab-ci.yml

container-package:

stage: deploy

image: docker:latest

before_script:

- docker --version

- docker login --username=$CI_REGISTRY_USER --password=$CI_REGISTRY_PASSWORD $CI_REGISTRY

script:

- docker build -t mcpi .

- docker tag mcpi $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA

- docker tag mcpi $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest

- docker push $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:$CI_COMMIT_SHORT_SHA

- docker push $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_NAME:latest

Re-building docker image, using docker tag to set the correct container registry name, and then pushing to registry

125

126 of 162

Add a container build + publish step to .gitlab-ci.yml

126

127 of 162

Add a container build + publish step to .gitlab-ci.yml

127

128 of 162

Conclusion

Now able to…

…set up automated package publishing via tags using CD
…set up automated container image publishing using CD

128

129 of 162

Last Few Notes…

129

130 of 162

Diverse team of software engineers, scientists, students…

Will want to look into GitLab Roles

GitLab Roles / Permissions: https://docs.gitlab.com/ee/user/permissions.html

Permissions in GitLab are Role-based

Roles: Guest, Reporter, Developer, Maintainer, Owner
Example statement: Can create / delete X if role is “developer” but can only view X if role is “guest”

Useful to limit what a project member can do (“guardrails”)

130

131 of 162

Example of Project Team => Gitlab Roles

Project Role	GitLab Role	Explanation
Principal Investigator / Lead Developer	Owner / Maintainer	Full control of software project in GitLab
Research Software Engineer / Computational Scientist / Scientific Software Developer / Operations Engineer	Maintainer / Developer	Developer who will use the platform regularly; provide enough permissions to cover for others in higher-permission roles
Junior Developers / Students / Interns	Developer / Reporter / Guest	Limit permissions to map to scope of work; guardrails for stuff like deleting a project / force push
Collaborators / Non-Developer Experimental or Instrument Scientist	Reporter / Guest	Can help create issues to report bugs or request new features; mostly view-only role

131

132 of 162

Visibility of Projects and Groups

Group

Project in Group

NOTE: Private group must have all projects private

132

133 of 162

Example ORNL Scientific Project in GitLab

133

134 of 162

Example GitHub vs. GitLab for ORNL Project

134

135 of 162

Conclusions and Wrap-up

135

136 of 162

Conclusions

Covered lots of GitLab features!

Creating a GitLab Account
Working with Projects and Groups in GitLab
Fork / Clone Existing Projects in GitLab
Working with Issues to Organize Project Work in GitLab
Issues Templates, Issue Boards, and Milestones in GitLab
Continuous Integration in GitLab: Automating Code Checks
Software Artifacts and Continuous Delivery / Deployment in GitLab

Lots of this is transferable over to GitHub

Open for feedback on the materials (too much of anything? something not covered? more context material related to scientific software?)

136

137 of 162

Thank you!

Questions?

Feedback:

US-RSE Slack:

Channel: #et-seminar-gitlab-tutorial-march-2023
Wanna join? It is free! https://us-rse.org/join/

Email: mcdonnellmt@ornl.gov

137

138 of 162

Extra Exercises / Homework

138

139 of 162

Exercises not covered in the live hands-on…

HANDS-ON / DEMO:

Setting up SSH keys locally with GitLab
Upload a pre-existing project to the repository
Additional practice for forking and cloning

139

140 of 162

Setting up SSH keys locally with GitLab

SSH keys save typing passwords for every git command

GitLab docs: https://docs.gitlab.com/ee/user/ssh.html

Overall idea: create public + private key pair and put public key on GitLab that will match your local private key on your computer for authentication

Notes:

Need a key pair (private + public) stored in GitLab for each machine you use
I feel mandatory for a developer / software engineer

140

141 of 162

Setting up SSH keys locally with GitLab

Create a public and private key on your machine:

ED25519: ssh-keygen -t ed25519 -C "<comment>"
RSA: ssh-keygen -t rsa -b 2048 -C "<comment>"

Copy the public key (not the private key!!! That is a secret!)

cat ~/.ssh/id_ed25519.pub | xclip

Upload public key to GitLab (Preferences -> SSH Keys)

141

142 of 162

Setting up SSH keys locally with GitLab

142

143 of 162

Setting up SSH keys locally with GitLab

Test SSH key on local machine

ssh -T git@gitlab.com

Accept the authenticity of GitLab (“yes”):

The authenticity of host 'gitlab.com (172.65.251.78)' can't be established.

ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.

Are you sure you want to continue connecting (yes/no/[fingerprint])?

143

144 of 162

Upload a pre-existing project to the repository

cd /path/to/local/repo
git remote -v

No origin?

git remote add origin <remote repo>

One exists?

First, maybe don’t use that repo… BUT, if you do…
git remote set-url origin <remote repo>

Git push -u origin <remote branch>

144

145 of 162

Additional practice for forking and cloning

Need some additional practice with forking and cloning on projects?

Good reference project to get started on GitHub!

Don’t know if same exists for GitLab

…yet, pretty much same between the two if you know how to fork in GitLab

Thanks to Steve Hahn @ ORNL for reference!

145

146 of 162

Backup Slides

146

147 of 162

Add an Epic

Apologies, this is a premium feature… :(
Overall, Epics help group multiple issues across repositories that are related

147

148 of 162

149 of 162

But… what is DevOps?

On the site, GitLab defines DevOps as the following:

DevOps is a combination of software developers (dev) and operations (ops). It is defined as a software engineering methodology which aims to integrate the work of software development and software operations teams by facilitating a culture of collaboration and shared responsibility.

Figure from: https://about.gitlab.com/topics/devops/

149

150 of 162

Where is all this GitLab “DevOps stuff?

Honestly, most is in this sidebar menu

Issue management
CI/CD
Artifact registries
Integrations to tools for:

Monitoring
Tracing
Logging
Infrastructure-as-Code

Documentation for it all!

….it does way more than just version control!

150

151 of 162

What if I need a deployment?

Very likely if you have a software service / web application

You need one or many deployment environments

DEV, TEST, NEXT, QA, PROD, etc.

Well… you need infrastructure again!

Yet, we are making lots of infrastructure now…

…what if we could “code” our infrastructure?!?

151

152 of 162

Where do I run software services?

Very likely if you have a web application

You need one or many deployment environments

DEV, TEST, NEXT, QA, PROD, etc.

Well… you need infrastructure!

Making lots of infrastructure now…

…what if we could “code” our infrastructure?!?

152

153 of 162

What is Infrastructure-as-Code?

Development and Operations (DevOps)

methodology of developing and maintaining reliable software
“culture shift” towards deployment (not just development)

Software is typically deployed and run on infrastructure

Examples: physical servers for scientific instruments, virtual machines on either virtual platforms or cloud, high-performance computing, physical or virtual networks, data storage, etc.

Infrastructure-as-Code (IaC)

a core DevOps practice to capture the requirements and automate the creation of this infrastructure instead of a team manually documenting, managing, and configuring these resources.

153

154 of 162

Infrastructure-as-Code vs. Configuration Management?

IaC (repeated)

“...automate the creation of this infrastructure…”

Configuration Management

a process for maintaining computer systems, servers, and software in a desired, consistent state.

My opinion:

IaC creates the infrastructure
Configuration management configures that infrastructure
They work best together; One does not replace the other

154

155 of 162

What is the Use Case for Infrastructure-as-Code?

It is almost “implied” but IaC is for cloud computing

Hard to create infrastructure using code for physical servers… sort of need an API to do so.

You can still manually create infrastructure in cloud computing via a Graphical User Interface

Example: https://aws.amazon.com/
Example: https://cloud.cades.ornl.gov/

Want to create, scale, and destroy your infrastructure for:

Reproducibility
Speed
Cost

155

156 of 162

What are the benefits using Infrastructure-as-Code?

Benefits for the adoption of IaC are:

Reduction of cost by automating deployments
Increased speed of deploying infrastructure and applications
Reduction of errors and better consistency with deployments.

Case studies exemplifying the IaC benefits:

Ambit Energy deploying x1,200 faster^[1]
TMX Group increasing infrastructure provisioning time by 75% ^[2]
Snowflake reducing cluster provisioning from a week to less than a day^[3]

[1] Puppet Labs, “Ambit Energy’s customer story for deploying 1,200 times faster due to IaC with Puppet” 2021. [Online]. Available: https://puppet.com/resources/customer-story/ambit-energy

[2] R. M. Mark Salam, “Up and to the right” 2021. [Online]. Available: https://www.hashicorp.com/case-studies/tmx-group

[3] Pulumi, “Pulumi’s case study for snowflake, a data platform.” [Online]. Available: https://www.pulumi.com/case-studies/snowflake

156

157 of 162

What are the tools for Infrastructure-as-Code?

My opinion: “top competitors”

Terraform
Pulumi

Others

AWS CloudFormation
*Ansible
*Puppet
*Chef
*SaltStack
…

*more “configuration management” tools than IaC-specific

157

158 of 162

What are the tools for Infrastructure-as-Code?

My opinion: “top ones”

Terraform
Pulumi

Others

AWS CloudFormation
*Ansible
*Puppet
*Chef
*SaltStack
…

*more “configuration management” tools than IaC-specific

158

159 of 162

Revisited: IaC vs. Configuration Management?

Figure from: https://www.ibm.com/cloud/blog/chef-ansible-puppet-terraform

159

160 of 162

What is Terraform?

Terraform is an open-source CLI tool for IaC created by HashiCorp

GitHub: https://github.com/hashicorp/terraform
Same company that created Vagrant, Packer, Vault, Nomad, etc.

Languages

Written in Golang
You use a DSL called HashiCorp Configuration Language (HCL)

Is “declarative”

“...describe the desired end result rather than outlining all the intermediate work steps…”^[1]
Side note: Kubernetes has same model for container orchestration

[1] Source: https://www.ionos.com/digitalguide/websites/web-development/declarative-programming/

160

161 of 162

How does Terraform work?

Terraform is created to talk to APIs

Cloud providers
Docker
Kubernetes

Write HCL to describe the state of infrastructure you want
Terraform stores this “state” of infrastructure
When you make a change to ask for a new “state”, Terraform will:

Look at you current state that it has stored
Look at your new desired state that you write in HCL
Create a “plan” for this state change
Modify your infrastructure to move from state N -> N+1

161

162 of 162

Terraform as a Team via CI/CD Pipeline

162

Networking

Servers

Infrastructure “Recipe”

Automated Pipeline

162