1 of 15

Using Globus v5 at NCAR

November 9, 2022

Brian Vanderwende

CISL Consulting Services

This material is based upon work supported by the National Center for Atmospheric Research, which is a major facility sponsored by the National Science Foundation under Cooperative Agreement No. 1852977.

2 of 15

Reviewing the Globus services provided by CISL

From multiple interfaces:

  • www.globus.org
  • Globus CLI (via NPL or data-access)
  • Globus Python API

2

Using Globus v5 at NCAR

Storage Endpoints

NCAR GLADE

NCAR Campaign Storage

NCAR Quasar

NCAR Stratus S3

NCAR AWS S3

Data Sharing Service

  • Space by request in /glade/p/datashare
  • Accessible by user-owned “shared endpoint”
  • Also possible to share data from Campaign Storage

​

Unattended Workflows

  • Access to NCAR endpoints using InCommon certificates
  • gci command to facilitate using certificates with Globus CLI

​

3 of 15

What’s changed in version 5 of Globus?

Terminology: users now interact with collections rather than endpoints

​

  • Mapped collection - a managed collection of files hosted on an endpoint. CISL collections like NCAR GLADE are of this type
  • Guest collections - a user-created subset of files from a mapped collection with distinct Globus sharing permissions (formerly shared endpoints)

​

Authentication: mapped collections use OpenID (OAuth) identity mapping rather than myproxy authentication

3

Using Globus v5 at NCAR

4 of 15

How do the changes affect Globus usage at NCAR?

4

Same names and UUIDS!

NCAR GLADE

NCAR Campaign Storage

NCAR Quasar

NCAR Stratus S3

NCAR AWS S3

Flexible data sharing

  • Most storage spaces can be shared via guest collections
  • Permissions model that layers Globus access on top of native permissions
  • Guest collections can be used to create URLs to files on a storage system

Unattended workflows

  • Create a guest collection (or two) for desired spaces
  • Grant self open access to data and use guest collections in CLI/API commands instead of mapped collections

​

Mapped Collections

Guest Collections

Using Globus v5 at NCAR

Data Sharing Service is deprecated

InCommon certificates no longer supported!

New OIDC authentication process

5 of 15

Authenticating to the new NCAR mapped collections

5

Using Globus v5 at NCAR

6 of 15

For each mapped collection, you will need to consent to access data via a linked identity the first time you use it

The “linked identity” will be your HPC account on Cheyenne/Casper �(NCAR OIDC Authentication)

6

Using Globus v5 at NCAR

7 of 15

Use your Cheyenne username and Duo token/passphrase

7

Using Globus v5 at NCAR

8 of 15

Finally, you will need to consent to www.globus.org accessing your data on the collection

8

Using Globus v5 at NCAR

9 of 15

Creating a guest collection: choose the root directory

9

Using Globus v5 at NCAR

10 of 15

Creating a guest collection: provide collection metadata

10

Using Globus v5 at NCAR

11 of 15

Creating a guest collection: managing permissions

11

Using Globus v5 at NCAR

use public sharing for direct URLs to files

avoid write permissions unless justified

12 of 15

Using our mapped collections in the Globus CLI also requires linking an identity and consent to access data

The collection you are trying to access data on requires you to grant consent for the Globus CLI to access it.

message: Missing required data_access consent

​

Please run

​

globus session consent 'urn:globus:auth:scope:transfer.api.globus.org:...'

​

to login with the required scopes

​

Even if you do this in the Web interface, you will still need to perform a one-time consent in the CLI too (as well as any custom Python APIs)

12

Using Globus v5 at NCAR

13 of 15

Current limitations of the Globus CLI

  1. Guest collections - these can be queried and managed via the CLI, but currently can only be created in the web interface
  2. Terminology - endpoint and collection are confusingly used by some CLI commands:��e.g., globus endpoint search is still used to find collection UUIDS�
  3. To search for our collections, use ncar@globusid.org for the search phrase, not the owner:��NO: globus endpoint search --filter-owner-id ‘ncar@globusid.org’ NCAR�YES: globus endpoint search ‘ncar@globusid.org’

13

Using Globus v5 at NCAR

14 of 15

Globus v5: Bottom Line for Different Workflows…

  1. User of NCAR host endpoints - new authentication method but otherwise no changes (collections keep the same name and UUIDs).�
  2. Data sharing service users - existing shared endpoints have become guest collections with same name and UUIDs. New users are instructed to create guest collections directly on work/scratch/campaign.�
  3. Unattended transfers - myproxy-based InCommon Certificates no longer work, and gci/gcert have been retired. Switch to guest collections with open-access for owner (will still require Globus login)�
  4. Data publishers - new URL-sharing provides direct access to files

14

Using Globus v5 at NCAR

15 of 15

Getting assistance from the CISL Help Desk

https://www2.cisl.ucar.edu/user-support/getting-help

  • Walk-in: ML 1B Suite 55
  • Web: https://rchelp.ucar.edu
  • Phone: 303-497-2400

​

Specific questions from today and/or feedback:

  • Email: vanderwb@ucar.edu

​

​

15

Using Globus v5 at NCAR