Introduction to OpenStack

1

1

Introduction to OpenStack Agenda

​

• Introductions and Context
• Part 1: What is OpenStack?
• Part 2: What is Ceph?
• Part 3: Operating OpenStack and Ceph using Kayobe
• Part 4: Self-Paced Learning

​

2

Adopting DevOps & ResOps in HPC

Four key measures of Software Delivery Performance:�

• Lead Time:�from customer request to being satisfied
• Mean Time to Restore (MTTR):�failure will happen, get good recovery
• Change Fail Percentage:�a proxy for quality throughout the process
• Deployment Frequency:�a proxy for small batch size

3

DevOps: “The Three Ways”

4

HPC Stack 1.0

HPC Specific Orchestration

HPC Hardware

Big Data Specific Orchestration

Big Data Hardware

AI Specific Orchestration

AI and Deep Learning Hardware

HPC

Big Data

AI and Deep Learning

5

HPC Stack 2.0

Big Data Specific Orchestration

Big Data Hardware

AI Specific Orchestration

AI and Deep Learning Hardware

HPC

AI and Deep Learning

Shared Configurable Hardware

APIs: AAAI, Storage Services, Compute Services

Big Data

6

HPC Stack 2.0

7

Scientific OpenStack

8

Scientific OpenStack

Scientist

Platform Operator

Infrastructure�Operator

Data

Custodian

Federation�Manager

Funding�Decision Maker

9

HPC Stack 2.0

Science Workflows

10

Traditional HPC and AI

HPC Specific Orchestration

HPC Hardware

Big Data Specific Orchestration

Big Data Hardware

AI Specific Orchestration

AI and Deep Learning Hardware

HPC

Big Data

AI and Deep Learning

11

Convergence of HPC, AI and Cloud

Big Data Specific Orchestration

Big Data Hardware

AI Specific Orchestration

AI and Deep Learning Hardware

HPC

AI and Deep Learning

Shared Configurable Hardware

APIs: AAAI, Storage Services, Compute Services

Big Data

12

DevOps Research and Assessment

Four key measures of Software Delivery Performance:�

• Lead Time:�from customer request to being satisfied
• Mean Time to Restore (MTTR):�failure will happen, get good recovery
• Change Fail Percentage:�a proxy for quality throughout the process
• Deployment Frequency:�a proxy for small batch size

13

Get Maximum Value from your Investments

https://unsplash.com/photos/2FaCKyEEtis

14

Get Maximum Value from your Investments

15

Self-Service HPC and AI

Reconfigurable and Isolated Infrastructure

Performance to extract maximum value

Azimuth Self-Service Reproducible Apps

16

LOKI: The OpenInfra Standard

More than 85% of OpenStack deployments include Kubernetes�(OpenStack 2022 User Survey)

• Linux�Open Operating System standard
• OpenStack�Open Cloud standard
• Kubernetes�Open Container Orchestration standard
• Infrastructure

17

Introduction to OpenStack

Part 1: What is OpenStack?

18

18

What is OpenStack?

• Defines itself as a cloud operating system
• Controls compute, storage, and networking (IaaS)
• Makes them manageable and available via APIs
• Can manage compute resources as bare-metal, VMs, or containers
• Composed of many loosely coupled services

19

Services

20

Services

21

Core Services of OpenStack

• Keystone: identity service
• Service catalog, authentication, authorisation, etc.
• Nova: compute service
• Manages compute instances (virtual through a hypervisor or bare-metal via Ironic)
• Works closely with Placement which allocates resources for the Nova scheduler
• Neutron: networking service
• Manages virtual networking: networks, routers, security groups…
• Glance: image service
• Stores virtual machine images and snapshots
• Cinder: block storage service
• Provides remote disks to compute resources (similar to Amazon EBS)

22

Release Policy

• New OpenStack version every six months
• Release name instead of number, follows the latin alphabet
• Recent past releases: Ussuri, Victoria, Wallaby, Xena
• Next releases: Yoga (April 2022), Zena, then wraps to A, B…
• Release schedule at https://releases.openstack.org/
• Regular bugfix releases for 18 months
• Extended maintenance effort beyond 18 months (but no releases)
• Change of release cadence being discussed

23

General Architecture

• Each service focuses on:
• A specific kind of resources: compute, network, storage…
• A specific task: orchestration, bare-metal deployment, monitoring…
• Inter-service communication via HTTP
• Each service implemented with multiple processes
• Intra-service communication via RPC over AMQP (RabbitMQ)
• Relational data kept in SQL (MySQL / MariaDB)
• Common libraries for reusing code and patterns
• oslo.cache, oslo.config, oslo.log, oslo.messaging, oslo.middleware…

24

APIs, User Interfaces�and Orchestration

25

Interfaces to OpenStack

• Web interface: Horizon (official OpenStack Dashboard Project)
• Command-line clients: OpenStackClient (OSC), legacy per-service clients
• OpenStackSDK (Python)
• Python bindings for each project
• Third-party bindings for most popular languages (Go, Java…)
• Orchestration tooling: Ansible, Terraform, Kubernetes Cluster API, …

Now let’s connect to Horizon!

26

Getting to Know Azimuth

• Aim: Create Science platforms on demand
• Logging in, selecting a project
• Create a Machine
• Create a Kubernetes
• … and run JuypterHub
• Create a Slurm Cluster

27

Getting to Know OpenStack Horizon

• Logging in
• Selecting a project
• Looking at available panels
• Uploading SSH key
• Viewing instance details
• Download client environment script (OpenRC file)

28

Keystone

29

Keystone, the OpenStack Identity Service

Authentication

​

Authorization

​

Service discovery

30

Authentication and Authorization with Keystone

• Users: individual API consumer
• Groups: collection of users
• Projects: base unit of ownership in OpenStack
• Projects can be nested
• Domains: containers for users, groups, projects
• A domain is based on a source of user authentication
• Roles: grant authorisation to users or groups at project or domain level
• member role for regular users: gives access to resources in the project/domain
• admin role (access to all resources)
• There are other roles, like reader (can view all information but not write)
• Custom roles can be created and associated with custom policies
• Stores information in its database or uses an external source (LDAP)

31

Tokens and Applications Credentials

• Users are generally authenticated by a password
• After successful authentication, a token is generated
• Token is used to perform requests against other OpenStack services
• Tokens expire after a timeout (default is one hour)

​

• Application credentials: allow applications to authenticate against Keystone
• Can choose an expiration date (or none)
• Can list all application credentials

32

Service Discovery

• Service catalog: allows to discover available services
• Service made accessible via endpoints
• Endpoint: URL giving access to the root API resource
• Multiple endpoints for each service
• Admin, internal, and public endpoints
• Selected by choosing an interface
• Each endpoint is associated to a region

33

Keystone Hands On

• All the following commands require admin role
• List all services and endpoints in the catalog:
• openstack service list
• openstack endpoint list
• Show user and projects
• openstack user list
• openstack project list
• Show roles and their assignments
• openstack role list
• openstack role assignment list --names
• Show domains
• openstack domain list

34

Nova: Compute Service

35

Nova: Compute Service

• Manages compute resources (hypervisors) and instances (VMs)
• Can also provide bare-metal capabilities via Ironic
• Architecture
• Nova API REST service: nova-api
• Internal controller services: nova-conductor and nova-scheduler
• A nova-compute service runs on each hypervisor
• Other services for serial console or VNC
• Interacts closely with other services:
• Resource allocation from Placement
• Configuration of network ports from Neutron
• Downloads images from Glance to create VM disks
• Attaches Cinder volumes to VMs

36

Nova Concepts

• Servers or instances configured with…
• Image / volume: disk image or remote volume containing the OS to run
• Flavor: amount of resources for the instance: vCPU / memory / disk
• Key pairs: SSH public key to authorize
• Security groups: Network traffic to accept
• User data: Data to pass to instance, can be a script executed on boot
• Scheduler hints: e.g. affinity / anti-affinity server group
• Integration with cloud-init for key pairs and user data
• Server operations:
• create / delete (terminate)
• start / stop / reboot, pause / unpause, suspend / resume, shelve / unshelve
• add / remove floating IPs, network ports
• snapshot

37

Nova: Things to Know

Instance data

• Data for image-backed instances is not persistent
• Disk image is deleted when instance is terminated
• Image can be saved with a snapshot operation
• Cinder volumes should be used when persistence is essential
• Optional ephemeral disk
• Not included in snapshot

Flavor extra specs

• Hardware video RAM: useful to run virtual desktops
• Tune scheduling decisions

38

Nova Scheduler Overview

Default scheduler is FilterScheduler

39

Picking Nova Flavor Sizes

Maximum VM Performance

​

• Dedicated CPU threads, Huge Pages, and NUMA aware placement exposed to VM
• Isolate from other VMs
• Some memory and cores for hypervisor
• Optionally: SR-IOV network, GPUs
• Ultimate: Ironic baremetal flavor

​

Maximum VM Density

​

• Overcommit vCPUs
• But usually no memory overcommit
• VM will see CPU steal time
• … but still ensure flavors tessellate on given hardware type (Pizza slices)

40

Nova Scheduler Concepts

• Host aggregates: groups of nodes for specific scheduling decisions
• Example: send all requests of a specific tenant to a group of nodes
• Flavor extra specs
• Metadata on flavors to customise scheduling decisions
• Example: Schedule gpu flavor only to GPU nodes
• Image properties
• Metadata on images to customise scheduling decisions
• Example: Schedule windows images on servers with licenses
• PCI passthrough
• Server groups and affinity / anti-affinity

41

Glance: Image Service

42

Glance: Image Service

• Stores virtual machines images
• Many backends: file, Swift object store, Ceph…
• VMs can be snapshotted with the snapshot stored in Glance
• Public or private images
• Images can be shared with other projects
• Properties on images
• Override specific behaviors defined for Nova flavors
• Affect behavior of Nova scheduler or instance configuration on hypervisors
• Other kind of artifacts: kernel and ramdisks (e.g. used by Ironic to PXE boot)

43

Glance: Image Service

To identify OS inside the image, enable performance optimisations (libosinfo):

• os_type=linux
• os_distro=centos
• os_version=8.4.2105

To improve network performance:

• hw_vif_multiqueue_enabled=true ( flavors with 2+ vCPUs)

https://docs.openstack.org/glance/latest/admin/useful-image-properties.html

44

Cinder: Block Storage Service

45

Cinder: Block Storage Service

• Without Cinder, VM disks stored by hypervisors in local/remote filesystems
• With Cinder, virtual machines are attached directly to a remote block device via a storage protocol, e.g. Ceph RBD client
• Cinder can use various backends: LVM, Ceph, GPFS…
• Architecture: cinder-api, cinder-scheduler, cinder-volume, cinder-backup
• Volumes attached to instances as secondary storage (vdb, vdc…)
• Need to be formatted with a file system like a regular disk
• Boot instances from volume
• Backup capabilities
• SCP OpenStack is configured to back up volumes to SCP Object Store
• Backups need to be performed explicitly, it is not automatic

46

Instance Data: Example

​

​

​

​

​

​

​

​

​

​

Hypervisor

VM instance

Ephemeral disk

/mnt

Root disk

/

Cinder service

Cinder

volume

/var/lib/app

Glance service

Ubuntu image

copy of

47

Instance Data: Internals

Hypervisor

VM instance A

Ephemeral disk

/mnt

Root disk

(overlay)

VM instance B

Ephemeral disk

/mnt

​

Image cache

Ubuntu image

Root disk

(overlay)

48

Neutron: Networking Service

49

Neutron: Networking Service

• Manages virtual overlay networks for virtual machines
• Networks: tenant, provider, external
• Routers
• Fixed and floating IPs
• Architecture:
• neutron-server (API)
• Agents: DHCP, L3, metadata, OVS
• OVN (Open Virtual Network)
• Extension for VPNaaS
• SR-IOV (ethernet and infiniband)

50

Other OpenStack Services

51

Ironic

• Bare Metal service
• “Cloudy” multi-tenant user experience
• Users interact with Nova
• Interaction is very similar to virtual machines
• Flavor, image, network, SSH keypair, etc.
• May coexist with hypervisors
• Flavor is used to select bare metal
• Some caveats/limitations: no security groups, no snapshots
• Bootstrapping: automated node enrollments, inspection uses LLDP, cleaning to wipe disks and update firmware
• Networking: change to iPXE VLANs using networking-generic-switch

52

Barbican

• Key Manager service
• Stores secrets (keys, certificates, raw binary data)
• Simple Crypto Plugin: pure software encryption
• Can be configured to use hardware devices or HashiCorp Vault
• Stores keys for Cinder volume encryption

53

Octavia

• Load Balancing service
• Replaced Neutron LBaaS (retired in the OpenStack Train release)
• Load balancing implemented by amphorae instances
• Reference Amphora image: HAProxy and configuration agent
• Driver interface for interaction with other services
• High availability: ACTIVE_STANDBY topology, anti-affinity scheduling

54

Octavia

55

Network Attached Filesystems

• Provider VLANs can be modeled inside OpenStack
• SR-IOV can provide RDMA access to Ethernet Provider VLAN
• Flat Infiniband network can be passed through to a VM using SR-IOV
• … Infiniband partition management not yet been integrated
• OpenStack Manila can manage on-demand isolated CephFS shares
• Filesystems can be cloud workloads
• … or they can run “outside” the cloud on a provider VLAN
• You can enable access to Lustre, GPS, BeeGFS et al
• But users have root in VMs
• Lustre does allow for submounts and squashing (root, uid, gid)

56

OpenStack Summary

57

What is OpenStack?

• Defines itself as a cloud operating system
• Controls compute, storage, and networking (IaaS)
• Makes them manageable and available via APIs
• Can manage compute resources as bare-metal, VMs, or containers
• Composed of many loosely coupled services

58

Pulling it together: Make a VM

59

Scientific OpenStack

60

Introduction to OpenStack

Part 2: What is Ceph?

61

61

What is Ceph

• Ceph delivers object, block, and file storage in one unified system.
• Ceph is highly reliable, easy to manage, and free.
• Ceph delivers extraordinary scalability–thousands of clients accessing petabytes to exabytes of data.
• A Ceph Node leverages commodity hardware and intelligent daemons, and a Ceph Storage Cluster accommodates large numbers of nodes, which communicate with each other to replicate and redistribute data dynamically.

62

Ceph architecture

​

63

Ceph and OpenStack services

64

Ceph Networking

65

Pools, PGs, OSDs

66

Ceph components

• Monitors: A Ceph Monitor (ceph-mon) maintains maps of the cluster state, including the monitor map, manager map, the OSD map, the MDS map, and the CRUSH map. These maps are critical cluster state required for Ceph daemons to coordinate with each other. Monitors are also responsible for managing authentication between daemons and clients. At least three monitors are normally required for redundancy and high availability.
• Managers: A Ceph Manager daemon (ceph-mgr) is responsible for keeping track of runtime metrics and the current state of the Ceph cluster, including storage utilization, current performance metrics, and system load. The Ceph Manager daemons also host python-based modules to manage and expose Ceph cluster information, including a web-based Ceph Dashboard and REST API. At least two managers are normally required for high availability.
• Ceph OSDs: A Ceph OSD (object storage daemon, ceph-osd) stores data, handles data replication, recovery, rebalancing, and provides some monitoring information to Ceph Monitors and Managers by checking other Ceph OSD Daemons for a heartbeat. At least 3 Ceph OSDs are normally required for redundancy and high availability.
• MDSs: A Ceph Metadata Server (MDS, ceph-mds) stores metadata on behalf of the Ceph File System (i.e., Ceph Block Devices and Ceph Object Storage do not use MDS). Ceph Metadata Servers allow POSIX file system users to execute basic commands (like ls, find, etc.) without placing an enormous burden on the Ceph Storage Cluster.

​

67

Ceph Filestore vs Bluestore

68

Introduction to OpenStack

Part 3: Operations with Kayobe

69

69

Infrastructure-as-Code

Deployment Process

70

70

Infrastructure-as-Code

Kayobe

71

71

Kolla and Kolla-Ansible

��Kolla-Ansible

• Deploys a containerised control plane in a simple, flexible manner.
• Broad support for OpenStack services.
• Ansible keeps things (relatively) simple

Containerised control plane using Kolla

​

Kolla

• Docker containers for OpenStack
• Follows application container pattern

72

72

Kayobe

• Completes Kolla-Ansible with missing functionality:
• Bare metal deployment of control plane
• Equivalent of TripleO’s undercloud
• Configuration of control plane host OS
• Bare metal compute node management
• Infrastructure-as-Code

73

73

Development System

New technologies

Continual redeployment

The Change Pipeline

Staging System

Miniature production

Regression testing

Regular redeployment

74

Monitoring and Logging

75

Monitoring Architecture – Components – Kayobe

76

Monitoring Architecture – Prometheus

77

Node Exporter

78

Node Exporter

79

Node Exporter

80

Node Exporter

81

Alerts

82

Monitoring Architecture – Prometheus

83

84

85

OpenStack Use Cases Examples – Redfish

86

OpenStack Use Cases Examples – Node Exporter

87

OpenStack Use Cases Examples – MariaDB

88

OpenStack Use Cases Examples – Ceph

89

Monitoring Architecture – Elasticsearch Fluentd Kibana

90

Kibana OpenStack Error Logs Search Example

91

Introduction to OpenStack

Overview of the Session

92

92

What is OpenStack?

• Defines itself as a cloud operating system
• Controls compute, storage, and networking (IaaS)
• Makes them manageable and available via APIs
• Can manage compute resources as bare-metal, VMs, or containers
• Composed of many loosely coupled services

93

Ceph and OpenStack services

94

Scientific OpenStack

95

Questions?

96

Introduction to OpenStack

Part 4: Self-Paced Learning

97

97