SAHARA IMAGE GENERATION

IN N AND O

ELISE GAFFORD

OCTOBER 28, 2016

SESSION OVERVIEW

1. Part I - Why are we talking about this?
1. Problems with image gen
2. Description of ideal implementation
3. The Plan to fix this (as of today)
2. Part II - What is done today?
• Overview of image gen CLI documentation and codebase
• Overview of image gen CLI usage
• Demo
3. Part III - What do we have to do now?
• In which Elise describes the glorious things which will hopefully be accomplished in O

PART I: WHAT ARE YOU TALKING ABOUT

Or, In Which Elise Talks Smack About DIB

WHERE WE WERE

Sahara had 2 flows that were relevant to image manipulation:

• Pre-Nova spawn image packing
• Used sahara-image-elements repository to generate images (to store in Glance)
• Post-Nova spawn cluster generation from “clean” (OS-only) images
• Logic maintained in Sahara process within plugins
• Pre-Configuration validation of images by plugins
• Remember how I said we had 2 flows relevant to image manipulation?
• We didn’t do this at all.

WHY WHERE WE WERE WAS KINDA BAD

• Duplication of logic
• Steps required for packing images and “clean” image clusters were often identical, but had to be expressed separately (in DIB and in Python).
• Poor validation
• Plugins did not validate that images provided to them met their needs.
• Failures due to image contents were late and sometimes difficult to understand.
• Poor encapsulation
• Image generation and cluster provisioning logic for any one plugin are really one application
• Maintaining them in two places allows versionitis and dependency problems
• Having one monolithic repo for all plugins makes them less pluggable
• BONUS TEAM - INTERNAL REASON NOT APPEARING IN PUBLIC
• DIB is really quite awful

​

OUR DREAM IMPLEMENTATION

• All flows share common logic:
• Image packing
• Image validation
• Clean image cluster gen
• Image manipulation is stored and versioned within plugins
• The user can still generate images with a CLI...
• But they can also use an API to generate images in clean build environments
• ... And both dev test cycles and user retries are as quick and painless as possible

​

THE PLAN

1. Move DIB elements into the plugins
1. We considered this and decided not to do it because it was much more crazy than we are
2. Build a validation engine that ensures that images meet a specification
• YAML-based spec definition
3. Extend that engine to optionally modify images to spec
4. Build a CLI to expose this functionality
5. Create and test specifications for each plugin to support this method
6. Deprecate sahara-image-elements (only when this method proves stable)
7. Build an API to:
• Spawn a clean tenant-plane image build environment
• Download a base image from Glance and modify it to spec
• Push the new image back to Glance and register it for use by Sahara

​

PART II: WHAT EXISTS TODAY

Or, In Which We Lament that the Universe Does Not in Fact Contain More Time and then Talk About Image Gen Forever

WHERE WE ARE AS OF N

1. Build a validation engine that ensures that images meet a specification
1. YAML-based spec definition
2. Extend that engine to optionally modify images to spec
3. Build a CLI to expose this functionality
4. Create and test specifications for each plugin to support this method
5. Deprecate sahara-image-elements (only when this method proves stable)
6. Build an API to:
• Spawn a clean tenant-plane image build environment
• Download a base image from Glance and modify it to spec
• Push the new image back to Glance and register it for use by Sahara

​

WAIT WE HAVE THIS ALREADY?

• Yes, totally.
• Specs:
• https://github.com/openstack/sahara-specs/blob/master/specs/mitaka/validate-image-spi.rst
• https://github.com/openstack/sahara-specs/blob/master/specs/newton/image-generation-cli.rst
• Doc:
• https://github.com/openstack/sahara/blob/master/doc/source/devref/image-gen.rst
• Code:
• Image gen SPI methods: https://github.com/openstack/sahara/blob/master/sahara/plugins/provisioning.py
• Core image gen module: https://github.com/openstack/sahara/blob/master/sahara/plugins/images.py
• Image gen CLI:
• Entry point: https://github.com/openstack/sahara/blob/master/sahara/cli/image_pack/cli.py
• Image packing API: https://github.com/openstack/sahara/blob/master/sahara/cli/image_pack/api.py
• There are unit tests too which are in predictable test places.

ELISE YOU WRITE TOO MUCH AND IT’S FRIDAY AFTERNOON AT SUMMIT JUST TELL US ABOUT IT

​

• Yup, that’s totally what we’re here to do.
• First we’ll see an image generation specification example.
• We’ll talk about the current state in 3 parts:
• The new plugin SPI methods
• The core image generation module that they use
• The CLI that leverages them
• Then we’ll do a demo.

YAMLS ARE OUR FRIENDS

• There’s part of a spec over there!
• Arguments at the top
• Specs are configurable
• But all the variables are explicit
• This lets the CLI (and API later) create real help text (and UI)
• Validators are next
• We’ll talk about validators soon

A NEW PLUGIN SPI METHOD

def get_image_arguments(self, hadoop_version):

Gets the argument set taken by the plugin's image generator.

Note: If the plugin can generate or validate an image but takes no arguments, please return an empty sequence rather than NotImplemented for all versions that support image generation or validation. This is used as a flag to determine whether the plugin has implemented this optional feature.

:returns: A sequence with items of type sahara.plugins.images.ImageArgument

​

class ImageArgument(object):

An argument used by an image specification.

def __init__(self, name, description=None, default=None, required=False, choices=None):

• “Name” is all of the argument name for the CLI, the env var name in scripts, and the reference name in specs
• “Choices” is a sequence of options (if given, the provided value must be in the sequence)
• There are two “magic” arguments that the images module automatically provides, “distro” and “reconcile”
• Distro is the distro of the remote machine, reconcile is whether the validators should modify or only test

​

​

ANOTHER NEW PLUGIN SPI METHOD

def pack_image(self, hadoop_version, remote, reconcile=True, image_arguments=None):

Packs an image for registration in Glance and use by Sahara

:param remote: A remote (usually of type sahara.cli.image_pack.api.ImageRemote) that serves as a handle to the image to modify. Note that this image will be modified in-place, not copied.

:param reconcile: If set to False, this method will only test to ensure that the image already meets the plugin's requirements. This can be used to test images without modification. If set to True per the default, this method will modify the image if any requirements are not met.

:param image_arguments: A dict of image argument name to argument value.

• So this is the method that actually packs images from the CLI
• We’ll talk about that in a bit.
• But that remote up there is going to be a libguestfs handle.

​

​

LAST NEW PLUGIN SPI METHOD YOU GUYS

def validate_images(self, cluster, reconcile=True, image_arguments=None):

Validates the image to be used by a cluster.

:param cluster: The object handle to a cluster which has active instances ready to generate remote handles.

:param reconcile: If set to False, this method will only test to ensure that the image already meets the plugin's requirements. This can be used to test images without modification. If set to True per the default, this method will modify the image if any requirements are not met.

:param image_arguments: A dict of image argument name to argument value.

• So this is the method that the server side uses to:
• Validate the image used to create instances if reconcile == False
• Modify instances to the specification if reconcile == True
• This takes a cluster rather than just a remote so that the plugins can decide:
• How many instances they want to test or modify
• Make smart choices about multithreading, etc.

​

​

GREAT THAT’S THE SPI HOW DOES IT WORK

• Plugins will probably want to call into sahara.plugins.images.
• They’ll probably want to build a set of validators from a yaml
• They’ll want to do this by calling SaharaImageValidator.from_yaml

def from_yaml(cls, yaml_path, validator_map=None, resource_roots=None):

"""Constructs and returns a validator from the provided yaml file.

:param yaml_path: The relative path to a yaml file.

:param validator_map: A map of validator name to class.

:param resource_roots: The roots from which relative paths to resources (scripts and such) will be referenced. Any resource will be pulled from the first path in the list at which a file exists.

:return A SaharaImageValidator built to the yaml specification.

• Validator_map allows you to pass in custom validators that don’t fit in the core if you want fanciness
• Extensibility is our friend
• This calls into the from_spec classmethod of subclasses of SaharaImageValidatorBase (whichever class maps to the validator type specified in the yaml, like “package” or “script”

WAIT WHAT’S A SAHARAIMAGEVALIDATORBASE

• It’s an abstraction that can build itself from a spec and validate or modify a remote machine.
• The action types we already have are:
• package: installs packages on the machine.
• script: runs a script on the machine
• The action types we probably should have:
• spec: runs the validators contained in another spec (for encapsulation of shared stuff)
• file and directory
• user and group
• The logical flow operators we have are:
• all: Runs all of the list of validators it contains
• any: makes sure that at least one of the list of validators it contains passes
• os_case: switches on the distro or family of the machine
• argument_case: switches on the value of an ImageArgument
• argument_set: sets the value of an argument to a new value

​

GREAT MY PLUGIN HAS A VALIDATOR WHAT NOW

• Call this method on your validator object (or objects if you hate the all validator for reasons)

def validate(self, remote, reconcile=True, **kwargs):

Validates the image.

:param remote: A remote socket to the instance.

:param reconcile: If false, all validators will only verify that a desired state is present, and fail if it is not. If true, all validators will attempt to enforce the desired state if possible, and succeed if this enforcement succeeds.

:raises ImageValidationError: If validation fails.

• This is where the magic happens.
• Notes on scripts:
• If you write a script and use it via the script validator, it should obey the $reconcile arg
• If you write a script you should also really try to make it idempotent
• (“Idempotent” means that if you run it again after it succeeds it will leave the same state)
• Notes on packages: nah, these are super easy

​

​

I’M EXCITED AND I WANT TO WRITE A NEW VALIDATOR RIGHT NOW

• That’s great!
• I won’t cover this in huge detail, but:
• You’ll need to write a from_spec classmethod
• And use jsonschema to validate the input (see existing classes)
• The constructor should know nothing about yamls or specs
• Then write a validate method
• And make it idempotent
• Or I will -1 you so hard
• It’s not so bad and there are several examples already in the images module

WE’RE JUST HERE TO SEE THE NEW CLI

Command structure:

sahara-image-pack --image ./image.qcow2 [--test-only] PLUGIN VERSION [plugin arguments]

Features:

• Auto-generates help text from arguments
• Idempotent and modifies images in-place
• Very fast test cycles and retries
• Allows freeform bash scripts and more structured resources
• Though it’s on you to make your scripts idempotent
• Test-only mode to validate without change (via --test-only argument)

Note: In dev, use the “images” tox env and install python-libguestfs and libguestfs-tools on your machine: python-libguestfs isn’t pip installable

DEMO TIME

• I’m gonna demo some stuff now.
• First I’m gonna show you a spec and a resource script
• Then I’m gonna show you a plugin impl in Ambari
• Here are the commands I’m gonna run:
• . ./sahara/.tox/images/bin/activate # to use the right venv
• cp CentOS-7-x86_64-GenericCloud.qcow2 Ambari.qcow2 # CLI modifies image in place
• sahara-image-pack -h # so we can see the help text
• sahara-image-pack --image ./Ambari.qcow2 ambari -h # help text for plugin w/o version
• sahara-image-pack --image ./Ambari.qcow2 ambari 2.4 -h # help text for plugin with version
• sahara-image-pack --image ./Ambari.qcow2 ambari 2.4 # run the thing!
• # Modify the yaml spec
• sahara-image-pack --image ./Ambari.qcow2 ambari 2.4 # run the thing again with changes!

PART III: WHAT’S NEXT

Or, haaalp plz kthxbye

HEY THAT EARLIER SLIDE SEEMS RELEVANT

• Build a validation engine that ensures that images meet a specification
• YAML-based spec definition
• Extend that engine to optionally modify images to spec
• Build a CLI to expose this functionality
• Create and test specifications for each plugin to support this method
• Deprecate sahara-image-elements (only when this method proves stable)
• Build an API to:
• Spawn a clean tenant-plane image build environment
• Download a base image from Glance and modify it to spec
• Push the new image back to Glance and register it for use by Sahara

​

WHAT I’D LOVE

• I personally work much better and faster collaborating actively with others
• I’d love to take our plugins and assign each to an engineer
• We can work on them in parallel
• I’m happy to help folks with problems or write new code as needed
• Having other folks use this early will find usability issues
• Once we write the specs and validate the images, we can put tests in CI
• Likely triggered on the resources/images dir in each plugin
• Which will totally exist when we’re done
• Then we move on
• I don’t believe we should target more than this for O
• The API can wait

ANY QUESTIONS?

Or volunteers?

THANK YOU

Let’s Do This Thing

- ELISE GAFFORD

​