Configuration Management

Traditional Configuration Management (CM)

Pressman: “is a set of tracking and control activities that are initiated when a software engineering projects begins and terminates when software is taken out of operation”.

• Identify all items related to software.
• Manage changes to those items.
• Enable variations of items and changes.
• Maintain quality of versions and releases.
• Provide traceability between changes and requirements.

Configuration management originates from the 1950s, when spacecraft failures resulted from undocumented changes.

Example configuration management plans

On the failure of traditional CM.

Traditional configuration management focused on preventing unregulated and undocumented change.

A worthy goal… however

In reality, a burdensome and incomplete process can be just as problematic as no process at all.

Reality: 200 page documents filled with out-of-date instructions and overly strict processes.

Toward Modern CM practices.

1. Source control already makes easy to identify software components and manage changes. Don’t need file listing in document.
2. Variations can be enabled with branches and feature flags. Don’t need manual baselines.
3. Better code review practices + CI/CD pipelines can enable quality control and traceability between requirements and code in production. You don’t need control board.

In short, a continuous deployment pipeline and associated development practices can help regulate and ensure quality changes to software better than a static word document.

Still, one more ingredient missing...

​

Recall, code alone is not enough.

We also need CM for Computing Environments!

Configuration + Code

Simple Example: When Time Breaks

Real-world occurrence: Mockito build fails:

Cause: Differences in timezone file structure, which is changed in Ubuntu 16.04, and breaks how timezones are implemented in JDK 7.

Why are bash scripts not enough?

Recall some of the basic issues you’ve faced running commands in bash scripts for HW0 and HW1...

Modern CM Components

git, branches

package managers, task and build managers

Inventory, configuration scripts.

Infrastructure update patterns

Orchestration and configuration tools

Testing and compliance

How important are dependencies?

Dependencies are common cause for build failures.

A Quantitative Study of Java Software Buildability

https://sulir.github.io/papers/Sulir16quantitative.pdf

Why dependency management is difficult for programmers?

• RQ1: Can gists be executed?

​

• RQ2: Can a naive algorithm enable executable gists?

​

• RQ3: Why might configuration of environment fail?

[Horton and Parnin 2018] ICSME’18

There are over 300k Python snippets, but limited api.

As a result, we scrapped the web UI.

10,259 public Python gists, with at least one star.

Gistable =

Gist + Executable

Can gists be executed?

Not really.

Can code be executed?

Only 25% of Python code snippets from Stack Overflow are runnable [Wang MSR’16]

We replicated result on Python Notebooks [Rule 2018 CHI].

​

Only 8% are executable.

Can the solution be easy?

Just pip install the list of packages!

Packages = [‘dashtable’];

Naive solution

Can the solution be easy?

Not quite.

ImportError: No

module named bs4.

We asked developers to create Dockerfiles for 240 failing gists.

Developers reported spending 20 minutes to 2 hours to create a Dockerfile.

For 33% of gists, developers could not create a successful Dockerfile.

We identified first root cause for failure based on successful Dockerfile

1. Names: Module-name != package-name

Package Managers

<dependencies>

<dependency>

<groupId>org.seleniumhq.selenium</groupId>

<artifactId>selenium-java</artifactId>

<version>2.25.0</version>

</dependency>�</dependencies>

Platform package managers

Install system dependencies and binaries.

Examples: choco (Windows), brew (Mac), apt-get, yum (Linux).

Library package managers

Install software dependencies.

Examples: npm, nuget, bower, maven, pip, rubygems.

Maven’s pom.xml

Tip: Use pinned versions on dependencies.

Why are Package Managers not enough?

• Often package managers themselves require transitive dependencies:� Install d3 with bower with grunt with npm with brew.
• Missing glue work:
• Configuration files
• Standing up services
• System administrative work:
• Creating users
• System resources (networking)
• Build and task managers (build lifecycle, publish tasks, etc.)...more in pipelines.
• Other reasons?

Infrastructure Inventory

What are all the assets you own?

• Servers and IP addresses.
• Service endpoints
• Roles
• SSH keys
• SSH host signatures
• Passwords
• API tokens
• ...
• Anything else?

[dbservers]

152.14.XX.XX

​

[workers]

45.55.XX.XX

45.55.XX.XX

104.236.XX.XX

45.55.XX.XX

104.131.XX.XX

45.55.XX.XX

138.197.XX.XX

104.236.XX.XX

45.55.XX.XX

104.131.XX.XX

Configuration is Code (infrastructure as code)

Any change to an asset should be checked in.

• Configuration can be just as large as source code.
• Continuous integration should ideally use configuration scripts to setup environment.
• 60,000 configuration changes at day at Netflix.

Challenges

• Fast vs. complete
• Dynamic configuration properties (running in live servers) can be difficult to manage and debug.
• Private data
• Managing and isolating access to certain servers and secret configuration.

Configuration Tools

Ansible - “Playbook” scripts run over ssh

---

# The playbook creates a new database test and populates data in the database to test the sharding.

- hosts: $servername

user: root

tasks:

- name: Create a new database and user

mongodb_user: login_user=admin login_password=${mongo_admin_pass} login_port=${mongos_port} database=test user=admin password=${mongo_admin_pass} state=present

- name: Pause for the user to get created and replicated

pause: minutes=3

- name: Execute the collection creation script

command: /usr/bin/mongo localhost:${mongos_port}/test -u admin -p ${mongo_admin_pass} /tmp/testsharding.js

- name: Enable sharding on the database and collection

command: /usr/bin/mongo localhost:${mongos_port}/admin -u admin -p ${mongo_admin_pass} /tmp/enablesharding.js

Configuration Tools

Puppet - Less hands on, agents run on server.

​

puppet module install thias-mongodb

​

mongodb::key { '/etc/mongodb.key':� content => 'c9otjehasAvlactocPiphAgC9',�}�class { '::mongodb':� bind_ip => '0.0.0.0',� auth => 'true',� rest => 'true',� replset => 'rs0',� keyfile => '/etc/mongodb.key',� verbose => 'true',�}

Configuration Tools

Chef - “Cookbooks” ruby-based task and build recipes

1. knife cookbook site download mongodb

mongodb_user { testuser:� username => 'testuser',� ensure => present,� password_hash => mongodb_password('testuser', 'p@ssw0rd'),� database => testdb,� roles => ['readWrite', 'dbAdmin'],� tries => 10,� require => Class['mongodb::server'],�}

Other useful configuration tools.

Post-configuration help with cloud-init:

​

​

​

​

​

Provisioning help with Terraform:

Provision cluster on VSphere.

Configuration Models

Idempotency

Definition 1: Applying the same operation, multiple times results in the same state.

​

Definition 2: A system is able to reach a desired state, regardless of its current state.

​

Example

Exercise + Discussion

Explain idempotency to your a partner.

​

What are some examples of idempotent and non-idempotent operations?

​

Infrastructure update patterns

Living infrastructure: Maintain same dedicated instance.

Immutable infrastructure: Deploy new read-only instance.

apply

construct

replace

​

Other Analogies

​

Pets vs. Cattle

​

Snowflake vs. Phoenix servers.

​

​

References

https://martinfowler.com/bliki/SnowflakeServer.html

https://martinfowler.com/bliki/PhoenixServer.html

http://cloudscaling.com/blog/cloud-computing/the-history-of-pets-vs-cattle/

​

Living Infrastructure

How

1. Skip provision step for each update.
2. Accumulate updates over time.

Implications

• Idempotent operations are desirable.
• Configuration drift is possible if manual updates are allowed.
• Simplicity from infrastructure remaining static.
• Resilience can be reduced.

​

​

Immutable Infrastructure

How

• Build new image.
• Provision new instance with read-only image.

Implications

• Images built offline and deployed with provisioning step.
• Updates slower due to additional provisioning and setup time.
• Immutability escape hatches: writable volumes, data subscriptions.

Patterns for Building Images

Online snapshot

• Provision new instance.
• Build filesystem (Run scripts to update services, packages, configuration)
• Save snapshot of instance as Image.

Examples: Packer

Rootfs construction

1. Build filesystem (in chroot or composing container layers).
2. Add init scripts and kernel, serialize as Image.

Examples: Linuxkit, Slim

​

Building Images

https://github.com/ottomatica/slim

​

Living infrastructure vs. immutable infrastructure

​

Activity

Explain the concepts to a partner.

Discuss the potential benefits and drawbacks of each approach.

Share insights with the class.

Environment verification and testing.

Reachability: Environment can access other internal and external resources and services.

• DNS is reachable.
• Can talk to log server/db server.

Availability: Environment provides expected service.

• API service is running on port 3000

Capability: Environment supports required operations.

• Need GPU support
• Need RAM, disk space

Identifiable: Environment contains required items.

• Node.js ^10.x.x is installed. Config files contain correct settings.

Insecure Configuration Patterns (ICPs)

Admin by default Use of Insecure HTTP

Empty password Use of weak cryptography algorithms

Hard-coded secret Unrestricted ip address

Read more: �https://akondrahman.github.io/papers/icse19_slic.pdf

In summary

Identify: all system and package dependencies using package managers. All infrastructure components with inventory and variables in configuration scripts. Secrets in vaults. All software components built with reproducible builds.

Manage: Make changes to computing environments with configuration tools.

Variations: Tag infrastructure instances, allow configuration change to be directed to subpopulation.

Quality: Linters, testing-in-production, automatic rollbacks.

Traceability: Track all infrastructure code is checked and tracked along with normal codes.

One possible configuration management plan for computing environments

CM Principles (by Michael DeHaan)

• Pinned versions: Are all dependencies version pinned (or is the repo at least snapshotted?)
• Store dependencies: Are you protected against web dependencies going away? Can you still deploy when github is down?
• No builds in prod: Make sure no builds are done on the remote system
• Use images: Use cloud like cloud where you can for more reliability - image builds where possible, some post-configuration done by cloud init, even locally executing ansible, is ok.
• Reproducible builds: Is a new build of this server going to look like one I built last week (snapshot of repos, etc) - or does this matter?
• Cache everything: Where possible, cache artifacts in your datacenter (example: yum reposync in the old days, maybe artifactory) and content get from there, don't DDOS some person's .com website with remote requests to copy a tarball, as this scales to 1000 servers doing it at once it will light things on fire or at least call failures

Some research...

Automatic inference and repair of computing environments.

Automatic analysis and transformation of computing environments.

DockerizeMe: Automatic inference of Dockerfile

Given a code snippet, can we infer associated set of dependencies?

[Horton and Parnin, ICSE 2019]

• Addresses name resolution, transitive and system dependencies.
• Builds graph database of dependencies from a wide-variety of knowledge sources.
• Resolves 892 ImportErrors naive solution cannot (31% improvement).

General approach: Reason over a large set of uncertain and incomplete offline database.

We automatically build wheels, extract all resources provided for top 10k packages.

Core Python Contributor asked:

Dynamically extract dependencies.

biopython package => Bio, BioSQL modules

Build association rules from Dockerfiles, Pipfiles, requirements.txt

Detecting when code breaks.

http://chrisparnin.me/pdf/V2-CameraReady.pdf

What about an “everything 🥯 bagel” docker image?

https://github.com/Kaggle/docker-python/blob/master/Dockerfile

​

10 GB “behemoth” image.

“everything 🥯 bagel” images can be problematic.

Just one more package, please?

Enormous security attack surface.

Maintenance nightmare.

Put docker image on a diet.

https://github.com/chrisparnin/docker-reduce

​

Can reduce Dockerfile to run given code snippet.

Still, each build of Dockerfile can be very slow! Needs specialized optimization.

Program synthesis

Automatically generate ansible task given a bash command?

Automatic mapping between commands and ansible modules

Image synthesis