1 of 23

Saltcheck

Easy Salt State Testing

William Cannon

​

| SaltConf17

1

2 of 23

SALTCHECK

DEMOS

EXTERNAL TOOLS

QUESTIONS

PRESENTATION OVERVIEW

ABOUT ME

LAST YEAR’S SALTCONF

WELCOME

SALT TOOLS

| SaltConf17

2

3 of 23

ABOUT ME

Infrastructure Architect @ hospitalityPulse

SaltStack Certified Engineer (#23) SSC ID: AE07A5E3

Host SaltStack Meetup in Minneapolis

SaltConf Presenter 2015, 2016

Use Salt for infrastructure automation, deployment of custom applications, business continuity and etc.

@wcannon - twitter / github

https://github.com/wcannon/salt-check

​

​

| SaltConf17

3

4 of 23

By automating room inventory management in real time,

hospitalityPulse enables hoteliers to maximize guest satisfaction:

…by maximizing attribute fulfillment levels!

…minimizing preventable downgrades!

…streamlining the check in process!

and in the process allow hoteliers to:

…present guests with attribute inventory during the booking process!

…manage their operation based on insightful, real-time data!

…gain new understanding of who their guests really are!

5 of 23

The big issue

At hospitalityPulse we use salt heavily

​

In the early years simple tests were sufficient to build confidence.

​

However, with time passing and complexity increasing we needed a way to catch issues early, and easily.

​

We were left with the question:

What is the best way to test salt states?

| SaltConf17

5

6 of 23

EXTERNAL TOOLS

Predict actions that will happen before a state run

SALT TOOLS

At SaltConf16… a search for the right way to test

Mostly ruby based tools requiring installation, libraries, configuration… lots of work

| SaltConf17

6

7 of 23

Salt Tools

Shows top data a minion will use for a highstate

Show Top

state.show_top

Show what would be applied to a minion

Test = True

state.apply my_state test=True

Shows highstate data from master

Show High State

state.show_highstate

Shows low data that will be applied to a minion

Show Low State

state.show_lowstate

Display state data from a specific sls (or list)

Show SLS

state.show_sls

State compiler processes sls data without calling state functions (good for validating arguments)

Mock = True

state.apply my_state mock=True

| SaltConf17

7

8 of 23

External Tools

RSpecPuppet

unit test

ChefSpec

unit test

RSpec

unit test

ServerSpec

integration testing

Test Kitchen

test harness

TestInfra

ServerSpec in python, plugin to pytest

| SaltConf17

8

9 of 23

EASIEST PROCESS

But not a great one

APPLY A STATE

INSPECT SERVER

Bonus points for spinning up a new VM / container to do this

WRITE / UPDATE A STATE

Reviewing output of state run for any unexpected issues

Manually review all the parts of the system that the state is managing

Hint: If you are testing this way, create an sls that removes all the changes. And, apply it in-between state runs

| SaltConf17

9

10 of 23

Let’s review why we need to test

  • A salt state is a translation of an idea

​

  • Translations are prone to errors, confusion

​

  • States start simple, and grow complex over time
    • add in jinja templating
    • add in pillar lookups
    • start using map.jinja to support different operating systems, or versions
    • use “include” statements to “simplify” your states

​

  • Need proof that states get complex over time?
    • Look up the salt formula for managing users - yikes!

| SaltConf17

10

11 of 23

What is the ideal situation?

  • Write a state and a test for that state, apply the state and run the test

​

  • Should only require the same knowledge as salt (yaml / jinja is enough)
    • Should not require learning a programming language or new DSL

​

  • Should fit into the salt platform cleanly - targeting, masterless, ssh, outputters

​

  • Should favor convention over configuration aka minimal setup

​

  • Test output will be clear, useful, and meaningful

​

  • Should scale - let’s test all the minions at once!

​

  • Ideally... should be able to test a highstate the same way!

​

​

| SaltConf17

11

12 of 23

Saltcheck! - what a nice module

​

  • Easy to write - yaml, and optional jinja
    • secretly supports all salt rendering

​

  • Implemented as execution module to take advantage of salt functionality
    • targeting
    • masterless
    • ssh transport
    • outputters
    • returners

​

  • A state is easily testable with one test file and zero configuration
  • A highstate test runs all tests for each state making up the highstate
  • Uses any salt execution module (469+) for testing
  • Pull request accepted - development branch of salt open

| SaltConf17

12

13 of 23

Saltcheck Requirements and Conventions

  • A state is set up as a directory (not an sls file)

​

  • Tests are organized as a specific subdirectory (“saltcheck-tests”) of the state

​

  • Test files end with “.tst”
    • all test files in the “saltcheck-tests” directory will be used by saltcheck

​

  • A test file consists of yaml with the bare minimum to call a salt module and function

  • A test file contains 1+ yaml blocks (1 block = 1 test)

​

  • Each test block also contains an assertion about the return value

​

​

| SaltConf17

13

14 of 23

saltcheck in state tree

| SaltConf17

14

15 of 23

CLI translation to saltcheck test

CLI Execution to translate (cli)

salt ‘*’ file.file_exists /tmp/hello

​

Translation to valid saltcheck test (yaml)

​

​

Test-1-hello-file:

module_and_function: file.file_exists

args:

- /tmp/hello

kwargs:

assertion: assertEqual

expected-return: True

​

| SaltConf17

15

16 of 23

Test Assertions Supported

assertEqual

assertNotEqual

assertTrue

assertFalse

assertIn

assertNotIn

assertGreater

assertGreaterEqual

assertLess

assertLessEqual

​

| SaltConf17

16

17 of 23

How to run saltcheck

​

salt ‘*’ saltcheck.run_state_tests MyState

​

salt ‘*’ saltcheck.run_highstate_tests

​

salt ‘*’ saltcheck.update_master_cache

​

Config flag - set on minion, or assigned via pillar

auto_update_master_cache: True

​

| SaltConf17

17

18 of 23

EASY AND BETTER PROCESS

APPLY A STATE

Bonus points for spinning up a new VM / container to do this

WRITE / UPDATE A STATE

Reviewing output of state run for any unexpected issues

Use salt execution modules for functionality

WRITE / UPDATE SALTCHECK TEST

RUN SALTCHECK

Review test output and update state accordingly

| SaltConf17

18

19 of 23

BASIC SALTCHECK TEST

| SaltConf17

19

20 of 23

SALTCHECK TEST WITH JINJA

| SaltConf17

20

21 of 23

SALTCHECK HIGHSTATE TEST

| SaltConf17

21

22 of 23

Compliance / Auditing

Trouble-Shooting

Catch issues early

Highstate Addition

Dynamic Testing

Use Cases

Other than simple testing

Document systems meeting company policies

Check status of production servers (in parallel)

Pull in testing data from central sources (eg. pillar) at run time

| SaltConf17

22

23 of 23

Thanks for coming!

Questions?

| SaltConf17

23