Using Puppet to Pull Strings:

a Gentle Introduction to Puppet

Thomas Uphill

http://ramblings.narrabilis.com

Latest Version

on Google Docs

Me

Ben

Overview

What is Puppet and why do I want it?

Puppet 101

Introductory Material

Puppet 102

Advanced Material

Questions sporadically throughout.

What is puppet?

Think of it as a language.

Describe state, not steps.

Paint a picture of your ideal and most clean system.

Puppet does the rest!

puppetlabs

  • Luke Kanies in 2005
  • http://puppetlabs.com/
  • puppet Enterprise
    • commercial support
    • windows support
    • puppet forge

http://en.wikipedia.org/wiki/Puppet_(software)

http://forge.puppetlabs.com/

Why do I want it?

you can do cool things with it

Human-parsable.

ensure that all your hosts are running ssh

service {'ssh': ensure => running}

Puppet 101

Resource Abstraction Layer

Syntax

trifecta

core types
(file, package, service)

puppet apply / puppet resource

facter/variables

conditionals

booleans/arithmetic

in

templates

types II
(augeas, exec, cron)

classes

functions

filebuckets

versions

0.20

0.25

2.6.0

2.7.0

2.7.21

0.20

0.25

2.6.0

2.7.0

2.7.21

3.0

3.0

3.1.1

3.1.1

http://projects.puppetlabs.com/projects/1/wiki/Release_Notes

this talk will be based on 2.7.14

most if not all of 101 should apply to lower versions

parts of 102 will not work in older releases.

learning puppet vm

http://docs.puppetlabs.com/learning/

VMware

OVF

VirtualBox

KVM

Get the Learning Puppet VM

those of you with laptops, you can download the learning puppet vm, it should help get you started

it includes puppet enterprise, so you can evaluate if that is the right solution for you.

We'll operate as though you don't have enterprise though.

learning puppet vm

KVM

[root@hv] vmware-mount -f learn_puppet.vmdk /mnt

[root@hv] qemu-img /mnt/flat -Oqcow2 learn_puppet.qcow2

to use the vmdk file in kvm you might have to convert it, I did.

I used vmware-mount which is included in the free vmplayer from vmware.

i converted to qcow2.

we'll use the learn vm in the demo's as we go along.

but now back to the language introduction.

Resource Abstraction Layer

types

providers

ubuntu

SLES

macosx

Windows

user

file

package

service

exec

mount

group

host

RHEL

CENTOS

Springdale

Puppetʼs resource abstraction layer (RAL). The RAL splits resources into

types (high-level models) and providers (platform-specific implementations), and lets you describe

resources in a way that can apply to any system.

Language (preview)

manifest

plugin

module

class

attribute

type

node

fact

variable

types have attributes

classes have types

classes have variables

nodes have variables, types and classes.

Syntax

type

{

"title":

attribute

=>

value

,

attribute

=>

value

}

syntax.pp

file {"testfile":

mode => 0644,

owner => root

path => "/etc/test",

ensure => present

}

[root@learn ~] # puppet parser validate syntax.pp

err: Could not parse for environment production: Syntax error at 'path'; expected '}' at /etc/puppetlabs/puppet/manifests/syntax.pp:4

err: Try 'puppet help parser validate' for usage

type identifier

brace

title with a colon

attirbute value pairs with arrows

remember comma's between attribute pairs

puppet parser useful, lets look at an example where we forgot to add the comma

run our syntax.pp manifest through puppet parse and see what happens.

trifecta

file

package

service

with these three things, you can do almost anything

they are

package

file

service

you can rule the world with these three

these are important enough that we will cover them in detail.

core types cheat sheet

docs.puppetlabs.com/puppet_core_types_cheatsheet.pdf

puppetlabs has some fantastic documentation

this sheet is very helpful, the front side has the trifecta on it.

file

Attributes

ensure

  • present
  • absent
  • file
  • directory
  • link

path

  • full path to the file
  • default is $title

title:

source

  • url to the file
  • local path

content

  • string contents of the file
  • template

target

  • symlink
    target => '/path/of/link/target'

recurse

  • recursively create
    (with ensure=>directory)

purge

  • remove non puppet files from directory
    recurse => true

owner

  • name or UID

group

  • name or GID

mode

  • octal mode

each type has attributes that can be set

the first for many is ensure.

for file, ensure can have the following values.

present checks the file exists and reports an error if it doesn't

absent removes the file from the system

file makes sure the file is a plain file

directory makes sure the file is a directory, creating it if necessary

link makes the file a symbolic link to whatever is defined in the target (later)

distributing file contents

file

Attributes

ensure

path

title:

source

content

target

recurse

purge

owner

group

mode

content

file {"issue":

path => "/etc/motd",

content => "Hello World!"

}

/etc/motd

Hello World!

this is our hello world with puppet.

when you run this manifest, it will create a hellow world file in /etc/motd if it doesn't exist. That is the point of puppet, it only makes changes when it has to

think state not action.

distributing file contents 2

file

Attributes

ensure

path

title:

source

content

target

recurse

purge

owner

group

mode

content

file {"issue":

path => "/etc/motd",

content => file("/etc/hello")

}

/etc/motd

Hello File!

/etc/hello

Hello File!

copy the contents of the local file /etc/hello into /etc/motd

Not terribly useful IMHO but does allow for a copy function.

distributing file contents 3

file

Attributes

ensure

path

title:

source

content

target

recurse

purge

owner

group

mode

content

file {"issue":

path => "/etc/motd",

content => template("/etc/hello.erb")

}

/etc/motd

Hello learn!

/etc/hello.erb

Hello <%= hostname %>!

apply the template hello.erb and put the contents in /etc/motd.

we'll cover erb templates in a few slides.

puppet apply (serverless operation)

apply changes using a locally defined manifest

[root@learn ~] # puppet apply mymanifest.pp

this is how you do it post 0.25...before that, just do puppet manifest.pp

file contents demo

hello.pp

hello2.pp

hello3.pp

login to learn as root (password is puppet)

cd /etc/puppetlabs/puppet/manifests

puppet apply hello.pp

puppet apply hello2.pp

puppet apply hello3.pp

distributing file contents 4

file

Attributes

ensure

path

title:

content

target

recurse

purge

owner

group

mode

source

source

file {"issue":

path => "/etc/motd",

source =>"puppet:///files/hello"

}

/etc/motd

Hello puppet!

/etc/puppet/manifests/hello

Hello puppet!

X

X

instead of specifying content, we'll specify source now.

source starts with puppet:///

this uses puppet's internal fileserver.

But this won't work until you have a puppet server running.

so we'll demo this later after we have the puppet master running.

but for now, lets just say that this works and in the end it will be the most common way you'll add a file.

package

Attributes

ensure

name

title:

source

  • present
  • lastest
  • [version]
  • absent
  • purged
  • defaults to title
  • location of the package

package installs or removes packages from the system.

demo VMs

henson.local

learn

jim

kermit

piggy

package demo

sar.pp

sar2.pp

sar3.pp

login to learn

cd /etc/puppetlabs/puppet/manifests

puppet apply sar.pp

yum --nogpgcheck downgrade localinstall /root/sysstat-7.0.2-3.el5.i386.rpm

service

Attributes

ensure

name

title:

enable

  • running
  • stopped
  • true
  • false
  • defaults to title
  • true
  • false

hasrestart

  • true
  • false

hasstatus

  • true
  • false

restart

status

Command-line true/false test

Command-line true/false test

service starts or stops a service on the agent

service demo

trifecta

ordering

file

package

service

apache

/etc/puppet/picc/service-ordering.pp

service {"httpd":

enable => true,

ensure => running,

hasrestart => true,

hasstatus => true,

}

[root@kermit lopsa]# puppet apply service-ordering.pp

err: /Stage[main]//Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/sbin/service httpd start' returned 1: at /etc/puppet/picc/service-ordering.pp:6

notice: Finished catalog run in 2.86 seconds

ordering is important

You have to have the package to have the directory to make the file, you have to have the file to have the service configured properly.

trifecta

file

package

service

host.conf

/etc/puppet/picc/ordering-file.pp

file {"host.conf":

path => "/etc/httpd/conf.d/$name",

mode => 0644,

owner => 'apache',

group => 'apache',

content => "<VirtualHost *:80>

ServerName kermit.henson.local

DocumentRoot /var/www/html/kermit

</Virtualhost>"

}

[root@kermit lopsa]# puppet apply ordering-file.pp

err: /File[host.conf]/ensure: change from absent to file failed: Could not set 'file on ensure: No such file or directory - /etc/httpd/conf.d/main.puppettmp_5866 at /etc/puppet/picc/ordering-file.pp:10

notice: Finished catalog run in 2.86 seconds

!

ordering is important

You have to have the package to have the directory to make the file, you have to have the file to have the service configured properly.

trifecta

file

package

service

host.conf

httpd

httpd

ordering is important

You have to have the package to have the directory to make the file, you have to have the file to have the service configured properly.

directed acyclic graph

DAG

directed acyclic graph

directed acyclic graph

host.conf

httpd

httpd

puppet redacts the resources to form a DAG.

The key is that it is a dag, vertices that have direction cannot have cycles.

cycles are when something depends upon the thing upon which it depends.

if that sounds terrible it is, don't do it. puppet will yell at you if you do this.

Referencing objects - case

Defining a new object uses lower case:

Referencing an existing object uses upper case:

Be careful not to define objects twice!

service["sshd"] { ensure => running }

require => Service["sshd"]

ordering

Every object has builtins to allow for logic involving other objects.

host.conf

httpd

httpd

  • require
  • before
  • notify
  • subscribe
  • tag

/etc/puppet/lopsa/ordering-file2.pp

package {'httpd':

ensure => present

}

file {"host.conf":

path => "/etc/httpd/conf.d/$name",

mode => 0644,

owner => 'apache',

group => 'apache',

content => "<VirtualHost *:80>

ServerName kermit.henson.local

DocumentRoot /var/www/html/kermit

</Virtualhost>",

require => Package['httpd']

}

[root@kermit lopsa]# puppet apply ordering-file2.pp

notice: /Stage[main]//Package[httpd]/ensure: created

notice: /File[host.conf]/ensure: defined content as '{md5}83c8b17d2d2ff241cc23b6dbc3a0d331'

notice: Finished catalog run in 130.29 seconds

/etc/puppet/lopsa/ordering.pp

package {'httpd':

ensure => present

}

service {"httpd":

enable => true,

ensure => running,

hasrestart => true,

hasstatus => true,

require => [Package['httpd'],File['host.conf']]

}

file {"host.conf":

path => "/etc/httpd/conf.d/$name",

mode => 0644,

owner => 'apache',

group => 'apache',

content => "<VirtualHost *:80>

ServerName kermit.henson.local

DocumentRoot /var/www/html/kermit

</Virtualhost>",

require => Package['httpd']

}

[root@kermit lopsa]# puppet apply ordering.pp

notice: /Stage[main]//Service[httpd]/ensure: ensure changed 'stopped' to 'running'

notice: Finished catalog run in 17.35 seconds

builtins:

puppet resource

[root@learn: ~] $ puppet resource --types

augeas

computer

cron

exec

file

filebucket

group

...

[root@learn: ~] $ puppet resource mount swap

mount { 'swap':

ensure => 'unmounted',

device => '/dev/vg00/swapvol',

dump => '0',

fstype => 'swap',

options => 'defaults',

pass => '0',

target => '/etc/fstab',

}

list available types

translate the current system state into a manifest.

You can use puppet resource to edit the state with --edit

man page is puppet-resource

types available on learn vm:

anchor

augeas

computer

cron

exec

file

file_line

filebucket

group

host

interface

k5login

macauthorization

mailalias

maillist

mcx

mount

nagios_command

nagios_contact

nagios_contactgroup

nagios_host

nagios_hostdependency

nagios_hostescalation

nagios_hostextinfo

nagios_hostgroup

nagios_service

nagios_servicedependency

nagios_serviceescalation

nagios_serviceextinfo

nagios_servicegroup

nagios_timeperiod

notify

package

resources

router

schedule

scheduled_task

selboolean

selmodule

service

ssh_authorized_key

sshkey

stage

tidy

user

vlan

whit

yumrepo

zfs

zone

zpool

Facter/Facts

memory

size

processor

ip

address

architecture

fqdn

is_virtual

selinux

time

zone

system

61 facts

facter is an accessory project, it's written in ruby, facter on the command line is a ruby script

there are many resources that are mapped to facts. and we'll see later, we can make our own custom facts as well.

Facter/Facts

memory

processors

ip

address

architecture

dns

VM?

selinux

time

zone

system

$processor0

$ipaddress

$architecture

$fqdn

$is_virtual

$memorysize

$selinux

$timezone

facts are available as variables.

Facter/Facts - demo

run facter on learn | less

variables

use as parameters

use in conditionals

facts

user defined

$processor0

$myvar = "My Variable"

$::processor0

Scope!

variables can be facts => click to show fact

user defined => click to show code.

well there is a problem with toplevel variables,click to show scope bubble. in version 2.7 and above you need to worry about scope...which is a gotcha we'll come back to later...

conditionals

  • if/else/unless
  • case
  • selector
  • booleans

we'll talk about conditionals, there are if/else/unless statements, case statements and selector statements.

variables allow us to do conditional operations. before we talk about conditionals though, lets see what boolean arithmetic we can do with puppet.

booleans

$x = 1

$y = 2

($x == $y)

($x > $y)

($x < $y)

($x != $y)

($x < $y) and !($x < $y)

false

false

true

true

false

arithmetic

$x=1

$y=2

$x+$x == $y

$x-$x

$y/2

$y >> 1

$y << 1

true

1

1

4

0

if/else/unless

if $myvar == "My Variable" {

file {'/my/variable':

content => "All is good",
ensure => present
}
} elsif $myvar {

file {'/my/variable':

content => $myvar,
ensure => present
}
} else {

file {'/my/variable':

content => "bad var",
ensure => present
}
}

$variablefile = "/my/variable"

if $myvar == "My Variable" {

file {$variablefile:

content => "All is good",
ensure => present
}
} elsif $myvar {

file {$variablefile:

content => $myvar,
ensure => present
}
} else {

file {$variablefile:

content => "bad var",
ensure => present
}
}

this is a fully populated if statement, we have if, elsif and a final else.

we could have used a variable for the title/path of the file, though

we'll revisit how we could clean this up a little later as well

if/else/unless

unless

unless $::operatingsystem == 'RedHat' {
file {'/etc/redhat-release':

content => 'This isn't RedHat'

}
}

if !(condition)

if !($myvar) {

file {"/my/file":

content => "no myvar set",

ensure => present

}

}

this is a fully populated if statement, we have if, elsif and a final else.

we could have used a variable for the title/path of the file, though

we'll revisit how we could clean this up a little later as well

case

case $myvar {

"My Variable": { $content = "All is good" }

"": { $content = "bad var" }

default: { $content = $myvar }

}

file {"/my/variable":

content => $content,

ensure => present

}

case $::hostname {

/^ldap/: { include ldapserver }

/^www/: { include webserver }

/^dns/: { include bind }

/^mx[12]/: { include mxprimary }

/^mx[3-9]/: { include mxsecondary }

default: { include base }

}

Regular Expressions

  • match =~

if $::hostname =~ /^ldap/ {

include ldapserver

}

  • not match !~

if $::hostname !~ /test/ {

include production

}

this is one way we could do the previous example, it doesn't really show the power of case in puppet though, so lets try another more useful example. click

we'll mention something ahead of time here, include, we'll get to that in a few slides.

this is fairly powerful syntax, we are including.

case

case $::hostname {

/^ldap/: { include ldapserver }

/^www/: { include webserver }

/^dns/: { include bind }

/^mx[12]/: { include mxprimary }

/^mx[3-9]/: { include mxsecondary }

default: { include base }

}

Regular Expressions

  • capture
    /^ldap(\d+)/ { include "ldap$1" }

this is one way we could do the previous example, it doesn't really show the power of case in puppet though, so lets try another more useful example. click

we'll mention something ahead of time here, include, we'll get to that in a few slides.

this is fairly powerful syntax, we are including.

selector

ternary operator (C)

x == y ? "they are equal" : "they are different"

attribute => $fact ? {

'value' => 'result',

'other_value' => 'other_result',

default => 'default_result'

}

Real World

apache config

file {'httpd.conf':

path => $operatingsystem ? {

'Ubuntu' => "/etc/apache2/$name",

'RedHat' => "/etc/httpd/conf/$name",

default => "/usr/local/etc/httpd/conf/$name"

}

content => "DocumentRoot /var/www/html"

}

Real World

mysql plugin

file {'mysql_plugin.so':

path => $architecture ? {

/i\d86/ => "/usr/lib/mysql/plugin/$name",

'x86_64' => "/usr/lib64/mysql/plugin/$name",

}

source => "puppet:///mysql_plugin.so"

}

selector is similar to the ternary operator in C (and others)

if is a powerful tool in puppet

you can do a lot of nice compact things with it.

"in"

if $::hostname in ['www','web'] {

service {'httpd':

ensure => true

}

}

if $::kernelversion in ['2.6.35-22','2.6.38.6-26.rc1.fc15'] {

service {'sshd':

ensure => false

}

}

backus naur (bnf)

<exp> ::= <exp> <arithop> <exp>
| <exp> <boolop> <exp>
| <exp> <compop> <exp>
| <exp> <matchop> <regex>
| ! <exp>
| - <exp>
| "(" <exp> ")"
| <rightvalue>

<arithop> ::= "+" | "-" | "/" | "*" | "<<" | ">>"
<boolop> ::= "and" | "or"
<compop> ::= "==" | "!=" | ">" | ">=" | "<=" | "<"
<matchop> ::= "=~" | "!~"

<rightvalue> ::= <variable> | <function-call> | <literals>
<literals> ::= <float> | <integer> | <hex-integer> | \

<octal-integer> | <quoted-string>

<regex> ::= '/regex/'

http://docs.puppetlabs.com/guides/language_guide.html

templates

ERB syntax

<% Ruby code %>

<%= Ruby expression %>

<%# comment %>

-%> no newline

<%% replace with <%

%%> replace with %>

ERB embedded ruby

http://ruby-doc.org/stdlib-1.9.3/libdoc/erb/rdoc/ERB.html

templates (conditionals)

puppet ERB

<%= @fact %> replace with fact/global variable

conditional

client.conf.erb

<% if @ipaddress_eth0 != "NONE" %>

ServerName <%= @printserver %>

<% end %>

http://docs.puppetlabs.com/guides/templating.html

conditional

Example is cups client.conf

Only configure client.conf if the machine is plugged into a wired network.

templates (iteration)

iteration

$nameservers = [ 'ns1.example.com',

'ns2.example.com',

'ns3.example.com']

$searchdomains = [ 'inside.example.com',

'outside.example.com',

'under.example.com']

file {"resolvconf":

path => "/etc/resolv.conf",

mode => 0644, owner => root, group => root,

content => template('resolvconf.erb')

}

iteration

$nameservers = [ 'ns1.example.com',

'ns2.example.com',

'ns3.example.com']

$searchdomains = [ 'inside.example.com',

'outside.example.com',

'under.example.com']

file {"resolvconf":

path => "/etc/resolv.conf",

mode => 0644, owner => root, group => root,

content => template('resolvconf.erb')

}

/etc/puppet/manifests/resolv.conf.erb

# resolv.conf build by Puppet

domain <%= @domain %>

search <% searchdomains.each do |domain| -%>

<%= domain -%><% end -%> <%= @domain %>

<% nameservers.each do |server| -%>

nameserver <%= server %>

<% end -%>

/etc/resolv.conf

# resolv.conf build by Puppet

domain example.com

search inside.example.com outside.com under.example.com example.com

nameserver ns1.example.com

nameserver ns2.example.com

nameserver ns3.example.com

/etc/puppet/manifests/resolv.conf.erb

# resolv.conf build by Puppet

domain <%= @domain %>

search <% searchdomains.each do |domain| -%>

<%= domain -%><% end -%> <%= @domain %>

<% nameservers.each do |server| -%>

nameserver <%= server %>

<% end -%>

/etc/resolv.conf

# resolv.conf build by Puppet

domain example.com

search inside.example.com outside.com under.example.com example.com

nameserver ns1.example.com

nameserver ns2.example.com

nameserver ns3.example.com

templates (concatenation)

iteration

$nameservers = [ 'ns1.example.com',

'ns2.example.com',

'ns3.example.com']

$searchdomains = [ 'inside.example.com',

'outside.example.com',

'under.example.com']

file {"resolvconf":

path => "/etc/resolv.conf",

mode => 0644, owner => root, group => root,

content => template('header.erb','resolvconf.erb')

}

/etc/puppet/manifests/header.erb

# This file is autogenerated by puppet

# updated <%= @_timestamp %>

# environment <%= @environment %>

/etc/resolv.conf

# This file is autogenerated by puppet

# updated Thu Apr 26 14:13:24 -0400 2012

# environment production

domain example.com

search inside.example.com outside.com under.example.com example.com

nameserver ns1.example.com

nameserver ns2.example.com

nameserver ns3.example.com

concatenation

This is a lame example though, the timestamp causes the file to be updated on every run, we'll need a better solution to that.

types II

cron

exec

augeas

List types available:

[root@learn ~] # puppet resource --types

augeas

computer

cron

exec

file

filebucket

group

...

cron

Attributes

command

hour

name:

minute

Command to run

Hour or numeric regex

Minute or numeric regex

month

Month or numeric regex

monthday

Day of the month or numeric regex

user

weekday

User that will run this command.

Day of the week or numeric regex

cron.pp

cron { 'yum_cleaner':

command => "/usr/bin/yum clean all",

user => root,

hour => 2,

minute => 0

}

exec

Attributes

command

creates

title:

cwd

  • command to execute
  • file this creates
    only run if file does not exist
  • directory to run command from
  • false

environment

  • environment variables to set

group

  • group to run as

logoutput

  • true
  • false
  • on_failure

onlyif

  • only run if $? == 0

path

  • search path
  • ['path1','path2']
  • 'path1:path2'

refresh

  • run something else when refreshing

refreshonly

  • only run when a dependent is changed

returns

  • expected return code

unless

  • only run if $? != 0
  • opposite of onlyif

exec.pp

exec {'setfacl_certmaster_thomas':

path => '/usr/bin',

command => 'setfacl -R -m 'u:thomas:rX' /var/lib/certmaster',

onlyif => 'getfacl -pac /var/lib/certmaster|grep user:thomas:r-x'

}

exec

http://docs.puppetlabs.com/references/latest/type.html#exec

tool to transform files into objects

config changes made to the objects

augtool and augparse command line tools

Why?

  • manipulate parts of a file
  • don't clobber another modules work
  • start from a known good config

augeas

Attributes

changes

context

title:

force

  • array of changes
  • set
  • setm
  • rm
  • clear
  • ins
  • mv
  • defvar
  • defnode
  • set a default path for changes in this instance
  • forget this
  • false

name

  • unique name

onlyif

  • get
  • match

augeas

http://docs.puppetlabs.com/references/latest/type.html#augeas

http://augeas.net/

Augeas is worthy of a complete talk on it's own, but if you know augeas, then you can use it in puppet.

Two simple examples that illustrate it's usefulness

Change a single parameter in sshd_config:

augeas

sshd_config

augeas{ "sshd password authentication":

context => "/files/etc/ssh/sshd_config",

changes => [

"set PasswordAuthentication no"

],

notify => Service["sshd"]

}

sysctl

augeas { "sysctl.conf":

context => "/files/etc/sysctl.conf",

changes => [

"set fs.file-max 80000",

"set net.ipv4.tcp_keepalive_time 300",

"set net.ipv4.ip_local_port_range \"1024 65000\""

],

notify => Exec["sysctl"]

}

sysctl

exec { "sysctl":

command => "/sbin/sysctl -p",

logoutput => true,

refreshonly => true,

}

change a single line in sshd_config

augeas{ "sshd password authentication":

context => "/files/etc/ssh/sshd_config",

changes => [

"set PasswordAuthentication no"

],

notify => Service["sshd"]

}

add entries to the kernel tuning:

augeas { "sysctl.conf":

context => "/files/etc/sysctl.conf",

changes => [

"set fs.file-max 80000",

"set net.ipv4.tcp_keepalive_time 300",

"set net.ipv4.ip_local_port_range \"1024 65000\""

],

notify => Exec["sysctl"]

}

exec { "sysctl":

command => "/sbin/sysctl -p",

logoutput => true,

refreshonly => true,

}

classes

class

attribute

type

fact

variable

attribute

type

variable

variable

class

include otherclass

class

class syntax

class name {

type {title: attributes => values },

type {title: attributes => values }

}

base/default

class base {

file {"puppetized":

path => '/etc/puppetized',

content => template('puppet.erb'),

mode => 0644, owner => root, group => root

}

include dns

include ntp

}

class base {

file {"puppetized":

path => '/etc/puppetized',

content => template('puppet.erb'),

mode => 0644, owner => root, group => root

}

include dns

include ntp

}

puppet.erb:

puppetized host <%= hostname %>

classes realized on this node:

<% classes.each do |klass| -%>

<%= klass -%>ppnnnppppnn

<% end %>

environment of this node <%= environment %>

class hierarchy

parent

nephew

top/global

child

class parent {

$var = "parent"

notice("var in parent is ",$var)

}

class child inherits parent {

$var = "child"

notice("var in child is ",$var)

}

class nephew {

notice("var in nephew is ",$var)

}

[root@learn: ~] $ puppet apply classhierarchy.pp

notice: Scope(Class[Parent]): var in parent is parent

notice: Scope(Class[Child]): var in child is child

notice: Finished catalog run in 0.01 seconds

[root@learn: ~] $

classes can be instantiated as children of defined classes using the "inherits" option in the class definition.

Example:

---

include child

class parent {

$var = "parent"

notice("var in parent is ",$var)

}

class child inherits parent {

$var = "child"

notice("var in child is ",$var)

}

class nephew {

notice("var in nephew is ",$var)

}

---

[root@learn: ~] $ puppet apply classhierarchy.pp

notice: Scope(Class[Parent]): var in parent is parent

notice: Scope(Class[Child]): var in child is child

notice: Finished catalog run in 0.01 seconds

[root@learn: ~] $

The notice in parent is triggered by the inheritance.

The notice in nephew is not triggered

---

include child

include nephew

class parent {

$var = "parent"

notice("var in parent is ",$var)

}

class child inherits parent {

$var = "child"

notice("var in child is ",$var)

}

class nephew {

notice("var in nephew is ",$var)

}

---

[root@learn: ~] $ puppet apply classhierarchy2.pp

notice: Scope(Class[Parent]): var in parent is parent

notice: Scope(Class[Child]): var in child is child

notice: Scope(Class[Nephew]): var in nephew is

notice: Finished catalog run in 0.01 seconds

[root@learn: ~] $

---

nephew is now called but the variable is not defined, we will address this in the scope section coming up.

functions

include

  • include class at this point

file

  • slurp file

join

  • join array object with string

split

  • split a string into an array

defined

  • verify if a class or resource is defined
  • true
  • false

require

  • add a class as a dependency

http://docs.puppetlabs.com/references/stable/function.html

include:

include a class at this point in the manifest

useful for building up classes of classes

file: read in a file

useful for grabbing file contents and placing into a variable

use with content to read in one file and place in another

join: join an array by a string

Example:

$nameservers = ['ns1','ns2','ns3']

file {'/etc/dns':

content => join($nameservers,',')

}

split: split a string into an array

Example:

$inter_array = split($::interfaces,",")

file {"/etc/interfaces":

content => template("interfaces.erb")

}

#interfaces.erb

<% inter_array.each do |eth| -%>

interface <%= eth %>

<% end -%>

defined: verify if a class or resource is defined.

functions

include

file

join

split

defined

require

template

  • apply ERB template and return as string

realize

  • make a virtual object real

notice

  • log a message at notice level
  • other valid levels:
    • crit
    • debug
    • err
    • info
    • warning

http://docs.puppetlabs.com/references/stable/function.html

include:

include a class at this point in the manifest

useful for building up classes of classes

file: read in a file

useful for grabbing file contents and placing into a variable

use with content to read in one file and place in another

join: join an array by a string

Example:

$nameservers = ['ns1','ns2','ns3']

file {'/etc/dns':

content => join($nameservers,',')

}

split: split a string into an array

Example:

$inter_array = split($::interfaces,",")

file {"/etc/interfaces":

content => template("interfaces.erb")

}

#interfaces.erb

<% inter_array.each do |eth| -%>

interface <%= eth %>

<% end -%>

defined: verify if a class or resource is defined.

inline_template

  • apply template to variables
  • pass to a parameter

# iterate over the interfaces

# print out their addresses into ifcfg-files

$ifs = split($interfaces,',')

define inline_template_test {

file {"/tmp/ifcfg-$name":

content => inline_template("IPADDR=<%= @ipaddress_${name} %>\n")

}

}

inline_template_test { $ifs: }

/tmp/ifcfg-lo

IPADDR=127.0.0.1

/tmp/ifcfg-eth0

IPADDR=192.168.122.4

another function, but it is important enough to discuss separately.

filebuckets

Defaults to local-only mode

Server mode:

filebucket { puppet: server => "puppet.example.edu" }

Command-line utility:

# puppet filebucket --local backup /etc/puppet/puppet.conf

/etc/puppet/puppet.conf: be50b3e9acc2c2de8df194b1466fd2c1

# puppet filebucket --local restore /etc/puppet/puppet.conf \

be50b3e9acc2c2de8df194b1466fd2c1

file

file

file

useful when you are puppetizing hosts in place.

Creates backups of files before modifying them. Currently, filebuckets are only useful for manual retrieval of accidentally removed files.

Recap

Terminology

  • everything lives in manifests.
  • clients are nodes
  • nodes have
    • classes
    • types
    • variables
    • facts
  • modules distribute code like plugins.

Types:

file - ways to distribute files.

package - install software

service - start services

Trifecta:

package - file - service

Ordering:

dependencies are specified with ordering

before, after, require, notify

Recap

conditionals/booleans/logic

templates

classes

functions

template,inline_template, split,join,include

filebuckets

Types:

file - ways to distribute files.

package - install software

service - start services

Trifecta:

package - file - service

Ordering:

dependencies are specified with ordering

before, after, require, notify

Break/Questions

Puppet 102 next!

...a bit of fun with vim

vim

.vimrc

" puppet style guide mode

filetype plugin indent on

.vim

└──indent

└──puppet.vim

└──ftdetect

└──puppet.vim

└──syntax

└──puppet.vim

└──ftplugin

└──puppet.vim

Add to .vimrc

" puppet style guide mode

filetype plugin indent on

Put the puppet.vim files in the correct directories under .vim or in /usr/share/vim/vimfiles

emacs also has a puppet mode for the emacs users

learn vm has this enabled by default.

vim demo

Puppet 102

advanced language

server configuration

puppet 102

plusignment

scope

parameterized classes

run stages

resource chaining

puppetmaster (server operation)

webrick

fileserver

client configuration

debugging

puppetca

site.pp

type defaults

referencing/upper case

modules

plugins

custom facts

custom types

pluginsync

passenger

pushing

reporting

enc

environments

stored configurations

virtualizing resources

exported resources

system integration

hiera

plusignment

class openssh {

file { "/etc/ssh/sshd_config":

content => "..."; }

service {

"sshd":

ensure => running,

subscribe => File["/etc/ssh/sshd_config"]; }

class keydistribute {

file {

"/etc/ssh/host_rsa_key":

content => "...";

"/etc/ssh/host_dsa_key":

content => "..."; }

Service["sshd"] {

subscribe +> [ File["/etc/ssh/host_rsa_key"],

File["/etc/ssh/host_dsa_key"] ] }

}

}

class openssh {

file { "/etc/ssh/sshd_config":

content => "..."; }

service {

"sshd":

ensure => running,

subscribe => File["/etc/ssh/sshd_config"]; }

class keydistribute {

file {

"/etc/ssh/host_rsa_key":

content => "...";

"/etc/ssh/host_dsa_key":

content => "..."; }

Service["sshd"] {

subscribe +> [ File["/etc/ssh/host_rsa_key"],

File["/etc/ssh/host_dsa_key"] ] }

}

}

scope!

scope

class base

hostname

class dns

$hostname

class dns

class base

hostname

dynamic scope

puppet was dynamically scoped.

versions 2.7 and above warn about dynamic scope being deprecated. versions 2.8 and above will likely not include dynamic scope.

scope

class base

class other

hostname

class dns

$::hostname

hostname

$myvar

$myvar

undefined

$other::myvar

class other

class dns

scope

include parent::child

$var = "top level"

class parent {

$var = "from parent"

$hostname = "hostname from parent"

}

class parent::child inherits parent {

$var = "from parent::child"

notice ( "parent::var => ",$parent::var )

notice ( "var => ",$var )

notice ( "hostname => ",$hostname )

notice ( "::hostname => ",$::hostname )

}

[root@learn: ~] $ puppet apply scope.pp

notice: Scope(Class[Parent::Child]): parent::var => from parent

notice: Scope(Class[Parent::Child]): var => from parent::child

notice: Scope(Class[Parent::Child]): hostname => hostname from parent

notice: Scope(Class[Parent::Child]): ::hostname => learn

notice: Finished catalog run in 2.69 seconds

http://docs.puppetlabs.com/guides/scope_and_puppet.html

# scope.pp

include parent::child

class parent {

$var = "from parent"

$hostname = "hostname from parent"

}

class parent::child inherits parent {

$local_var = "from parent::child"

notice ( "parent::var",$parent::var )

notice ( "var",$var )

notice ( "local_var",$local_var )

notice ( "hostname",$hostname )

notice ( "::hostname",$::hostname )

}

[root@learn: ~] $ puppet apply scope.pp

notice: Scope(Class[Parent::Child]): parent::var from parent

notice: Scope(Class[Parent::Child]): var from parent

notice: Scope(Class[Parent::Child]): local_var from parent::child

notice: Scope(Class[Parent::Child]): hostname hostname from parent

notice: Scope(Class[Parent::Child]): ::hostname burnaby

notice: Finished catalog run in 0.01 seconds

[root@learn: ~] $

parameterized classes

# unparameterized

include resolver

class resolver {

$dnsservers = ['8.8.8.8','8.8.4.4']

file {"resolver-conf":

path => "/tmp/resolv.conf",

content => template("/root/resolver-conf.erb")

}

}

resolver-conf.erb

domain <%= domain %>

<% dnsservers.each do |dns| -%>

nameserver <%= dns %>

<% end -%>

/tmp/resolv.conf

domain localdomain

nameserver 8.8.8.8

nameserver 8.8.4.4

parameterized classes

# parameterized

class {"resolver":

dnsservers => ['192.168.1.1','192.168.4.1']

}

class resolver ($dnsservers =['8.8.8.8','8.8.4.4']) {

file {"resolver-conf":

path => "/tmp/resolv.conf",

content => template("/root/resolver-conf.erb")

}

}

resolver-conf.erb

domain <%= domain %>

<% dnsservers.each do |dns| -%>

nameserver <%= dns %>

<% end -%>

/tmp/resolv.conf

domain localdomain

nameserver 192.168.1.1

nameserver 192.168.1.4

parameterized classes

# parameterized2

class resolver ($mydnsservers = '') {

$maindnsservers = '8.8.8.8,8.8.4.4'

if $mydnsservers {

$dnsservers = split(sprintf("%s,%s",$mydnsservers,$maindnsservers),",")

} else {

$dnsservers = split($maindnsservers,",")

}

file {'resolver-conf':

path => '/tmp/resolv.conf',

content => template('/etc/picc/resolver-conf.erb')

}

}

class {'resolver':

mydnsservers => "192.168.1.1,192.168.4.1"

}

resolver-conf.erb

domain <%= domain %>

<% dnsservers.each do |dns| -%>

nameserver <%= dns %>

<% end -%>

/tmp/resolv.conf

domain localdomain

nameserver 192.168.1.1

nameserver 192.168.1.4

nameserver 8.8.8.8

nameserver 8.8.4.4

custom defines

Move beyond the singletons of typical puppet manifests.

Defines are used to model repeatable chunks of puppet code.

Common examples include versioning repositories and apache virtual hosts.

Not much to explain, let's just take a look at an example...

class git {

package{ "git": }

file {'/var/lib/git':

ensure => 'directory',

mode => 0755,

owner => 0, group => 0

}

}

define gitrepo($owner, $group, $ispublic = false) {

include git

$mode = $ispublic ? { true => 2774, false => 2770 }

file { "/var/lib/git/${title}":

ensure => directory,

owner => "${owner}",

group => "${group}",

mode => $mode,

require => [Package['git'],File['/var/lib/git']]

}

exec { "${title}_initialize":

command => '/usr/bin/git --bare init .',

cwd => "/var/lib/git/${title}",

unless => "/bin/test -f /var/lib/git/${title}/HEAD",

require => [Packages['git'],File["/var/lib/git/${title}"]]

}

}

lopsa.pp

gitrepo {"lopsa":

owner => 'thomas',

group => 'puppet',

}

mkdir /var/lib/git/lopsa

chown thomas /var/lib/git/lopsa

chgrp puppet /var/lib/git/lopsa

chmod 2770 /var/lib/git/lopsa
(cd /var/lib/git/lopsa; /usr/bin/git --bare init .)

Run stages & resource chaining

"chain" operators

Class["setup_repos"] -> Class["install_packages"] -> Class["ldap::client"]

Package["openssh-server"] -> File["/etc/ssh/sshd_config"] -> Service["sshd"] <- File["/etc/ssh/ssh_host_rsa_key.pub"]

setup_repos

install_packages

ldap::client

before =>

Class['install_packages',

'ldap::client']

before => Class['ldap::client']

require => Class['setup_repos']

require =>

Class['setup_repos',

'install_packages']

make server on vms

test servered operation tomorrow morning

Run stages - the parameterized way

stage {

"pre":

before => Stage["main"];

"post":

require => Stage["main"];

}

node "server.example.com" {

class {

"setup_repos":

stage => "pre";

"install_packages":

stage => "main";

"ldap::client":

stage => "post";

}

}

puppetmaster

Allows all puppet work to be done in a single location and then distributed via a pull-based puppet client.

puppetmaster

Web

server

DHCP

server

DNS

server

Database

server

TCP

port 8140

SSL

x509

Puppet communication is http. The port number is 8140.

Communication is done through SSL encryption. The certificate authority for puppet is called puppetca and is managed via the command of the same name or via "puppet cert"

puppetmaster

  • Pairs well with a versioning system such as git or svn to create living manifests.

  • Options controlled through [master] section in puppet.conf

  • "puppet-server" package. Creates user/group puppet.

  • Simplest case is webrick, works out of the box.

webrick

Works without any additional configuration

service puppetmaster start

iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 8140 -j ACCEPT

[root@jim ~] # lsof -i |grep 8140

puppetmas 3211 puppet 6u IPv4 17188 0t0 TCP *:8140 (LISTEN)

file {"issue":

path => "/etc/motd"

source =>"puppet:///files/hello";

}

/etc/motd

Hello puppet!

/etc/puppet/files/hello

Hello puppet!

http://en.wikipedia.org/wiki/WEBrick

WEBrick was primarily written by Masayoshi Takahashi and Yuuzou Gotou

Ruby, used by ruby on rails

Does not scale well, useful in installations of less than 10 clients.

Easy to configure, just start puppetmaster service

Good for toy systems and testing

Allows for use of puppet:/// paths for files using source

puppetmaster as a fileserver

If you use "source => puppet:///" in any of your file definitions, this is how those files get distributed.

/etc/puppet/fileserver.conf

[files]

path /var/lib/puppet/files

allow *.example.com

allow *.friendly.example.org

deny *.malice.example.net

deny impostor.friendly.example.org

puppet:/// or puppet://?

puppet:/// implies

puppet://$puppetserver/

...you could do this though...

puppet://filespuppet.example.com/

have autosign on at this point, show that later.

server configuration

puppet.conf

auth.conf

fileserver.conf

manifests/site.pp

/etc/puppet/auth.conf

# ACL configuration

# allow nodes to retrieve their own catalog

path ~ ^/catalog/([^/]+)$

method find

allow $1

# allow all nodes to access the certificates services

path /certificate_revocation_list/ca

method find

allow *

/etc/puppet/puppet.conf

[main]

logdir = /var/log/puppet

rundir = /var/run/puppet

ssldir = $vardir/ssl

[agent]

classfile = $vardir/classes.txt

localconfig = $vardir/localconfig

http://docs.puppetlabs.com/references/stable/configuration.html

#puppet.conf

[main] - options that apply to the puppet master

logdir - where to log everything

rundir - where to store the PID files

ssldir - where to store certificates for puppetca

[agent] - options that apply to the agent (client)

classfile - a file that stores the classes applied to this node (client)

localconfig - where to cache local configuration

This is a minimal configuration, we will be adding to it in later sections.

#auth.conf

# allow nodes to retrieve their own catalog (ie their configuration)

path ~ ^/catalog/([^/]+)$

method find

allow $1

# allow nodes to retrieve their own node definition

path ~ ^/node/([^/]+)$

method find

allow $1

# allow all nodes to access the certificates services

path /certificate_revocation_list/ca

method find

allow *

# allow all nodes to store their reports

path /report

method save

allow *

# inconditionnally allow access to all files services

# which means in practice that fileserver.conf will

# still be used

path /file

allow *

### Unauthenticated ACL, for clients for which the current master doesn't

### have a valid certificate; we allow authenticated users, too, because

### there isn't a great harm in letting that request through.

# allow access to the master CA

path /certificate/ca

auth any

method find

allow *

path /certificate/

auth any

method find

allow *

path /certificate_request

auth any

method find, save

allow *

# this one is not stricly necessary, but it has the merit

# to show the default policy which is deny everything else

path /

auth any

defaults in this file are usually ok

fileserver.conf

server configuration

/etc/puppet/fileserver.conf

[files]

path /var/lib/puppet/files

allow *.example.com

[modules]

allow *.example.com

[facts]

path /var/lib/puppet/facts

allow *.example.com

[plugins]

path /var/lib/puppet/plugins

allow *.example.com

http://docs.puppetlabs.com/references/stable/configuration.html

#puppet.conf

[main] - options that apply to the puppet master

logdir - where to log everything

rundir - where to store the PID files

ssldir - where to store certificates for puppetca

[agent] - options that apply to the agent (client)

classfile - a file that stores the classes applied to this node (client)

localconfig - where to cache local configuration

#auth.conf

# allow nodes to retrieve their own catalog (ie their configuration)

path ~ ^/catalog/([^/]+)$

method find

allow $1

# allow nodes to retrieve their own node definition

path ~ ^/node/([^/]+)$

method find

allow $1

# allow all nodes to access the certificates services

path /certificate_revocation_list/ca

method find

allow *

# allow all nodes to store their reports

path /report

method save

allow *

# inconditionnally allow access to all files services

# which means in practice that fileserver.conf will

# still be used

path /file

allow *

### Unauthenticated ACL, for clients for which the current master doesn't

### have a valid certificate; we allow authenticated users, too, because

### there isn't a great harm in letting that request through.

# allow access to the master CA

path /certificate/ca

auth any

method find

allow *

path /certificate/

auth any

method find

allow *

path /certificate_request

auth any

method find, save

allow *

# this one is not stricly necessary, but it has the merit

# to show the default policy which is deny everything else

path /

auth any

defaults in this file are usually ok

fileserver.conf

client configuration

/etc/sysconfig/puppet

PUPPET_SERVER=puppet.example.com

#PUPPET_PORT

#PUPPET_LOG

#PUPPET_EXTRA_OPTS=--waitforcert=500

/etc/puppet/puppet.conf

[agent]

server = puppet.example.com

by default puppet will look for a puppet server at puppet.$domain

RHEL/Fedora based systems have a /etc/sysconfig/puppet config file.

To be agnostic, use the puppet.conf file instead.

it's go time.

on the master:
service puppetmaster start

on the client:
service puppet start

...hope for the best.

an aside...debugging clients

[root@client ~] # service puppet stop

Run and print the log output to the shell:
[root@client ~] # puppet agent -o --no-daemonize -v
or
[root@client ~] # puppet agent -t

View even more information on the shell:
[root@client ~] # puppet agent -o --no-daemonize --debug

View a crazy, incomprehensible amount of info on the shell:
[root@client ~] # puppet agent -o --no-daemonize --trace

info: Creating a new SSL key for kermit.henson.local

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for ca

warning: peer certificate won't be verified in this SSL session

warning: peer certificate won't be verified in this SSL session

info: Creating a new SSL certificate request for kermit.henson.local

info: Certificate Request fingerprint (md5): F4:E2:10:61:1C:F7:F8:E4:03:98:14:B0:F7:47:C8:9E

warning: peer certificate won't be verified in this SSL session

warning: peer certificate won't be verified in this SSL session

warning: peer certificate won't be verified in this SSL session

Exiting; no certificate found and waitforcert is disabled

This is stupid, it doesn't work. Nothing is happening.

puppetca

puppetca

Distributes and manages x509 certificates.

Command line utility: puppet cert
(puppetca for oldtimers)

Supports auto-signing

/etc/puppet/autosign.conf

*.example.com
*.subdomain.example.net

it's go time

server

[root@jim puppet]# puppet cert list

kermit.henson.local (F4:E2:10:61:1C:F7:F8:E4:03:98:14:B0:F7:47:C8:9E)

[root@jim puppet]# puppet cert sign kermit.henson.local

notice: Signed certificate request for kermit.henson.local

notice: Removing file Puppet::SSL::CertificateRequest kermit.henson.local at '/var/lib/puppet/ssl/ca/requests/kermit.henson.local.pem'

start up a passenger based config, start httpd

show some logs

it's go time

[root@kermit ~]# puppet agent -v --no-daemonize -o

warning: peer certificate won't be verified in this SSL session

info: Caching certificate for kermit.henson.local

info: Caching certificate_revocation_list for ca

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find default node or by name with 'kermit.henson.local, kermit.henson, kermit' on node kermit.henson.local

notice: Using cached catalog

err: Could not retrieve catalog; skipping run

[root@kermit ~]#

start up a passenger based config, start httpd

show some logs

site.pp

master manifest

needs to define each node or a default node

simplest* site.pp

node default {

}

I

technically a completely empty site.pp is still valid, just not useful.*

Actually an empty site.pp with all configuration in modules and nodes defined in an ENC is completely valid *and* useful

site.pp

import 'nodes.pp'

import 'functions.pp'

import 'variables.pp'

import 'defaults.pp'

/etc/puppet/manifests/nodes.pp

node www {

include webserver

}

node ldap {

include ldapserver

}

node default {

include base

}

/etc/puppet/manifests/functions.pp

define append_if_no_such_line($file, $line, $refreshonly = 'false') {

exec { "/bin/echo '$line' >> '$file'":

unless => "/bin/grep -Fxqe '$line' '$file'",

path => "/bin",

refreshonly => $refreshonly

}

}

/etc/puppet/manifests/variables.pp

$osmajor = regsubst($operatingsystemrelease, '([^.]*)[.].*','\1')

reasonable configuration:

--- site.pp ---

import 'nodes.pp'

import 'functions.pp'

import 'variables.pp'

import 'defaults.pp'

--- nodes.pp --

node www {

include webserver

}

node ldap {

include ldapserver

}

node default {

include base

}

--- functions.pp ---

define append_if_no_such_line($file, $line, $refreshonly = 'false') {

exec { "/bin/echo '$line' >> '$file'":

unless => "/bin/grep -Fxqe '$line' '$file'",

path => "/bin",

refreshonly => $refreshonly

}

}

--- variables.pp ---

$osmajor = regsubst($operatingsystemrelease, '([^.]*)[.].*','\1')

*** we'll do something better with this in a little bit.

--- defaults.pp ---

Package { ensure => present }

Service { ensure => true, enable => true, hasrestart => true }

Mount { ensure => "mounted" }

Group { ensure => present }

User { ensure => present,managehome => true, }

type defaults (defaults.pp)

capitalize the type:

...but do not give a title

These "attribute => value" pairs will now be the defaults for all objects of type "file". You are free to overwrite them later.

File {

user => root,

group => root,

mode => 0644

}

defaults.pp is a convention only, you still need to include the file in sites.pp with an include call.

tip: tag template

tagprinter.erb

tags

----

<% tags.each do |tag| -%>

<%= tag %>

<% end -%>

classes

-------

<% classes.each do |cls| -%>

<%= cls %>

<% end -%>

variables

-------

<% scope.to_hash.keys.each do |var| -%>

<%= var -%> => <%= scope.lookupvar(var) %>

<% end -%>

/tmp/tags

tags

----

class

tagprinter

...

classes

-------

base

tagprinter

kermit

...

variables

-------

rubyversion => 1.8.7

uptime_seconds => 2468

macaddress_eth0 => 52:54:00:3F:3A:AC

class tagprinter ($class = 'default') {

file {$class:

path => "/tmp/${class}-tags.pp",

content => template("tagprinter/tagprinter.erb")

}

}

modules

{modulepath}

└── {modulename}

└── files

└── manifests

└── templates

modules

{modulepath}

└── {modulename}

└── files

└── manifests

└── init.pp

└── templates

modules

init.pp

class modulename {

file {"example.conf":

path => "/tmp/example.conf",

source => "puppet:///modules/modulename/example.conf"

}

}

{modulepath}/{modulename}/files/example.conf

{modulepath}/{modulename}/manifests/init.pp

modules

subclass.pp

class modulename::subclass {

file {"subclass.conf":

path => "/tmp/subclass.conf",

source => "puppet:///modules/modulename/subclass.conf"

}

}

{modulepath}/{modulename}/files/subclass.conf

{modulepath}/{modulename}/manifests/subclass.pp

modules

subclass.pp

class modulename::subclass {

file {"subclass.conf":

path => "/tmp/subclass.conf",

content => template("modulename/subclass.erb")

}

}

{modulepath}/{modulename}/templates/subclass.erb

module cheat sheet

http://docs.puppetlabs.com/module_cheat_sheet.pdf

http://docs.puppetlabs.com/module_cheat_sheet.pdf

style guide

http://docs.puppetlabs.com/guides/style_guide.html

  • two-space soft tabs (no literal tab characters)
  • no trailing white space
  • 80 character line width
  • fat comma arrows (=>) within blocks of attributes
  • # for comments
  • 'my variable' unless "my ${variable}"
  • ensure first
  • defaults in site.pp
  • separate files for all classes in a module

two-space soft tabs, no actual tab characters

no trailing white space (git will love you)

80 characters per line

align the => within a block using the longest attribute name + 1 space

only use # for comments

single quotes for variables unless interpolating a variable within the variable

always put ensure first if ensure is specified.

resource defaults should be in site.pp or something included from site.pp to avoid issues with scope.

classes within modules should have separate files

if: the module is mymodule, and there is a class mymodule::mysubclass

then: manifests would contain init.pp for mymodule and mysubclass.pp for mymodule::mysubclass

forge

http://forge.puppetlabs.com/

> 2.7.14

puppet module

< 2.7.14

puppet-module

plugins

{modulepath}

└── {modulename}

└── files

└── lib

└── facter

└── puppet

└── parser

└── functions

└── provider

└── exec

└── package

└── anytype

└── type

└── util

pluginsync needs to be configured.

custom facts

/etc/puppet/modules/videocard/lib/facter/videocard.rb

# Josko Plazonic - lifted from Josko March 14, 2011 by Thomas Uphill

require 'facter'

Facter.add("videocard") do

confine :kernel => :linux

ENV["PATH"]="/bin:/sbin:/usr/bin:/usr/sbin"

setcode do

controllers = []

lspciexists = system "/bin/bash -c 'which lspci >&/dev//null'"

if $?.exitstatus == 0

output = %x{lspci}

output.each {|s|

controllers.push($1) if s =~ /VGA compatible controller: (.*)/

}

end

controllers

end

end

custom types

/etc/puppet/modules/newtype/lib/puppet/type/newtype.rb

# this is beyond the context of this talk...

# best to look on http://forge.puppetlabs.com/

Puppet::Type.newtype(:newtype)do

newproperty(:myprop, :array_matching => :all)do

end

end

There are many good plugins on puppet forge

http://forge.puppetlabs.com/

http://docs.puppetlabs.com/guides/plugins_in_modules.html

pluginsync

plugins synced from master to the clients

/etc/puppet/module

something

lib

facter

puppet

/var/lib/puppet

lib

facter

puppet

puppet.conf

[agent]

pluginsync = true

pluginsync

videocard example, videocard.rb

puppet.conf -> [agent] pluginsync = true

info: Retrieving plugin

notice: /File[/var/lib/puppet/lib/videocard.rb]/ensure: defined

info: Loading downloaded plugin var/lib/puppet/lib/videocard.rb

DEMO

kermit factor | videocard

Modules + Facts + Templates

Control access to system users via cgroups

Filestructure:

/etc/puppet/modules/cgroups

└── templates

└── cgconfig.conf

└── files

└── cgrules.conf

└── manifests

└── init.pp

└── lib

└── facter

└── cgroups.rb

manifests/init.pp

class cgroups {

package {

"libcgroup":

ensure => present;

}

service {

"cgconfig":

ensure => running,

enable => true,

require => Package["libcgroup"];

"cgred":

ensure => running,

enable => true,

require => [ Package["libcgroup"], Service["cgconfig"] ];

}

file {

"/etc/cgconfig.conf":

content => template("cgroups/cgconfig.conf"),

notify => [ Service["cgconfig"], Service["cgred"] ];

"/etc/cgrules.conf":

source => "puppet:///modules/cgroups/cgrules.conf",

notify => [ Service["cgconfig"], Service["cgred"] ];

}

}

files/cgrules.conf

@staff cpu staff/

@student cpu,memory users/

@faculty cpu,memory users/

@grad cpu,memory users/

@guest cpu,memory users/

@alumni cpu,memory users/

* cpu system/

templates/cgconfig.conf

group staff {

cpu {

cpu.shares = 20;

}

}

group users {

cpu {

cpu.shares = 60;

}

memory {

memory.limit_in_bytes = <%= mem80pct %>;

memory.memsw.limit_in_bytes = <%= memswap80pct %>;

}

}

group system {

cpu {

cpu.shares = 20;

}

lib/facter/cgroups.rb

Facter.add(:mem80pct) do

setcode do

Facter::Util::Resolution.exec('\

free -bto | \

grep Mem | \

awk \'{OFMT = "%.0f"; print $2*.8}\'\

')

end

end

Facter.add(:memswap80pct) do

setcode do

Facter::Util::Resolution.exec('\

free -bto | \

grep Total | \

awk \'{OFMT = "%.0f"; print $2*.8}\'\

')

end

end

Filestructure:

[root@puppet cgroups]# pwd

/etc/puppet/modules/cgroups

[root@puppet cgroups]# find .

./templates

./templates/cgconfig.conf

./files

./files/cgrules.conf

./manifests

./manifests/init.pp

./lib

./lib/facter

./lib/facter/cgroups.rb

--- manifests/init.pp ---

class cgroups {

package {

"libcgroup":

ensure => present;

}

service {

"cgconfig":

ensure => running,

enable => true,

require => Package["libcgroup"];

"cgred":

ensure => running,

enable => true,

require => [ Package["libcgroup"], Service["cgconfig"] ];

}

file {

"/etc/cgconfig.conf":

content => template("cgroups/cgconfig.conf"),

notify => [ Service["cgconfig"], Service["cgred"] ];

"/etc/cgrules.conf":

source => "puppet:///modules/cgroups/cgrules.conf",

notify => [ Service["cgconfig"], Service["cgred"] ];

}

}

Production - Use passenger

Apache a better webserver than webrick, mongrel support is deprecated.

Much more stable.

config.ru - must be owned by puppet

Passenger - Apache configuration

LoadModule passenger_module modules/mod_passenger.so

<IfModule mod_passenger.c>

PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-3.0.2

PassengerRuby /usr/bin/ruby

</IfModule>

RackAutoDetect Off

RailsAutoDetect Off

Listen 8140

<VirtualHost *:8140>

RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e

RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e

RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

DocumentRoot /etc/puppet/rack/public/

RackBaseURI /

</VirtualHost>

Performance Passenger

Puppet docs recommend the use of the following apache config directives:

# Refresh old puppetmaster processes after 5 minutes...

PassengerPoolIdleTime 300

# Set to num_clients * 1.15, 15% more than required...

PassengerMaxPoolSize 50

# Puppetmaster takes a while, just use one global queue...

PassengerUseGlobalQueue on

# We don't need most of the passenger stuff, disable it...

PassengerHighPerformance on

Keep an eye on memory usage!

Puppet pushing

Puppet is strictly pull-based by design.

Cannot truly push changes.

"puppetrunner" allows the master to remotely trigger the agent to run.

/etc/puppet/puppet.conf

[agent]

listen = true

Reporting

Agent creates a state.yaml file detailing the results of each run. Have the agent send the report to the master:

[agent]

report = true

Report plugins parse state.yaml from the agents and sends the data off to some other utility.

[master]

reports=<report_format>

Reporting Engines

Common report formats include:

  • http
  • log
  • rrdgraph
  • tagmail
  • puppetdashboard (PE Console)
  • foreman

Providers are located in:

${RUBY_LIBDIR}/puppet/reports/*.rb

Puppet dashboard

Foreman

foreman

Can extend an existing puppet database, removing need for puppet/foreman syncing scripts.

Extensible, can control DHCP, DNS, etc

http://theforeman.org/

http://theforeman.org/

ENC

external node classifier

  • script
  • ldap

  • classes
  • top scope variables
  • environment

http://docs.puppetlabs.com/guides/external_nodes.html

ENC is useful if you have separated all your configuration out into classes and modules. A site where nodes are defined solely by classes will fit well with ENC and possibly be much simpler.

ENC

external node classifier

  • execute a script to determine classes to apply to a node.

  • script receives fqdn as argument 1
  • must return yaml:
    • classes
    • top scope variables
    • environment

http://docs.puppetlabs.com/guides/external_nodes.html

http://yaml.org/

http://en.wikipedia.org/wiki/YAML

ENC

/usr/bin/puppet_node_classifier

#!/usr/bin/ruby

require 'yaml'

# create an empty hash

@enc = Hash.new

@enc["classes"] = Hash.new

@enc["classes"]["base"] = Hash.new

@enc["parameters"] = Hash.new

@enc["environment"] = 'production'

if ARGV[0] =~ /^www/

@enc["classes"]["webserver"] = Hash.new

end

if ARGV[0] =~ /^kermi/

@enc["classes"]["kermit"] = Hash.new

end

puts @enc.to_yaml

exit(0)

/usr/bin/puppet_node_classifier

#!/usr/bin/python

import yaml

import sys

# create an empty hash

enc = {}

enc["classes"] = {}

enc["classes"]["base"] = {}

enc["parameters"] = {}

enc["environment"] = 'production'

if (sys.argv[1].find("www") != -1):

enc["classes"]["webserver"] = {}

if (sys.argv[1].find("kermi") != -1):

enc["classes"]["kermit"] = {}

# output the ENC as yaml

print "---"

print yaml.dump(enc)

sys.exit(0)

[root@jim bin]# puppet_node_classifier kermit.henson.local

---

parameters: {}

environment: production

classes:

kermit: {}

base: {}

[root@jim bin]# puppet_node_classifier_python kermit.henson.local

---

classes:

base: {}

kermit: {}

environment: production

parameters: {}

/etc/puppetlabs/puppet-dashboardexternal_node

#!/bin/bash

set -e

set -u

ENC_BASE_URL="https://localhost:443/nodes"

curl -k -H "Accept: text/yaml" "${ENC_BASE_URL}/${1}"

[root@jim bin]# external_node kermit.henson.local

---

classes: []

http://www.ruby-lang.org/en/documentation/

http://tryruby.org/levels/1/challenges/0

#!/bin/bash

echo "---"

echo "classes:"

echo " base:"

echo

echo "environment: production"

#!/usr/bin/ruby

ENC LDAP

[master]

node_terminus = ldap

ldapserver = ldap.henson.local

ldapbase = ou=hosts,dc=henson,dc=local

schema:

attributeTypes:

puppetclass

parentnode

environment

puppetvar

puppetClient

# kermit.henson.local servers, hosts,kermit.henson.local

dn: cn=kermit.henson.local,ou=servers,ou=hosts,dc=henson,dc=local

cn: kermit.henson.local

ipHostNumber: 192.168.122.4

puppetclass: server

puppetclass: logpolicy

puppetclass: netwatch::server

objectClass: top

objectClass: iphost

objectClass: puppetclient

environments

[myenviro]

modulepath = $confdir/environments/myenviro/modules

manifest = $confdir/environments/myenviro/site.pp

manifestdir = $confdir/environments/myenviro/manifests

templatedir = $confdir/environments/myenviro/templates

modules

alice's

modules

bob's

modules

environments

[master]

modulepath = $confdir/environments/$environment/modules:$confdir/modules

manifest = $confdir/environments/$environment/manifests/site.pp

[production]

modulespath = $confdir/production/modules

manifest = $confdir/production/manifests/site.pp

[dev]

modulespath = $confdir/test/modules

manifest = $confdir/test/manifests/site.pp

environments + git

production

bobsbranch

machineX

production

bobsbranch

environments + git

  • make new branches
    git branch bobsbranch
  • ssh-keys from git to puppetmaster
  • update environments directory on commit to git
    puppet.git/hooks/post-receive
  • tell agent to use the new environment
    puppet agent -t --environment bobsbranch
    or
    /etc/puppet/puppet.conf
    [agent]
    environment=bobsbranch

Stored Configurations

Enables use of more advanced puppet features.

Used as a cache, speeds up catalog creation.

puppet.conf

[master]

storeconfigs=true

dbadapter=mysql

dbname=puppetmaster

dbuser=puppetmaster

dbpassword=superdupersecret

dbserver=localhost

http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration

/usr/share/puppet/ext/puppetstoredconfigclean.rb

Stored Configurations DB Schema

fact_names

inventory_facts

param_values

param_names

inventory_nodes

resources

hosts

fact_values

source_files

puppet_tags

resource_tags

Ties fact IDs to human-parsable names.

Values at the intersection of fact IDs and host IDs

Ties host IDs to human-parsable names.

Used in conjunction with the facts_terminus config directive.

Used in conjunction with the facts_terminus config directive.

Ties fact IDs to human-parsable names. (e.g. ensure, notify, content)

Values at the intersection of param IDs and host IDs

Assigns an ID to every "tag" parameter set throughout manifests.

Intersection of resource IDs and puppet tag IDs

Assigns an ID for every object created, along with some extras.

IDs files that the puppetmaster is parsing

Stored Configurations DB Schema

Sample entry from "resources" table:

id: 11518

title: node001.example.com

restype: Sshkey

host_id: 28

source_file_id: 3

exported: 1

line: 307

updated_at: 2012-04-23 15:25:15

created_at: 2012-04-23 15:25:15

Virtualizing Resources

Create a resource, but don't actually send it to the host, keep it local to this manifest.

@ operator

@file { ... }

@service { ... }

@my_custom_type { ... }

The inverse - resource realization

Ok I changed my mind, send this resource to the host.

<| |> operator, or more recently, the realize() function.

File <| |> realize(File["title"])

Service <| |> realize(Service["title"])

My_custom_type <| |> realize(My_custom_type["title"])

Realize use case

base.pp:

class base {

file {

"/tmp/a":

mode => 644;

}

@user {

"lopsa":

ensure => present;

}

}

class manifest1 inherits base {

realize(User["lopsa"])

...

}

class manifest2 inherits base {

realize(User["lopsa"])

...

}

class manifest3 inherits base {

...

}

exported resources cheat sheet

Exporting resources

Just sets the "exported" flag in mysql. These resources are now visible to all hosts.

@@ operator

@@file { ... }

@@service { ... }

@@my_custom_type { ... }

requires storedconfigs

Collecting exported resources

This setup is really helpful for inter-machine communication.

<<| |>> operator

File <<| |>>

Service <<| |>>

My_custom_type <<| |>>

See where this is heading?

In base.pp:

@@sshkey {

"${::fqdn}":

ensure => present,

type => "ssh-rsa",

key => "${::sshrsakey}",

require => Service["sshd"];

}

Sshkey <<| |>>

Done! Now, all of your machines know every other machine's SSH key.

The SSH warning, "are you sure you want to continue? (yes/no):" is a thing of the past!

Of course, this only works for machines within the puppet infrastructure.

Exported resources "gotcha"

Be careful when picking the title of any resource that's to be exported.

@@file["/etc/motd"] { mode => 644 }

You'd think this would be fine, but you'll get a "cannot redefine" error if more than one host runs it. Instead, do something like:

@@file {

"/etc/motd_${::fqdn}":

path => "/etc/motd",

mode => 644;

}

Grouping exported resources

Remember the builtin "tag" parameter?

@@sshkey{

"${::fqdn}":

key => ${::sshrsakey},

tag => "group1";

}

Sshkey <<| tag == "group1" |>>

Multiple group membership

Tags can be set as an array, but collected based on one value.

@@sshkey{

"${::fqdn}":

key => "${::sshrsakey}",

tag => [ "internal", "external" ];

}

if $::hostname =~ /internal.example.com/ {

Sshkey <<| tag == "internal" |>>

}

else {

Sshkey <<| tag == "external" |>>

}

Modify-on-collect

Interact with resources that only exist in the local manifest.

service {

"sshd":

ensure => running;

}

Sshkey <<| |>> {

notify => Service["sshd"]

}

Systems Integration

Using exported resources, one can build a network adaptive to IP address & hostname changes.

Most common cases are system installation and some form of monitoring.

Puppet has builtin types for integrating with nagios. We'll add in bits to handle monitoring system firewall rules.

simple server/client autoconfig

client

serverA

192.168.1.1

80

192.168.1.2

allow 192.168.1.2 dport 80

server address is 192.168.1.1:80

serverB

192.168.2.1

80

allow 192.168.1.2 dport 80

server address is 192.168.2.1:80

FILE

FIREWALL

simple server/client autoconfig

File (tell client where to find the server)

client stores config in /etc/service.conf

class server {

@@file {"server_$::fqdn":

path => '/etc/service.conf',

content => inline_template('server=<%= ipaddress -%>'),

tag => 'server-config'

}

}

class client {

File <<| tag == 'server-config' |>>

}

simple server/client autoconfig

Firewall (tell server to allow client)

class client {

@@firewall {"80 allow client $::fqdn port 80":

proto => 'tcp',

source => "$::ipaddress",

dport => "80",

action => 'accept',

tag => 'allow-client'

}

}

class server {

Firewall <<| tag == 'allow-client' |>>

}

https://github.com/puppetlabs/puppetlabs-firewall

Integration with anaconda/kickstart

%post

chvt 6

/bin/false

count=0

exit=1

while [ $exit -ne 0 ]; do

puppet agent -t \

--server puppetmaster.example.com --waitforcert 60 >/dev/tty6 2>&1

exit=$?

count=$((count + 1))

if [ $count >= 5 ]; then

break

fi

done

if [ $exit == 0 ]; then

chkconfig puppet on

else

read "Puppet didn't run successfully, investigate"

fi

%end

%packages --nobase

@core

puppet

%end

hiera

common

RedHat

SUSE

x86_64

RedHat

SUSE

6

5

x86_64

hiera.yaml

---

:hierarchy:

- %{lsbmajdistrelease}

- %{operatingsystem}

- %{architecture}

- common

:backends:

- yaml

:yaml:

:datadir: '/etc/puppetlabs/puppet/hieradata'

hiera.yaml

hiera.pp

$lopsa = hiera('lopsa')

$lopsa_all = hiera_array('lopsa')

notify {"hiera says lopsa is $lopsa": }

notify {"hiera says lopsa could be $lopsa_all": }

file {'/tmp/hiera':

content => inline_template("<% @lopsa_all.each do |lop| -%><%= lop -%>,<% end-%>")

}

CentOS.yaml

---

lopsa : west

common.yaml

---

lopsa : - east

- cascadia

notice: hiera says lopsa could be westeastcascadia

notice: /Stage[main]//Notify[hiera says lopsa could be westeastcascadia]/message: defined 'message' as 'hiera says lopsa could be westeastcascadia'

notice: hiera says lopsa is west

/tmp/hiera

west,east,cascadia,

Final Q & A

Thanks for attending!

Any questions?

Thanks to:

Ben Rose

allmybase.com

Thank You for Attending LOPSA-East ‘13

Please fill out the Trainer Evaluation


Rate LOPSA-East ‘13

http://lopsa-east.org/2013/training-survey

http://www.lopsa-east.org/2013/rate-lopsa-east-13

Nagios integration

Only if there is enough time...

Uses everything we've talked about.

Ben in charge of this part...

Naginator

Started as a module/plugin. Became so useful, stable, and well-tested, it was cooked straight into base installs.

We'll use this following example as a "final review" of sorts. The example will get more complex the further into the manifest we go.

In the following examples, assume we've already created (or downloaded from the puppet forge!) the modules "apache" and "iptables".

nagios_host

Attributes

address

alias

title:

check_command

IP Address for Connections

check_period

contacts

ensure

host_name

target

Display Address

check_ping,

check_ssh,

etc

e.g. "24x7",

"workhours"

Who shall we notify?

present

absent

FQDN for connections.

Filename

service starts or stops a service on the agent

nagios_service

Attributes

check_command

check_interval

title:

check_period

notification_interval

ensure

host_name

max_check_attempts

target

check_ping,

check_ssh,

etc

Checking frequency

"24x7",

"workhours",

etc

Notification frequency

present

absent

Corresponds to host definition

Failures before notification occurs.

File destination

service starts or stops a service on the agent

Bare minimum nagios manifest [1/9]

class nagios {

package {

"nagios":

ensure => present;

"nagios-plugins-all":

ensure => present;

}

service {

"nagios":

ensure => running,

enable => true,

require => Package["nagios"];

}