1 of 72

puppet troubleshooting

Thomas Uphill

<thomas@narrabilis.com>

https://goo.gl/8LyZzN

2 of 72

Mastering Puppet / Puppet Cookbook

Packt Pub

https://goo.gl/8LyZzN

3 of 72

LOPSA / SASAG / PUGS

https://goo.gl/8LyZzN

4 of 72

When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth

Sir Arthur Conan Doyle

4

5 of 72

Troubleshooting 101

  • document current state
  • discover recent changes
    • audit everything, even things you don't touch
    • never assume it's magic
  • change one thing at a time
    • if it doesn't fix the problem, revert it

​

5

1 une uno um unis odin jeden jedward yksi 一 एक واحد אחד

https://goo.gl/8LyZzN

6 of 72

Troubleshooting 101

  • don't be afraid of diving deep
    • you can't fix what you don't understand
  • use low level tools

​

6

https://goo.gl/8LyZzN

7 of 72

TAKE HOME

puppet is an HTTPS service

7

https://goo.gl/8LyZzN

8 of 72

problems

connection

certificate�catalog compilation�catalog application�scope�pry

8

9 of 72

can't find puppet

9

$ puppet config print server

yourface

$ puppet config print confdir

/home/thomas/.puppetlabs/etc/puppet

​

# puppet config print server

puppet

# puppet config print ca_server

puppet

# puppet config print confdir

/etc/puppetlabs/puppet

https://goo.gl/8LyZzN

10 of 72

puppet agent Runs as root…

puppetserver Runs as puppet.

10

Basic UNIX permissions

11 of 72

11

https://goo.gl/8LyZzN

12 of 72

can't find puppet

12

dns

files

/etc/hosts

nsswitch.conf

# host puppet

Host puppet not found: 3(NXDOMAIN)

# nslookup puppet

Server: 10.0.2.3

Address: 10.0.2.3#53

​

** server can't find puppet: NXDOMAIN

# dig puppet

​

​

​

​

​

​

# getent hosts puppet

127.0.0.1 puppet.example.com puppet localhost localhost.localdomain localhost4 localhost4.localdomain4

# ping puppet

PING puppet.example.com (127.0.0.1) 56(84) bytes of data.

64 bytes from puppet.example.com (127.0.0.1): icmp_seq=1 ttl=64 time=0.035 ms

​

​

https://goo.gl/8LyZzN

13 of 72

can't connect to puppet

13

node

puppet

server

network

pixies

8140

server

masterport

https://goo.gl/8LyZzN

14 of 72

can't connect to puppet

Verify you can connect:

# nc -v puppet.example.com 8140

Ncat: Version 7.12 ( https://nmap.org/ncat )�Ncat: Connected to 192.168.1.1:8140

14

https://goo.gl/8LyZzN

15 of 72

can't connect to puppet

Q: How can you know you are talking to puppet?

​

A: Use the REST API

15

https://goo.gl/8LyZzN

16 of 72

can't connect to puppet

$ curl -k https://puppet.example.com:8140/puppet-ca/v1/certificate/ca?environment=production

-----BEGIN CERTIFICATE-----�MIIFhDCCA2ygAwIBAgIBATANBgkqhkiG9w0BAQsFADAoMSYwJAYDVQQDDB1QdXBw�ZXQgQ0E6IHB1cHBldC5leGFtcGxlLmNvbTAeFw0xNjEwMDgyMDEyNDRaFw0yMTEw�…�qsHOzPEQwJS5q1yPciBQ5jsKCB5vXJEdnPMhA5GBx9rp91hBsiWURlyWt0DMqMTq�VqcUEGNc267LrCzKJsFBHCNO53e8z2I0�-----END CERTIFICATE-----

16

masterport

ca_server

puppet or

puppet-ca

api endpoint

key

environment

https://docs.puppet.com/puppet/latest/reference/http_api/http_api_index.html

​

https://goo.gl/8LyZzN

17 of 72

can't connect to puppet

$ gnutls-cli --port 8140 puppet.example.com� --x509cafile=ca.pem

$ openssl s_client -connect puppet.example.com:8140

GET /path/to/thing/you/want

Accept: PSON/YAML/Text/???

17

puppet

master

(puppetserver)

node

HTTPS/8140

lsof -i :8140

mtr

traceroute

ping

https://goo.gl/8LyZzN

18 of 72

in order to authenticate...

you must first authenticate.

18

https://goo.gl/8LyZzN

19 of 72

can't connect to puppet

$ curl --cert certs/cottage.pem --key private_keys/cottage.pem.orig --cacert certs/ca.pem https://puppet.example.com:8140/puppet/v3/environments | jq

…� "environments": {� "production": {� "settings": {� "modulepath": [� "/etc/puppetlabs/code/environments/production/modules",� "/etc/puppetlabs/code/modules",� "/opt/puppetlabs/puppet/modules"� ],� "manifest": "/etc/puppetlabs/code/environments/production/manifests",� "environment_timeout": 0,� "config_version": ""

19

https://goo.gl/8LyZzN

20 of 72

API FTW

$ curl --cacert certs/ca.pem \

--cert certs/cottage.pem \

--key private_keys/cottage.pem \

-H 'Accept: pson' \

https://puppet.example.com:8140/puppet/v3/catalog/cottage?environment=production \

> cottage.pson

​

20

catalog

https://goo.gl/8LyZzN

21 of 72

PSON

Puppet JSON → PSON

$ jq <pson>

$ curl … | jq

sed for JSON

$ jq .environment catalog.pson

$ jq .classes catalog.pson

$ jq '.resources | length' catalog.pson

$ jq '.resources[] | select(.type == "Stage")' catalog.pson

21

JQ?

https://goo.gl/8LyZzN

22 of 72

PSON

Puppet JSON → PSON

$ jq <pson>

$ curl … | jq

sed for JSON

$ jq .environment catalog.pson

$ jq .classes catalog.pson

$ jq '.resources | length' catalog.pson

$ jq '.resources[] | select(.type == "Stage")' catalog.pson

22

JQ?

Sometimes it's good to go...

https://goo.gl/8LyZzN

23 of 72

problems

connection

certificate

catalog compilation�catalog application�scope�pry

23

24 of 72

can't get certificate

  • already signed (clean)
  • dates off - expired CA, expired cert

24

openssl x509 -in cert.pem -text

puppet cert fingerprint host.example.com

puppet cert print host.example.com

puppet cert clean host.example.com

ntpq -p

chronyc sources

rm /etc/puppetlabs/puppet/ssl/*/hostname*

https://goo.gl/8LyZzN

25 of 72

Intermission

25

x.509

the

crash

course

26 of 72

x.509

PKI

CA

SSL

TLS

CRL

REQ

RSA

DSA

DH

PKCS7

PKCS8

PKCS12

​

MD2

MD5

SHA1

SHA256

SHA384

SHA512

PEM

DER

OID

RFC

KS

TS

​

BOB

ALICE

26

"ENCRYPTION"

"ENCRYPTION"

https://goo.gl/8LyZzN

27 of 72

x.509

PKI

CA

SSL

TLS

CRL

REQ

RSA

DSA

DH

PKCS7

PKCS8

PKCS12

​

MD2

MD5

SHA1

SHA256

SHA384

SHA512

PEM

DER

OID

RFC

KS

TS

​

BOB

ALICE

27

https://goo.gl/8LyZzN

28 of 72

x.509

Public Key Infrastructure

Certificate Authority

Trust

28

https://www.flickr.com/photos/genvessel/431100596

https://goo.gl/8LyZzN

29 of 72

x.509

29

ca.pem

puppet

master

CERTIFICATE

node

CA?

ca.pem

server

ca_server

https://goo.gl/8LyZzN

30 of 72

x.509

30

node.crt

ca.crt

puppet

master

CERTIFICATE

node

node.csr

node.crt

server

ca_server

cert me pls?

https://goo.gl/8LyZzN

31 of 72

x.509

31

puppet.pem

ca.pem

puppet

master

CERTIFICATE

node

server

ca_server

https://goo.gl/8LyZzN

32 of 72

OpenSSL

View Certificate:

openssl x509 -in ca.pem -text -noout |less

​

View Fingerprint:

openssl x509 -in ca.pem -fingerprint -noout

openssl x509 -in ca.pem -fingerprint -noout -sha256

Verify Certificate:

openssl verify -CAfile ca.pem puppet.pem

32

Validity

X509v3 Key Usage

SHA1 (default)

https://goo.gl/8LyZzN

33 of 72

OpenSSL (Puppet shortcuts)

View Certificate:

puppet cert print ca

​

View Fingerprint:

puppet cert fingerprint ca

List Certificates:

puppet cert list -a

33

https://goo.gl/8LyZzN

34 of 72

CRL

34

node.crt

ca.crt

puppet

master

CERTIFICATE

node

crl?

crl

crl

https://goo.gl/8LyZzN

35 of 72

CRL

# openssl crl -in crl.pem -text -noout |grep Serial

Serial Number: 02

​

​

​

# openssl x509 -in cert.pem -text -noout |grep Serial

Serial Number: 3 (0x3)

​

35

https://goo.gl/8LyZzN

36 of 72

CRL

# cat ca_crt.pem ca_crl.pem >combined.pem

# openssl verify -CAfile combined.pem -crl_check cottage.pem

cottage.pem: OK

# puppet cert clean cottage

Notice: Revoked certificate with serial 3�Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/ca/signed/cottage.pem'�Notice: Removing file Puppet::SSL::Certificate cottage at 'puppet/ssl/certs/cottage.pem'

# cat ca_crl.pem ca_crt.pem >combined.pem

​

​

36

again?

# openssl verify -CAfile combined.pem -crl_check cottage.pem

cottage.pem: CN = cottage�error 23 at 0 depth lookup:certificate revoked

​

https://goo.gl/8LyZzN

37 of 72

OpenSSL

# openssl rsa -noout -modulus -in ca_key.pem |sha256sum�69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38

# openssl x509 -noout -modulus -in ca_crt.pem |sha256sum�69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38

# [ $(openssl rsa …) == [ $(openssl x509) ]

37

Modulus

n = pq

BIG *RSE PRIME

https://goo.gl/8LyZzN

38 of 72

OpenSSL

# openssl rsa -noout -modulus -in ca_key.pem |sha256sum�69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38

# openssl x509 -noout -modulus -in ca_crt.pem |sha256sum�69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38

38

Modulus

n = pq

BIG *RSE PRIME

$ puppet agent -t

Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.

Certificate fingerprint: D4:D3:76:F1:6B:51:83:3C:4B:72:69:BF:BC:B0:80:94:79:75:1A:3B:D8:29:F5:EF:81:2C:44:35:21:93:CE:FD

To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certificate.

On the master:

puppet cert clean cottage

On the agent:

1a. On most platforms: find /home/thomas/.puppetlabs/etc/puppet/ssl -name cottage.pem -delete

1b. On Windows: del "\home\thomas\.puppetlabs\etc\puppet\ssl\certs\cottage.pem" /f

2. puppet agent -t

The certificate retrieved from the master does not match the agent's private key.

https://goo.gl/8LyZzN

39 of 72

OpenSSL recap

  • x509�view certificate�check expiration�check serial number
  • crl�revoked cert serial#
  • verify�verify cert with CA and/or CRL
  • modulus�how the cert was encrypted

39

https://goo.gl/8LyZzN

40 of 72

Problem workers

40

node

puppet

server

puppet

server

load

balancer

devel

problem

puppet

server

problem

production

mod_proxy_balancer

https://goo.gl/8LyZzN

41 of 72

Problem workers

41

puppet

server

--logdest /var/log/puppetlabs/puppetserver/problem.log

--debug

--profile

logrotate

https://goo.gl/8LyZzN

42 of 72

problem/bugfixes environments

configure r10k to make environments automatically

environment.conf

  • modulepath (debug module) ← what is that?

42

https://goo.gl/8LyZzN

43 of 72

Compiling

43

$ sudo puppet master \

--compile problem.example.com \

--debug --trace \

--logdest /tmp/problem.puppet.log \

--environment sandbox

​

{� "data": {� "resources": [� {� "title": "main",� "exported": false,� "tags": ["stage"],� "type": "Stage",� "parameters": {� "before": "Stage[post]",

JSON

Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'bucketdir': 'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/bucket"}'�Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}'�Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}'�Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'fileserverconfig': 'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}'�

https://goo.gl/8LyZzN

44 of 72

Apply

44

$ sudo puppet apply \

--debug --trace \

--environment sandbox code.pp

​

$date = "+%S"

exec {'epoch':

command => "echo \$((`date $date` / 86400))",

path => '/bin:/usr/bin:/sbin:/usr/sbin',

}

Info: Applying configuration version '1415729233'

Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` / 86400))'

Debug: Executing 'echo $((`date +%S` / 86400))'

Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully

​

​

https://goo.gl/8LyZzN

45 of 72

problems

connection

certificate

catalog compilation

catalog application�scope�pry

45

46 of 72

catalogs

46

https://goo.gl/8LyZzN

47 of 72

catalog

fails to compile

  • duplicate resource
  • modulepath/bad module name

fails to apply

  • unpredictable exec
  • bad/broken service
  • bad/missing variable

47

48 of 72

fails to compile

48

https://goo.gl/8LyZzN

49 of 72

help yourself

function verb() {� erb -P -x -T '-' $1 | ruby -c� is_ok $?�}

function vyaml() {� ruby -ryaml -e "YAML.load_file '$1'"� is_ok $?�}

function ppv() {� puppet parser validate $*� is_ok $?�}

​

​

​

function vepp() {

puppet epp validate $*

is_ok $?�}

​

49

function is_ok() {

if [ $1 == 0 ]; then

if [[ "$TERM" == *"xterm"* ]] || \

[[ "$TERM" == *"vt100"* ]]; then

echo -e "\033[1mSyntax \033[32;1mOk!\033[0m"

else

echo "Syntax Ok!"

fi

else

return $1

fi

}

https://goo.gl/8LyZzN

50 of 72

duplicate resource

separate into subclass ( package {'httpd'} )

virtual resources ( @user, @package, @service)

modulepath

puppet config print modulepath

​

50

root@puppet:~# puppet config print modulepath --environment production

/etc/puppet/environments/production/public:/etc/puppet/environments/production/modules

root@puppet:~# puppet config print modulepath --environment master

/etc/puppet/modules:/usr/share/puppet/modules

https://goo.gl/8LyZzN

51 of 72

problems

connection

certificate

catalog compilation

catalog application

scope�pry

51

52 of 72

fails to apply

52

https://goo.gl/8LyZzN

53 of 72

Unpredictable exec

#!/bin/bash

​

echo $JAVA_HOME

53

/home/javadev/.bashrc

JAVA_HOME=/your/face

It works for me!

puppet runs as

puppet

https://goo.gl/8LyZzN

54 of 72

Broken Service

service provider

hasstatus => true

​

54

/sbin/service $service status

/etc/init.d/$service status

/usr/bin/systemctl is-active $service

https://goo.gl/8LyZzN

55 of 72

Bad/Missing Variable

$one = "1"

file {"pcone":

path => "/tmp/pc$one",

ensure => 'directory',

}

file {"pc1":

path => "/tmp/pc1",

ensure => 'file',

}

55

Info: Caching catalog for puppet.example.com

Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[pc1] to ["/tmp/pc1"] at /root/pc.pp:6; resource ["File", "/tmp/pc1"] already declared at /root/pc.pp:2 at /root/pc.pp:6:3 on node puppet.example.com

​

​

https://goo.gl/8LyZzN

56 of 72

Bad/Missing Variable

$PC = 'puppetconf'

pc {'one':� place => "/tmp/$PC",� type => "directory",� }

pc {'two':� place => "/tmp/$PC",� type => "file",� }

56

define pc (

String $place,

String $type,

) {

file {"$title":

path => $place,

ensure => $type,

}

}

Info: Caching catalog for puppet.example.com

Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Resource Statement, Cannot alias File[two] to ["/tmp/puppetconf"] at /root/define.pp:5; resource ["File", "/tmp/puppetconf"] already declared at /root/define.pp:5 at /root/define.pp:5:2 at /root/define.pp:15 on node puppet.example.com

​

​

https://goo.gl/8LyZzN

57 of 72

Printing - Notify

notify {"$variable": }

​

57

https://goo.gl/8LyZzN

58 of 72

chaining

​

notify {'something':

}->exec{'thingthatfails':

}->notify{'after': }

58

https://goo.gl/8LyZzN

59 of 72

Checking

exec{'before resolv.conf': � command => '/usr/local/bin/puppet-debug before resolv.conf',� require => Class['debug']�} -> file { '/etc/resolv.conf':� source => template("dns/resolv.conf"),� noop => true,�}

59

class debug { � file {'puppet-debug':� path => '/usr/local/bin/puppet-debug',� source => 'puppet:///modules/debug/puppet-debug',� mode => 0755,� }�}�

https://goo.gl/8LyZzN

60 of 72

Debug Script… just an example

#!/bin/bash

LOG=$(mktemp /tmp/puppet-debug.XXXXXX)

echo Puppet Debug -- $@ -- $(date) | tee $LOG

echo "-- Disk --" | tee -a $LOG

df -h |tee -a $LOG

df -i |tee -a $LOG

echo "-- Mem --" | tee -a $LOG

free | tee -a $LOG

echo "-- Files --" | tee -a $LOG

PUPPET=$(pgrep puppet)

for proc in $PUPPET

do

lsof -p $proc |tee -a $LOG

done

60

Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014

-- Disk --

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root

6.7G 2.5G 3.9G 39% /

tmpfs 246M 0 246M 0% /dev/shm

/dev/vda1 485M 80M 380M 18% /boot

Filesystem Inodes IUsed IFree IUse% Mounted on

/dev/mapper/VolGroup-lv_root

440640 79253 361387 18% /

tmpfs 62783 1 62782 1% /dev/shm

/dev/vda1 128016 50 127966 1% /boot

-- Mem --

total used free shared buffers cached

Mem: 502268 415488 86780 0 22176 172036

-/+ buffers/cache: 221276 280992

Swap: 835580 0 835580

-- Files --

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

puppet 2058 root cwd DIR 253,0 4096 14 /root

puppet 2058 root rtd DIR 253,0 4096 2 /

puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/ruby

puppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.so

puppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so

​

​

https://goo.gl/8LyZzN

61 of 72

Printing - Template

  • scope.to_hash
  • reject a few
  • sort
  • print, one per line

61

file { "/tmp/puppet-debug.txt":� content => inline_template("<% vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>\n<% end %>"),� }�

vars = scope.to_hash.reject

{ |k,v| !( k.is_a?(String) &&

v.is_a?(String) ) };

​

vars.sort.each do |k,v|

k=v\n

end

https://goo.gl/8LyZzN

62 of 72

Printing - Template

62

_timestamp=2014-10-23 22:29:52 -0700

architecture=x86_64

augeasversion=1.0.0

bios_release_date=01/01/2011

bios_vendor=Bochs

bios_version=Bochs

blockdevice_vda_size=8589934592

blockdevice_vda_vendor=6900

blockdevices=vda

caller_module_name=

clientcert=cookbook.example.com

clientnoop=false

clientversion=3.7.1

concat_basedir=/var/lib/puppet/concat

domain=example.com

environment=production

facterversion=2.2.0

filesystems=ext4,iso9660

fqdn=cookbook.example.com

gid=root

hardwareisa=x86_64

hardwaremodel=x86_64

hostname=cookbook

id=root

interfaces=eth0,lo

​

​

https://goo.gl/8LyZzN

63 of 72

problems

connection

certificate�catalog compilation�catalog application�scope

pry

63

64 of 72

Scope

The scene:

  • roles and profiles
  • ntp server

64

class role::ntp {

include ntp

}

class ntp {

include ntp::server

}

https://goo.gl/8LyZzN

65 of 72

Scope

The solution:

  • fully scope everything
  • remember scope

65

class role::ntp {

include ::ntp

}

class ntp {

include ntp::server

}

https://goo.gl/8LyZzN

66 of 72

problems

connection

certificate�catalog compilation�catalog application�scope

pry

66

67 of 72

pry

IRB replacement

REPL

available at runtime

67

pryrepl.org

require 'pry'

…

# amazing code here

# wow, much amaze

…

binding.pry

…

https://goo.gl/8LyZzN

68 of 72

pry demo

68

module Puppet::Parser::Functions

newfunction(:pry) do |args|

require 'pry'

binding.pry

end

end

modules/pry/lib/puppet/parser/functions/pry.rb

​

node default {

…

pry()

…

}

manifests/site.pp

​

https://goo.gl/8LyZzN

69 of 72

pry demo

#

#

69

… �From:�/etc/puppetlabs/code/environments/production/modules/pry/lib/puppet/parser/functions/pry.rb @ line 4�#<Module:0xfb588d1>#real_function_pry:� 2: newfunction(:pry) do |args|� 3: require 'pry'� => 4: binding.pry� 5: end�[1] pry(#<Puppet::Parser::Scope>)>

puppet agent -t

puppetserver foreground

… �Puppet Server has successfully started and is now ready to handle requests

exit

… �Puppet Compiled Catalog for xxx.example.com in y.z seconds

Info: Caching catalog for xxx.example.com�Info: Applying configuration version 'XXX'

https://goo.gl/8LyZzN

70 of 72

where to go for help

  • IRC #puppet / #puppet-dev�
  • slack puppetcommunity.slack.com�#pug�#puppet�
  • google group / mail list�https://groups.google.com/forum/#!forum/puppet-users�
  • PUG�https://www.meetup.com/Seattle-Puppet-Meetup/

70

https://goo.gl/8LyZzN

71 of 72

Summary

Puppet is an HTTPS service

End-to-end (gethostbyname, nc mtr)

OpenSSL is your friend (x509,crl, verify, s_client

make a debug class

remember scope

71

basic UNIX permissions

https://goo.gl/8LyZzN

72 of 72

72