Bash for production systems

Survival guide for quality scripts

Adrien Mahieux - Sysadmin++ (Performance Engineer)

gh: github.com/Saruspete

tw: @Saruspete

gm: adrien.mahieux@gmail.com

$(whoami)

Adrien Mahieux - @Saruspete�French pun, because I constantly grumble...

Work in highly critical environment…�Ok, maybe not that critical. No life will be lost if we reboot the wrong server.

My prod is not your prod�If you think you can handle it, we hire !

Time^W Latency is money�You don’t have time to rewrite everything ? You certainly have even less time when it’s broken at the worst moment.

​

Why this guide ?�- Because I always forgot how to do some snippets�- Sharing is caring�- Snippets and samples are easy to read / understand / copy�- Understand use-cases before using them

What to do with it ?�- It’s not a tutorial. More like examples of safe ways to write usual patterns.�- help you do enlightened choices. If you have all the cards in hand, you’re the best to know what choice to make�- share with your friends & coworkers.�- stop whining "shell scripts are unreadable", and rewrite them cleanly.�- If your script is unreadable, you don’t know bash enough (or you shouldn’t have used it at first).

2

Different shells, different features

3

Why and When to use bash ?

Define your use-case

• Commercial Support ?�Accountability and robustness�
• Supported systems ?�Wide or short list, tests, versions, duration of support...�
• Why shell script ?�Other languages available�
• bash != GNU != POSIX�Check support with your target

List the features you’ll need

• Required complexity ?�Shell is bad at handling complex parsing and long texts�
• Interactions with other binaries ?�Shell is the glue between processes but parsing is tricky.�
• Are shell features enough ?�May require a lot of external tools to process content as required

4

Others shells available

5

What’s the target OS ?

​

6

"/usr/bin/env bash" to the rescue (looks in $PATH).

A word on the shebang...

7

Bash is only one of the many "shell command interpreter". But it’s the default one on GNU/Linux, which is the most known and deployed *nix distribution.

​

Consider POSIX standard if your script is simple enough. This will ease portability.

​

The choice of the correct shebang is depending on target usage:

#!/bin/bash

• If your script is not meant to be re-used in an untested distribution (eg, part of commercial product with strict list of supported OS)
• If your script is tightly tied to the target system, and you don’t want to rely on the $PATH value.

#!/usr/bin/env bash

• If your script will be open-source in some way, you should ease the work of all contributors, whatever their OS.
• If your script is more for users than system, it will help

POSIX or bash ?

Features & keywords not in POSIX. Mostly coming from KSH.�Full list with introduction version: wiki.bash-hackers.org/scripting/bashchanges

• Arrays and associative arrays (bash-4)
• [[ extended operator
• Replacement ${var//search/replace} or substring ${var:offset:length}
• function f { }
• declare / typeset / type
• Arithmetic for loop "for ((i=0; i<10; i++)); do … ; done"
• extended glob
• $RANDOM $BASH_*
• /dev/tcp/$host/$port
• Named pipes
• Mapfile / readarray

8

Differences

[ is a binary & a built-in. �[[ is a keyword.

function f {}�f() { }�function f() {}

$0�$FUNCNAME�$LINENO�

${PIPESTATUS[@]}

[ is using arguments. No smart parsing�[[ is more lisible, faster and smarter

function does not exists in POSIX�"function f ()" only works in bash

In ksh, $0 behaves differently within “function f” and “f()”�In bash, use $FUNCNAME

The pipestatus is an array with the return code of all commands in a pipeline. So "$?" Is equivalent to "${PIPESTATUS[-1]}"

9

Differences - built in VS binary

$ time ps � PID TTY TIME CMD� 4345 pts/16 00:00:00 ps�30198 pts/16 00:00:00 bash

real 0m0.009s�user 0m0.004s�sys 0m0.004s

​

$ /usr/bin/time ps� PID TTY TIME CMD� 4349 pts/16 00:00:00 time� 4350 pts/16 00:00:00 ps�30198 pts/16 00:00:00 bash

0.00user 0.00system 0:00.00elapsed 85%CPU (0avgtext+0avgdata 2140maxresident)k�0inputs+0outputs (0major+110minor)pagefaults 0swaps

10

Differences - Variable visibility

v1="global"�v2="global"�f1 () { � v1="func"� typeset v2="func" v3="func"�} ; f1�echo "v1=$v1 v2=$v2 v3=$v3"�v1=func v2=global v3=

recurse() (� [[ $1 -le 0 ]] && return� v2="$1 $(recurse $(($1 -1)))"� echo "$v2"�)�recurse 3 ; echo "$v2"�3 2 1�global

POSIX: all vars are global (even those defined inside a function)

bash: With keyword, kept local inside a function, even if overlap a global var.

ksh: differs between “function f” and “f()” : �- function f : local, like bash�- f() : all global, like posix

Recursion ? use function parameters, or use ( ) instead of { } to declare the function body

11

Snippets and features

12

12

The magic header

Shebang (change according to your use)

All vars must be declared�All erasing stream must be explicit

Reset the lang to avoid localization�Reset the path to a secure one�More useful debugging prefix than “+”

Always work with absolute dirs ! Your script can be called from anywhere.

If GNU readlink is not available, use : �"MYPATH="$(cd -- "$(dirname -- "$0")" && pwd -P)"

Prefer “typeset” over “declare” for KSH�None will work on sh / ash / dash

#!/usr/bin/env bash

set -o nounset # or "set -u"�set -o noclobber # or "set -C"

export LC_ALL=C�export PATH="/bin:/sbin:/usr/bin:/usr/sbin:$PATH"�PS4=' ${BASH_SOURCE##*/}:$LINENO ${FUNCNAME:-main}) '

readonly MYSELF="$(readlink -f $0)"�readonly MYPATH="${MYSELF%/*}"

# If you need a temporary folder you can use:�#readonly MYTEMP="$MYPATH/temp/run.$$"�#mkdir -p "$MYTEMP"�#trap '(rm -fr $MYTEMP)' TERM QUIT INT

13

The magic header - implications

All variables accessed must be defined, even through tests. So you must provide a default value if you want to test an unknown variable: ${varname:-defaultvalue}

​

​

�With noclobber, bash will refuse to truncate a file if you don’t explicitly ask so with >|

set -o nounset�[[ -n "${1:-}" ]]

# For array test, disable it in subshell:�if (set +u; [[ -n "${ARR[key]}" ]]); then� echo 'Key exists'�fi

# And test if array is not empty:�if [[ -n "${ARR+${ARR[@]}" ]]

�set -o noclobber�echo "Start instance" >| file.log

​

14

Variable assignment - usual rules

typeset VAR1="Can by used anywhere"

�ftest () {� typeset tmpval="only used localy"� echo "$VAR1 != $tmpval"�}

# Also, declare vars in loop�function listpath {� typeset dir� for dir in ${PATH//:/ }; do� ls -al "$dir"� done�}

UPPERCASE variables are global to the whole script

lowercase variables are local to a function.

This won’t change the variable visibility, but you’ll have a hint on whether you should use it or not.

This is a commonly accepted rule, but not a language requirement. You should adhere to it anyway.

15

Variable assignment - String

hostn="$(uname -n)"�strdyn="value with spaces on $hostn"��strraw='value with $special chars /tmp/*'��

�strcat='The quick brown fox'�strcat+="Jumps over the lazy dog"

strraw="Multiline"$'\n\t'"Example"

Always enclose interpretable strings with double quotes !

Always enclose non-interpretable (literal) strings with single quotes

�Concatenation is also way easier with bash "+=" syntax.

$' … ' will prevent all parsing, except \n \t and \xnn (for bytes in hex)

16

Variable assignment - String Quoting

hostn="$(uname -n)"�strdyn="value with spaces on $hostn"�strraw='value with $special chars /tmp/*'�

echo $strdyn�value with spaces on odin��echo $strraw�value with $special chars /tmp/pulse-PKdhtXMmr1 /tmp/runtime-adrien /tmp/ssh-OwoBLZD6leEN

echo "$strraw"�value with $special chars /tmp/*

Let’s take our previous strings and focus on the quoting...�

If you don’t put double quotes between echo, it may not change the output…

But if your string has any special char, it will hurt you… badly… because they will get interpreted, then passed on to echo

So, ALWAYS double quotes your variables. Especially if they are strings.

17

Variable assignment - Integer

typeset -i i=5

i="hello"�echo $i�i=0�

i+=2

If you declare a variable as an int, it will save you from mis-usage or bad values

�

And also ease your life with += operator

18

Variable assignment - Integer bases

typeset -i i=5�i=080�bash: 080: value too great for base

echo $((16#A))�10

echo $((36#helloworld))�1767707668033969�

echo $((16#deadbeef))�3735928559�printf "%x" 3735928559�deadbeef

But beware of 0 as octal prefix... (may happen if using “date +%j”)�

​

You can also convert any base (up to 64 with charset: a-z,A-Z,0-9,@_) to base 10�

As expected works back and forth

19

Variable assignment - nameref (4.3+)

nameref are kind of a pointer. It will avoid you a lot of troubles with “eval”. The nameref is assigned during the typeset.

Then, you can manipulate directly the targeted variable with a common name.

If you modify the nameref, you’ll modify the referenced variable (not the nameref)

If you want to change which variable is referenced, you have to use typeset.

Same for unset: to unset the nameref (and not the referenced var) use “-n”.

typeset foo="bar"�typeset -i num=2�typeset foo2="barbaz"�typeset -n ref="foo" # bash 4.3+

echo "The referenced var = $ref"�The referenced var = bar

ref="foo$num"�echo "And now, foo=$foo"�And now, foo=foo2

typeset -n ref="foo$num"�echo "This time, foo$num=$ref"�This time, foo2=barbaz

unset -n ref

20

String expansion

rm /tmp/$prefix.$$.*�

​

shopt -s globstar�ls /proc/**�bash: /bin/ls: Argument list too long�

�sudo ls /root/.ssh/id_*�ls: cannot access /root/.ssh/id_*: No such file or directory�

​

Expansion (var, list, star, completion…) is done by the shell, not the application.

�If too many files matches a pattern, you’ll have error “too many arguments”

�If your shell doesn’t have access to read files and do the expansion, you’ll have an error, and the pattern will be provided as is (unless you set globfail or nullglob option)

21

String Heredoc

$ cat >/tmp/toto <<EOT�Usage: $0 <required> [optional]�This is a multiline text�EOT

$ cat <<-EOT� Using "<<-", the tabs at start of this text� are stripped, so you can keep indent� EOT

$ cat << 'EOT'�This text won't be taken as a litteral.�No $variable or $(cat /etc/passwd) will expand�EOT

$ (echo 'cat << _EOT'; cat file.cfg; echo '_EOT')|/bin/sh

22

Heredoc is very useful to put large block of text inside a script, like a configuration template, a help section… It’s a stream, so uses stdin/out, not arguments.

​

Why use tabs instead of space for indenting (aside the fact it’s their primary usage) ? Because bash can strip the tabs on this heredoc, not spaces.

​

�As usual, if you don’t want any expansion to happen, use single quotes between your separator keyword

​

​

You may also use heredoc as a cheap environment variable and command processing inside a configuration file for your application (but shouldn’t)

Process - Execution

cat /proc/self/mounts

usrs="`getent passwd|awk -F: '{print $1}'`"�grps="$(getent group|awk -F: '{print $1}')"

​

alias ls="ls --color"�\ls

​

cd() { � builtin cd "$@" && echo "$PWD"�}

​

To execute a cmd, just call it…

To capture its output, you can use `cmd` or $(cmd). But for sake of clarity, always use $(cmd) format.

Even when overridden, you can still call the binary by prefixing \

�But if you want a builtin, just prefix the command by "builtin"

​

23

Process - Background processes

tail -f /var/log/messages >/dev/null &�disown

for i in {1..10}; do� sleep $i &�done

wait $!

wait

​

You can run cmds in background...�and forget about them (daemon like)

Or launch multiple applications in background…

And only wait for the last one you launched

Or wait for all of them

​

24

Process - Job specification

for i in {1..9}; do� sleep 10$i &�done

jobs�[1] Running sleep 10$i &�[2] Running sleep 10$i &�[ . . . ]�[7] Running sleep 10$i &�[8]- Running sleep 10$i &�[9]+ Running sleep 10$i &

�kill %5

fg %-

Lets start a few commands in background.

And you can see their jobspec ID and status using “jobs”

%X : where X is the jobspec ID�%+ : (current job) the command you act on if you don’t specify a jobspec ID.�%- : one-before last command.

Builtin like wait, disown, kill, fg, bg… understand the % as a jobspec id.

25

Process - Signal handling

readonly TMPDIR="/tmp/data.$$"�mkdir -p "$TMPDIR"

trap INT QUIT TERM cleanup�trap USR1 status

cleanup () {� \rm -rf "$TMPDIR"�}

status () {� echo "Current TMPDIR: $TMPDIR"�}

Use “trap” to catch signals like Ctrl+C or “kill”.

You can list available signals with “trap -l”, but don’t trap them if you don’t know what they’re doing !

26

Process - Coprocess

Don’t.

Just. Fuckin’. Don’t.

If you need coprocess, you need another scripting language, like python or perl.

EXCEPTION

if you are Clifford Williams level and can produce scripts like the one pasted here. But in that case, you won’t learn anything new within this document

27

#!/usr/bin/env ksh93

#Author: G. Clifford Williams

#email: gcw-ksh93@notadiscussion.com

#Purpose: This is a simple producer/consumer example written in KSH93 to

# demonstrate parallel execution with discreet I/O.

#

#Note: This example script requires KSH93. It will not work with BASH, PDKSH,

# MKSH, ZSH, KSH88, etc. Further it incorporates features foun in version r+

# and later. KSH93 u+ has been around for 5 years and is the version I used

# but your mileage may vary.

​

typeset -a inputs=()

typeset -a outputs=()

set +o bgnice

​

for counter in {A..F}{0..9} ; do

#create a background job that has output on it's STDOUT at a random interval

{ while true ; do

#print -n "pump ${counter}: $(date)" >> ${my_file}

integer sleep_time=$(( $(( $RANDOM + 1 )) / 32768.0 * 5 ))

printf "pump %s/%i: %(%H:%M:%S)T\n" ${counter} ${sleep_time} now

sleep ${sleep_time}

done

}|&

# The '|&' above is how ksh launches a co-process in the background with

# bi-directional I/O. We wrap everything in a command-list ({;}) for clarity

#The bit below can seem confusing. The co-process produces output which is

#input to the consumer. We aren't sending any messages to the co-process in

#this case but if we were our output would be the input to the co-process.

#The [in,out](put) variables are named from the perspective of the parent.

exec {in}<&p #store the output fdesc in $in

inputs+=( ${in} ) #append the value in $in to the $inputs list

exec {out}>&p #store the input fdesc in $out

outputs+=( ${out} ) #append the value in $out to the $outputs list

done

​

while true; do #perpetual loop

for counter in ${inputs[*]}; do #iterate through the list of input fdescs

unset line_in #clear our holder variable

read -t0.1 -u${counter} line_in #timeout after .1 seconds of no input

if [[ -z "${line_in}" ]] ; then

print "${counter}: timed out"

else

print "${counter}: ${line_in}"

fi

done

done

​

Tests - “if test1 / else” VS “test1 && { } || { }”

# if/elif/else/fi are real test keywords�# the result of the test in them are the only�# way to act on a branch. The result inside�# their block doesn’t change the validity of�# the differents branches.�

if grep root /etc/passwd > /dev/null; then� [[ -e /nonexistantfile ]]�else� echo "This should not be printed"�fi

# Nothing is displayed

# { ... } is a ‘group command’. �# its last command will be the return code�# of the whole group.�# By using a || or && after, bash will act�# on this final return code, not just the�# test at the beginning.

grep root /etc/passwd > /dev/null && {� [[ -e /nonexistantfile ]]�} || {� echo "This should not be printed" �}

# Display "This should not be printed"

28

Variable tests - [ or [[

29

# In pure bash, always use [[

typeset -i foo=15�[[ $foo -gt 7 ]]�[[ $foo > 7 ]]

[[ $foo = 15 && $foo < 30 ]]

​

# Finding a string�haystack='spaces and * $$ >'�needle='and'�[[ "$haystack" = *$needle* ]]

​

# same, but case insensitive�shopt -s nocasematch�[[ "$haystack" = *$needle* ]]

​

# Use [ for POSIX compatibility�foo=15�[ "$foo" -gt 7 ]�[ $(($foo > 7)) -ne 0 ]�[ "$foo" = 7 ] && [ "$foo" -lt 30 ]

​

# You’ll need to use grep or a switch�case $haystack in� *and*) echo 'found !' ;;�esac

​

# Could also be done with "grep -i"

case $haystack in� *[aA][nN][dD]*) echo 'found !' ;;�esac

​

# Beware of [ and keywords, like ‘>’�[ "$foo" > 7 ] # will create file '7'

Variable tests

30

Variable default value

31

# You can use :+ for options mapping between options and external tools�ratio="500"�crop=""�myimagetool ${ratio:+--ratio=$ratio} ${crop:+--crop=$crop} file.png

Variable modification

32

v="/path/to/prefix_file.txt"

Variable modification

33

s="hello World fooBAR"

Variable modification - arrays

typeset -a ifaces=($(find /sys/class/net/ -type l))�echo ${ifaces[@]}�/sys/class/net/lo /sys/class/net/enp0s31f6 /sys/class/net/wwp0s20f0u6 /sys/class/net/wlp4s0 /sys/class/net/virbr0

echo ${ifaces[@]##*/}�lo enp0s31f6 wwp0s20f0u6 wlp4s0 virbr0

​

typeset -a opts=("hello" "world")�echo cmd ${opts[@]/#/--text=}�cmd --text=hello --text=world

​

34

You can act on all values in an array too.

​

​

Decomposing it:�ifaces[@] : all values�## : Remove all matches�*/ : Pattern to remove�

For all values of array opts, replace from begining ("/#") all empty values (no pattern given) by "--text=" (text after the 2nd "/")

Variable expansion - Position parameters

35

./test.sh "A1" "A2" "A3.1 A3.2"

​

for v in $@; do

echo "($v)"

done

99% of the time, you’ll want to use “$@” to keep intact your elements with spaces within.

​

Without quotes, $@ and $* have same behaviour.

​

The only caveat comes with "$*" : It use the first char of $IFS as separator (which happens to be space by default).

​

​

​

poorManCsv() {� IFS=';'� echo "$*"�}

poorManCsv hello world "1 2 3"�Hello;world;1 2 3

Variable expansion - On array

usesOnlyArgs2and3 () { � for v in "${@:2:2}"; do� echo "($v)"� done�}

usesOnlyArgs2and3 v1 v2 "v3.1 v3.2" v4�(v2)�(v3.1 v3.2)

36

If you apply the substring format to an array (${@:x:y} or ${t[@]:x:y}), instead of doing a substring, it will select Y elements of the array starting from element number X

​

Beware: when using an array, elements IDs starts at 0.�When using $@, $@[0] is equivalent to $0 which is the name of the script. So elements ID are shifted by 1.

Brace expansion

$ echo prefix-{x,y,z}-suffix�prefix-x-suffix prefix-y-suffix prefix-z-suffix

$ echo mv /path/to/file{,.bak}�mv /path/to/file /path/to/file.bak

$ echo mkdir -p "/home/www/"{bin,cron,html,{conf,logs,priv}/{nginx,phpfpm}}�mkdir -p /home/www/bin /home/www/cron /home/www/html�/home/www/conf/nginx /home/www/conf/phpfpm�/home/www/logs/nginx /home/www/logs/phpfpm�/home/www/priv/nginx /home/www/priv/phpfpm

# But beware, chars are expanded from their ASCII code... �$ echo {d..Z}�d c b a ` _ ^ ] [ Z Y X

37

Array - Indexed

typeset -a t

# Push (ksh way):�t[${#t[@]}]="don't"�t[${#t[@]}]="use this"�t[${#t[@]}]="syntax"

# Push (bash style)�t+=("Prefer this")�t+=("Much cleaner")�t+=("Bash syntax")

# Count:�${#t[@]}

​

# Iterate over values�for value in "${t[@]}"; do :; done

# Iterate over values using index�for key in "${!t[@]}"; do� typeset value="${t[$key]}"�done

# Replace over an array�t[0]="hello world"�t[1]="hi there"�for i in "${t[@]//h/z}"; do echo "$i"; done�zello zorld�hi zhere

38

Array - Associative

Same as indexed arrays but declared using "typeset -A"

Available on bash4 and ksh88+

Automatically ordered on ksh

Can expand and filter keys with ${!pattern}

typeset -A t

# Like a foreach�for key in ${!t[@]}; do� val="${t[$key]}"�done

# Can also autocomplete indexes�echo "${!BASH@}"�BASH BASHOPTS BASHPID BASH_ALIASES BASH_ARGC BASH_ARGV BASH_CMDS BASH_COMMAND BASH_COMPLETION_VERSINFO BASH_LINENO BASH_SOURCE BASH_SUBSHELL BASH_VERSINFO BASH_VERSION

unset ${!PYTHON@}

39

Array - mapfile / readarray

# Read file.txt and put lines in ARR�readarray ARR < file.txt

# read only 5 lines after the 2nd line�readarray -s 2 -n 5 ARR < /etc/passwd

# There is also a callback, to be called�# every X quantum�cb () { � num="$1"� str="${2%?}"� echo "$num = $str"�}�mapfile -C 'cb' -c 1 ARR < /etc/passwd

40

Easily map a stream into an array

​

It’s also easier than play with tail + head.

​

As the line ($2) contains also the separator (\n here), I’m removing it with "${2%?}" as it’s the last char.

Loops - iterate over elements

words="hello world foo bar"�for w in $words; do� echo "($w)"�done

typeset -a words=("hello world" "foo bar")�for w in "${words[@]}"; do� echo "($w)"�done

for file in /var/tmp/*; do� echo "==== $file: $(< $file)"�done

for i in {1..10}; do� echo "Im the $i"�done

41

For loop are used to iterate over a list of words (based on $IFS).

​

​

If you have a word that has an $IFS inside, it’ll be split in two. Double quotes won’t work: “$word” will be seen as a single element: use arrays

​

If you want to list files, don’t call for “ls”, simply use patterns

​

​

Loops - while read

while IFS=: read login pass uid gid gecos home shell _canary; do� [[ -n "$_canary" ]] && {� echo "Line for user $login is malformated (canary: $_canary)"� continue� }� echo "$login - $uid - $home" �done < /etc/passwd

42

• The last var in the list will contain all remaining words (useful for reading the whole line with “while read line”).
• If you need to change IFS, put it just after the "while", like an environment variable set only to “read”, so it will not "contaminate" the loop, nor after.
• If you have a fixed number of elements, add a “canary” (_junk) at the end. If it not empty, that means you had more fields than awaited, and some vars are potentially incorrect.

Loops - while read + var assignation inside

typeset -A users�awk -F: '$3>100{print $1,$5}' /etc/passwd | while read login gecos; do� users[$login]="$gecos"�done�for login in ${!users[@]}; do� echo "$login = ${users[$login]}"�done

43

Solution 1 : Bashism

​

while read login gecos; do� users[$login]="$gecos"�done < <(awk -F: '$3>100{print $1,$5}' /etc/passwd)

Solution 2 : Set hack

​

set +m

shopt -o lastpipe

Won’t work : pipe will execute left expression in a subshell, not propagating variables to parent shell.

Streams - redirection

cat file1 | sort > /tmp/$$.f1�cat file2 | sort > /tmp/$$.f2�vimdiff /tmp/$$.f1 /tmp/$$.f2�rm /tmp/$$.f1 /tmp/$$.f2

​

​

# Is equivalent to :�vimdiff <(cat file1 | sort) <(cat file2 | sort)

When using tools that requires files to work on (eg, diff 2 content), but the output is from commands, you need to use temporary files.

​

With bash, no need to use temporary files, it’ll provide the stdout’ fd to other commands

44

Streams - redirection

# Prefix the stderr redirection�echo >&2 "This is an error message"

# Redirect stderr to stdout�exec 2>&1

# Or close them all before daemonizing�exec 0>&- 1>&- 2>&-

# You can create new out FD�exec 99>/tmp/debug.log

# But beware of the order!�exec 2>&1 1>/dev/null

# Is different from �exec 1>/dev/null 2>&1

Outputs the message to stderr. Useful for logging

You can also redirect streams while in the script. Close them with "&-"

If you try to write to a closed fd, you’ll have the error “Bad file descriptor”

Redirections are using dup() to duplicate a stream, so calling order is critical !

If you send a stream to a target(2>&1) then close the target (1>/dev/null) 2 will still have a fd opened to original target.

45

Streams - Embedded TCP client

# No telnet ? bash can act like one...

# Open the socket as a stream�exec 5<> /dev/tcp/$server/$port

# Write something to it�echo -e "HEAD / HTTP/1.0\nHost: ${server}\n\n" >&5

# and read from it�cat 0<&5

# When finished, close it�exec 5>&-

Virtual folder /dev/tcp is handled by bash.

You can use it as a standard stream.

Then, a lot of tools can be rewritten as pure bash, given you implement the protocol :)

46

Debugging

# Combine the topics we saw

# A simple log to stderr�# Put the >&2 before the text for visual alignment �echo >&2 "Starting debugging"

# The debugging prefix when 'set -x' is enabled. Default = “+”�# The 1st char of PS4 will be repeated to show process depth (here, a space)�PS4=' ${BASH_SOURCE##*/}:$LINENO ${FUNCNAME[0]:-main}) '

# Push the debug log in a dedicated file�exec 99>$0.dbg�BASH_XTRACEFD=99

# And activate debug�set -x

47

Debugging - Stackdump

Excerpt from https://github.com/Saruspete/ammlib/blob/5ae299a/ammlib#L902-L933

function Stackdump {

typeset -i skip=${1:-1}

typeset -i cnt=${2:-255}

​

# Required for BASH_ARGV

typeset extdbg="$(shopt extdebug)"

extdbg="${extdbg##*$'\t'}"

typeset -i i j argoff=0

for i in ${!BASH_SOURCE[@]}; do

if [[ $i -lt $skip ]]; then

argoff+=${BASH_ARGC[$i]:-0}

continue

fi

[[ $i -ge $(($skip + $cnt)) ]] && break

echo -n "${BASH_SOURCE[$i]##*/}::${FUNCNAME[$i]}::${BASH_LINENO[$(($i-1))]} "

echo

argoff+=${BASH_ARGC[$i]:-0}

done

}

48

# @description: Display the stackdump of current script

# @arg $1 (int) Stack levels to skip. Default 1 (= skip this function)

# @arg $2 (int) Max levels to return. Default 255

​

Flags to change bash behavior

49

49

Shell Options - stricture

All variables used must be defined first

Do not truncate an existing & non empty file

Do not execute any command (check syntax)

Exit on first return code != 0 (beware!)

Monitor processes

Return code will be the one of the first leftmost failing (non-0) program in the pipeline, instead of only the last (rightmost) called in pipeline.

​

set -o nounset # set -u

set -o noclobber # set -C

set -o noexec # set -n

set -o errexit # set -e

set -o monitor # set -m

set -o pipefail��

​

50

Shell Options - debug

set -o monitor # set -m

set -o verbose # set -v

set -o xtrace # set -x

​

Monitor processes

Show content of code being processed

Enable cross-calls debug (execution and tests): function arguments are in BASH_ARGC

​

51

Shell Options - nullglob

$ for f in /etc/toto*; do� echo "== $f"�done�== /etc/toto*�$

​

​

$ shopt -s nullglob�$ for f in /etc/toto*; do� echo "== $f"�done�$

​

By default, if a pattern doesn’t match any file, the shell will use the pattern itself as a literal string. Then, the for loop will use this literal string as if it matched a filed called “/etc/toto*”, which does not exists.

​

By enabling nullglob, if a pattern doesn’t match any file, it will expand to null (empty), so the loop is never entered.

​

52

Shell Options - extglob / dotglob / globstar

$ shopt -s extglob

+( ) @( ) !( ) *( )

$ ls -d *config�ls: cannot access *config: No such file or directory

$ shopt -s dotglob

$ ls -d *config�.config �.gitconfig

$ ls /proc/$$/**�/proc/

$ shopt -s globstar�$ ls /proc/$$/**�

53

Shell Options - nice for interactive shell

shopt -s autocd�shopt -s dirspell�shopt -s direxpand�shopt -s histappend

set -o notify # set -b

��PROMPT_COMMAND="history -a;history -r"�

54

​

​

Report the status of terminated background job immediately, rather than waiting for next prompt.

This will sync the history between multiple instances (zsh-like)

Special Variables

55

Bonus - grep and awk useful snippets

56

56

grep useful snippets

# Avoid grep in process list�ps aux | grep process | grep -v grep

# Escape some regex-related chars�echo "Hello+.+" | grep '\.+'

# Grep in shell scripts�find -name '*.sh' -exec grep DOH {} \;

# What about "Useless Use Of cat" ?�cat bar.txt | grep foo

# And what about couting ?�grep ^processor /proc/cpuinfo | wc -l

# This one self-excludes�ps aux | grep [p]rocess

# Use -F to avoid any regex�echo "Hello+.+" | grep -F '.+'

# Grep can do recursion itself�grep -r --include '*.sh' DOH

# Just call grep�grep foo bar.txt

# Same thing : only grep�grep -c ^processor /proc/cpuinfo

​

57

awk useful snippets

# Awk is often reduced to "print $1"..�grep foo bar.txt | awk '{print $1}'

# Instead of multiple programs�cut -d: -f1 /etc/passwd |grep -v ^root

# Operations can be complex...�typeset -i s=0�for i in $(cut -d: -f3 /etc/passwd);do� s+=i�done�echo $s�

# But awk is self-sufficient for grep�awk '/foo/{ print $1 }' bar.txt

# simply change the separator�awk -F: '$1!="root"{print $1}' /etc/passwd

# while summing in awk is easy too !�awk -F: '{s+=$3} END{print s}' /etc/passwd

58

Tools and Bibliography

59

59

Tools

AMM-Lib : Bash dynamic library to manage Linux systems and common operations�https://github.com/Saruspete/ammlib/ please contribute :-)

ShellCheck : static analysis (linter) for shell code.�https://www.shellcheck.net/ / https://github.com/koalaman/shellcheck

Checkbashims : Check your script for elements not available in posix flavour.�https://packages.debian.org/devscripts

�Google Doc source of this document (comments open) : https://docs.google.com/presentation/d/1a4IAux4tNo7F7mQ6fbzIVPEHxQQ0buD15Cm8vSMJFb0/edit?usp=sharing

60

Bibliography

POSIX Shell: http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html �POSIX.1-2017: http://pubs.opengroup.org/onlinepubs/9699919799/ �Bash POSIX mode: https://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html�Bashims: http://mywiki.wooledge.org/Bashism�Bash reference sheet: http://mywiki.wooledge.org/BashSheet �POSIX Tricks: http://www.etalabs.net/sh_tricks.html �Operators: https://unix.stackexchange.com/questions/306111/what-is-the-difference-between-the-bash-operators-vs-vs-vs�ksh/bash differences: https://www.dartmouth.edu/~rc/classes/ksh/print_pages.shtml �Bash pitfalls: http://mywiki.wooledge.org/BashPitfalls�Options parsing: https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash �

​

61

Many thanks for their contribution

Benjamin Riou�Aurelien Rougemont - Beorn�Graham Christensen - grhmc�G. Clifford Williams - gcw@notadiscussion.com

62