Intro to Linux Administration
Welcome to Null Space Labs!
– Electronics�– Programming�– Hacking/Security/Lockpicking�– Networking�– Gaming
Linux Bootcamp Agenda
Class 1
Intro to the command line & BASH� Understanding the filesystem and managing files� Input/Output between commands and files� Using system documentation� Installing new software
Class 2
Class 3
Linux Bootcamp Agenda
Class 1
Class 2
Advanced file topics� Advanced shell topics� User and group management� Process management, scheduling, services� System logs
Class 3
Linux Bootcamp Agenda
Class 1
Class 2
Class 3
Advanced shell scripting� Advanced terminal usage� Special filesystems� Monitoring the system� ...and more!�
Filesystem hierarchy
/ - the “root” of it all. Every file and directory resides under here
/home - User’s personal directory i.e. /home/sandra
/root - The root user’s home directory
/bin - binaries, aka programs you can run
/sbin - system binaries, administrative commands, usually requiring sudo
/usr - “user” applications, also has its own /usr/bin /usr/sbin subdirectories
Filesystem hierarchy, cont.
/etc - system-wide configuration files for services and software
/tmp - temporary files, often wiped between reboots
/lib - system-wide libraries
/opt - applications installed outside your system’s package management
/dev - device files - interact with real or virtualized hardware
/boot - Linux kernel images for booting the system
/var - files that software writes to during operation, i.e. logs, database files
Finding files with locate
locate <filename>
Searches a local database instead of a live search like find
Must be manually updated with updatedb
POSIX vs BASH
Portable Operating System Interface
POSIX is a family of standards to make some uniform specifications around programming interfaces and command-line shell utilities provided by Unix-like operating systems.
The Pedantic Difference
POSIX - the Unix-like environment that is familiar to BSD and *nix
Unix and BSD are “older” implementations of POSIX and have various levels of closed source or open-source licensing.
Linux is just the kernel, but when combined with the GNU Toolchain, GNU Core Utils and a bunch of other stuff, it becomes a full OS.
GNU = “GNU’s Not Unix” - the acronym that literally references itself. GNU comprises a large amount of the open-source software that makes Linux usable.
Makefiles
Installing from source code
wget - download a file from the web
tar - extract compressed files
wget https://hisham.hm/htop/releases/2.2.0/htop-2.2.0.tar.gz
tar -xzvf htop-2.2.0.tar.gz
cd htop-2.2.0
less INSTALL # This is the important help file for installing applications from source
Briefly, the shell command `./configure && make && make install' should configure, build, and install this package.
Dependencies - Get used to it
./configure
configure: error: You may want to use --disable-unicode or install libncursesw.
configure was nice enough to tell us that it needed libncursesw
nsl@linuxclass-000:~/htop-2.2.0$ sudo apt install libncursesw5�...�libncursesw5 is already the newest version (6.0+20160213-1ubuntu1).
When installing from source, you often need the “-dev” version of some packages. Dev packages contain the headers for a library’s interface.
sudo apt install libncursesw5-dev
Back to the INSTALL
It says: ./configure && make && make install
But let’s break down these parts specifically
./configure - a shell script that tests your system for compatibility and proper tooling to compile
make - triggers the default Makefile target which should compile the software to a single executable file. You should have an executable htop in the directory after this step
make install - the Makefile target that puts the output executable in your system’s “user-installed” directory, usually “/usr/local/bin/” and usually requires sudo!!
./configure && make && sudo make install
Conditional brackets
POSIX defines [ ] as the format for conditionals
BASH adds [[ ]] format
No filename expansion (*) or word splitting takes place between [[ and ]], but there is parameter expansion and command substitution.
Using the [[ ... ]] test construct, rather than [ ... ] can prevent many logic errors in scripts. For example, the &&, ||, <, and > operators work within a [[ ]] test, despite giving an error within a [ ] construct.
I find it much easier to work with and solve problems with [[ ]] but you may lose some portability with other types of shells.
Job control
Niceness
Cron
* * * * * command
0 * * * * do_hourly_backup
0 2 * * 7 do_weekly_backup
30 0,4,8,12,16,20 * * * send_tweet
*/15 * * * * check_email.sh
System logs
Application Logs
/var/log/mysql/
/var/log/nginx/
/var/log/httpd/
/var/log/redis/
...etc…
Usually configurable, but if you have to choose, putting them in /var/log/ is a good idea.
“Syslog”
Syslog can refer to a few different things:
The syslog service, which receives, processes, stores or transmits log messages from all across the system or even other hosts. Mostly deprecated by rsyslogd.
The syslog protocol (RFC 5424), which is a transport protocol that specifies how to transmit logs over a network and defines a data format for how messages are structured.
A syslog message, such as any log formatted in the syslog message format.
Actual message format:�<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - - - 'su root' failed for lonvick on /dev/pts/8
How it usually gets saved to disk:�Oct 11 22:14:15 su: 'su root' failed for lonvick on /dev/pts/8
rsyslog
The “rocket-fast” syslog server
Accepts inputs from a huge variety of source and outputs almost any format you want
Pre-installed on many different Linux distributions
/etc/rsyslog.d/
man rsyslogd
man rsyslog.conf
PAM
Pluggable Authentication Modules
man pam
How Linux boots
Services
Services
AT&T System V UNIX init
Circa 1983
Layers and layers of shell scripts, cross-linked to runlevel-specific directories telling the system which services to run.
No dependency model, so startup and shutdown scripts have to be run in a numeric order maintained by you, the administrator.
Scripts can’t execute until everything ahead of them has finished, so they can’t run in parallel, so the system takes a long time to change state.
init process
Make sure the system is running the right services and daemons
Which services are defined by a certain mode or “runlevel”
Most common examples of runlevels:
Single-user mode (runlevel 1)� “Safe mode” - minimal filesystems, no services, root shell only� Multi-user mode (runlevel 5)� Regular operation, all filesystems, network, services, graphics Server mode (runlevel 3)� Similar to multi-user, but no graphics
Replacements for init
init wasn’t really powerful enough to handle the needs of modern systems, and the benefits of multi-core and hyperthreaded processors
Ubuntu Upstart - circa 2009 - discontinued in 2016
systemd - circa 2010 - widely adopted in most modern Linux variants 👍
How systemd works
Unified theory of how services should be configured, accessed, and managed
Like a package manager, defines a robust dependency model, not just for services but for “targets” (the new name for runlevels)
The scope creep is real - systemd also manages network (networkd), logging (journald), logins (logind), and more (tmpfiles, timedated, udevd, libudev, systemd-boot, homed)
systemd units
More than just services - sockets, devices, mount points, startup items, watched filesystem paths, timers, resource management slices, externally created processes…
man systemd.unit
The unit file defines where the executable is, how to start it, stop it, and any dependencies it needs to run
systemd unit files are located in these dirs: �/lib/systemd/system/ - 🚫package installations, don’t modify🚫�/etc/systemd/system/ - 👌User-configured services, overrides👌
systemctl status UNITNAME will tell you where the unit file is for UNITNAME
A systemd unit file
[Unit]�Description=A high performance web server and a reverse proxy server�After=network.target��[Service]�Type=forking�PIDFile=/run/nginx.pid�ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'�ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'�ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload�ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid�TimeoutStopSec=5�KillMode=mixed��[Install]�WantedBy=multi-user.target
systemd unit file sections
[Unit] - This section contains the metadata and configurations of the relationship to other units
[Service] - This could be any unit type, but we care most about services. This section configures how the service starts, runs, restarts, stops, etc.
[Install] - Optional, not interpreted during runtime. Defines if the service is enabled or disabled (starts on boot), and what should happen when its enabled.
systemd dependencies
Wants - Units that should be co-activated if possible, but not required
Requires - Strict dependencies; failure of any Requires terminates this service
Conflicts - Negative dependencies; cannot be co-active with these units
man systemd.service
systemctl - managing systemd
systemd targets
Run level | Target | Description |
0 | poweroff.target | System halt |
1 (single) | rescue.target | Single-user mode |
2 | multi-user.target | Multiuser mode (shell) |
3 | multi-user.target | Multiuser mode with networking |
5 | graphical.target | Multiuser mode + net + GUI |
6 | reboot.target | System reboot |
Using targets
See all available targets: systemctl list-units --type=target
sudo systemctl isolate <target>
sudo systemctl isolate multi-user.target
Activates stated target and its dependencies, and deactivates all other units.
The old command to change runlevels was called telinit and some systems have a compatibility shim to make this work with systemctl.
See the default target: systemctl get-default
Systemd unit statuses
bad | Some kind of problem with systemd, usually bad unit file |
disabled | Present, but not configured to start automatically |
enabled | Installed and runnable, will start automatically |
indirect | Disabled, but has peers in “Also” clauses that may be enabled |
linked | Unit file available through a symlink |
masked | Banished from the systemd world from a logical perspective |
static | Depended on by another unit; has no install requirements |
Systemd Logging / journalctl
journalctl is the program that is used to query the contents of the systemd log journal.
journalctl -t <UNIT> - show logs for a syslog identifier (SyslogIdentifier in unit file under [Service])
sudo journalctl _SYSTEMD_UNIT=nsl.service - show logs for a specific unit
journalctl --disk-usage - view the disk usage of journal files on disk
journalctl -n <NUMBER> - view the last NUMBER entries
journalctl --since=<DATE> - view logs since DATE i.e. “yesterday”
Turning our Software into a Service
/etc/systemd/system/nsl.service
[Unit]�Description=NSL software stack
[Service] �Environment="FLASK_APP=/opt/nsl/hello.py"�ExecStart=flask run�User=nsl�Group=nsl
[Install] �WantedBy=multi-user.target
�A target unit is used to provide synchronization points for other units when booting up or changing states. They also can be used to bring the system to a new state. Other units specify their relation to targets to become tied to the target’s operations.
Running the service
It failed!
Process Creation
When you call a new process, the existing process is cloned, and then the clone can change the program it’s running for a different one.
The original process is referred to as a parent, and then copy is called the child.
PID - Process ID�PPID - Parent Process ID (ps -ef)�UID - The user ID who started the process�EUID - “Effective” user ID, which user’s permissions are applied to the process��Also GID and EGID… guess…!
ps - the process listing
ps - shows processes for the current shell
ps -A or ps -e - Display every active process on a Linux system
ps aux - Display every process in the BSD format
ps -ef - Full-format listing
ps axf - tree-style view of process hierarchy
1178 ? Ss 0:00 /usr/sbin/sshd -D
10784 ? Ss 0:00 \_ sshd: jfineberg [priv]
10856 ? S 0:00 \_ sshd: jfineberg@pts/0
10859 pts/0 Ss 0:00 \_ -bash
12789 pts/0 R+ 0:00 \_ ps axf
ps aux output columns
USER - process owner�PID - process ID�%CPU - Percentage of the CPU this process is using�%MEM - Percentage of the real memory this process is using�VSZ - Virtual size of the process�RSS - (Sometimes displayed as RES) Resident set size (number of pages in memory)�TTY - Control terminal ID�STAT - R = Runnable D = Uninterruptible sleep� S = Sleeping T = Traced or stopped� Z = Zombie�TIME - CPU Time the process has consumed�COMMAND - Command name and arguments (may have been modified by the process)
Kernel processes
The kernel starts some “automatic” processes
Those processes have [brackets] around them in ps*
Kernel also starts the system management daemon init/systemd as PID 1
You do not want to mess with these directly - use systemd/service manager instead
* The brackets actually mean that arguments to the process weren’t available, which mostly common occurs in kernel threads, but occasionally means the process was executed without arguments, or overwrote argv[] with empty data.
�
top - live updating processes and stats
top - find out what processes are running and how many resources they are consuming
Interactive!
q to quit�h for help�O (capital o) for sort menu�z to add color highlighting of running processes�d to change refresh interval�
uptime / load averages
uptime shows current time, how long the system has been running, how many users are logged on, and the system load averages.
System load averages occur as 3 numbers: the last 1, 5, and 15 minutes
Load averages are based on a single-core system, so if you have a quad-core system, your system can handle up to 4.0 load before it’s CPU bottlenecked. Check CPU utilization with top or ps to verify %.
Process Signals
Signals are a form of inter-process communication.
You can stop, interrupt, or suspend processes with special control keys.
You can run a command (kill) to send signals to processes.
The kernel may notify a process of an “interesting” condition such as the death of a child process or the availability of data on an I/O channel.
A process “catches” a signal to handle for it. A process might terminate upon getting a certain signal, or generate a core dump.
Signals
HUP - Hangup�INT - Interrupt�QUIT - Quit�KILL - Kill�SEGV - Segmentation fault�TERM - Software termination�STOP - Stop�WINCH - Window Changed�USR1 - User-defined #1�USR2 - User-defined #2��kill -l - list all process signal options
Keyboard Control Key Signals
Some keyboard combinations can send signals to processes running in your shell.
Ctrl+C - SIGINT - Stop a running process
Ctrl+\ - SIGQUIT - Stop and coredump (save memory to file)
Ctrl+Z - SIGSTP - Suspends a process to the background (bring it back with fg)
Note: Most Linux documentation uses “control character syntax” of ^C, ^Z to denote a control character (Ctrl+key). Most terminals do not care which Ctrl key you use.
Note 2: BASH/your terminal handles these key combinations to attempt to send the appropriate signal to the current process. Programs other than BASH may capture your input and act differently (example: ^C in Python interpreter).
Control Keys Continued
BASH has many more control keys other than signals. They are almost all awesome.
Process management
Memory
free - total system memory in bytes - free -m to display as megabytes
vmstat - virtual memory stats
dmidecode - memory hardware information
egrep 'Mem|Cache|Swap' /proc/meminfo - show more memory information
I’m out of memory, help!
A special space on the disk may be reserved for “swap.”
When you run out of memory, the kernel will try to take memory pages and save them to the swap space on disk. This process is called “thrashing,” and it’s super slow.
cat /proc/sys/vm/swappiness - a value between 0 and 100, the higher the number, the more frequently the system will swap. ( 100 - n = % of memory used before swapping begins ) ( swappiness 60 means swapping starts after 40% memory is used )
Your cloud instances have disabled swap for a variety of reasons. If swap is not available, the kernel OOM killer mechanism kicks in and has to free memory by killing processes.
Disk
mount - display the physical and virtual disks attached to the system
df - the current disk format allocation (df -h for human-readable sizes)
du - disk usage by file and directory (du -h for human-readable sizes)�du -sh - get summary of a directory
lsof - list open files (lsof -p <PID> to search by process ID)
iotop - top for file input/output, not installed by default, needs sudo
Network
ifconfig OR ip - network interface configuration and stats
route OR ip - network routing tables
/etc/resolv.conf - file that configures your DNS resolvers
netstat - see active network connections
iftop - top for network traffic, requires sudo
tcpdump - Capture network packets for troubleshooting and hacking :)
The /proc filesystem
Virtual filesystem - “process information pseudo-filesystem”
Doesn't contain 'real' files but runtime system information (e.g. system memory, devices mounted, hardware configuration, etc).
A control and information center for the kernel. In fact, quite a lot of system utilities are simply calls to files in this directory.
By altering files located in this directory you can even read/change kernel parameters (sysctl) while the system is running.
The /proc filesystem, cont.
/proc/<PID>/ - Numbered directories correspond to an actual process ID
/proc/<PID>/cmdline - Command-line arguments
/proc/<PID>/cwd - Link to current working directory
/proc/<PID>/environ - Values of environment variables
/proc/<PID>/exe - Link to the process’ executable
/proc/<PID>/fd/ - Directory of all the file descriptors held
The /proc filesystem, cont. (again)
/proc/<PID>/status - Process status in human-readable form
Other system-wide proc files:
/proc/cpuinfo - Hardware CPU information
/proc/meminfo - Hardware memory information
/proc/sys - Read and write kernel parameters
How to use /proc
Even though most of the files in /proc have a size of 0, you can read them!
sudo cat /proc/<PID>/status
/proc/<PID>/environ is all scrunched up because strings are null-terminated in C, and /proc is just looking at kernel memory. Fix this by using “strings:”
sudo strings /proc/<PID>/environ
Most of this filesystem is read-only, but some “files” can be changed to tweak the kernel without needing to recompile and reboot.
sysctl - changing kernel parameters
sysctl -a - print all sysctl variables
sysctl <PARAMETER> - view a specific variable
sysctl -w <PARAMETER>=<VALUE> - write a variable value*
You can also use /proc/sys/ to change parameters
Example: vm.swappiness = /proc/sys/vm/swappiness
Read variable: cat /proc/sys/vm/swappiness
/etc/sysctl - keeping kernel parameters
If you reboot a system, all changes via sysctl and /proc/sys are lost :(
To persist kernel parameters through a reboot, write <PARAMETER>=<VALUE> into /etc/sysctl.conf or file within /etc/sysctl.d/ directory.
Useful sysctl parameters
Settings to determine if an idle network connection is dead and should be closed:�net.ipv4.tcp_keepalive_time �net.ipv4.tcp_keepalive_intvl �net.ipv4.tcp_keepalive_probes
Network buffers�net.core.somaxconn�net.core.netdev_max_backlog�net.core.netdev_budget�net.core.netdev_budget_usecs
Virtual Memory dirty data (cached but not written to disk)�vm.dirty_ratio�vm.dirty_background_ratio
vm.vfs_cache_pressure - adjust virtual file system cache reclaim rate
Other programs you should know
tee - send stdin to stdout and a file at the same time
wc - word count (characters, words, and lines)
cut - select certain parts of stdin based on delimiters and fields
sort - alphabetically sort from stdin
uniq - remove sequential duplicate lines (use with sort)
tar / zip / unzip - create and extract archive files
curl / wget - make web requests
dig / nslookup - query DNS
Great Books to check out
Unix and Linux System Administration Handbook, 5th Edition
How Linux Works: What Every Superuser Should Know, 2nd Edition
Essential System Administration: Tools and Techniques for Linux and Unix Administration, 3rd Edition
Learning the bash Shell: Unix Shell Programming
CompTIA Linux+ Powered by Linux Professional Institute Study Guide
The Linux Programming Interface: A Linux and UNIX System Programming Handbook