CSE 391

Shell Commands

More Redirection

​

Slides created by Josh Ervin and Hunter Schafer. �Based off slides made by Marty Stepp, Jessica Miller, Ruth Anderson, Brett Wortzman, and Zorah Fung

​

polleverywhere

AGENDA

• Minor logistics
• HW2 Q&A + questions
• for-sure: Matt will discuss the last question!
• Reviewing pre-lecture concepts
• Many discussion questions :)
• New commands (xargs, tee, cut) + new concepts (“control flow”)
• HW3 tips & tricks

ADMINISTRIVIA

• Upcoming git pre-lectures are new!
• Trying to improve based on feedback + student observations
• Generally a very challenging topic
• Suggestion: ask us questions if you’re confused about the pre-lecture
• Next week: midterm feedback :o
• Sometime this week: we will set up a GitLab project for you
• If you get an email, that’s expected :)
• You may not get an email, that’s ok too :’)
• Minor website updates
• Recordings, lecture note links

DISCUSSION QUESTION #1

Question 1: Suppose we have a file directories.txt where each line contains the name of a directory that we want to create.

Our friend proposes that we run cat directories.txt | mkdir into order to do this?

Will this command work? If not, how can we change this command to get it to work/could we improve it in any way?

PRIMING SOME QUESTIONS

Questions you may have from pre-lecture:

• new commands: xargs, tee, cut
• new “control flow”: &&, ||, ;

Questions you may have from homework/last week:

• concepts: stdin, stdout, stderr
• manipulating streams: pipes, redirection
• other new commands (e.g. grep)

DISCUSSION QUESTION #2

Question 7: Suppose we know that Compile.java compiles with no errors. If we were to run javac Compile.java || java Compile what would be the result? Would the program compile? Would it run?

DISCUSSION QUESTION #8

Question 8: What would be some security issues with running some of the commands you learned from lecture (think of some of the practical implications of rm, find, and xargs in regards to malicious code injection)?

TIPS FOR HW3

• I say this every week, but: break up big problems into small problems!
• pipes are your friend :)
• don’t grep for too much at once
• don’t stress too much about “efficiency”
• From the last question in HW2, we saw that order matters.
• prioritize correctness over efficiency
• when could you accidentally filter out too much data?
• you can always filter more, but you can’t “unfilter”!
• make sure you know why xargs exists
• if not, review the pre-lec/lecture content on stdin & argument

AGENDA

• Logistics, Roadmap
• Combining Commands
• More input/output redirection
• cut, reading log files

ROADMAP

• Introduction to the command line
• Input/output redirection, pipes
• More input/output redirection, tee, xargs
• Git: Fundamentals
• Git: Branches and rebasing
• Regular expressions
• More regular expressions, sed
• Users and permissions
• Bash scripting
• Industry applications

PIPES

command1 | command2

• Execute command1 and send its standard output as standard input to command2.
• This is essentially shorthand for the following sequence of commands:

command1 > filename

command2 < filename

rm filename

• This is one of the most powerful aspects of unix - being able to chain together simple commands to achieve complex behavior!

COMBINING COMMANDS

command1 ; command2

• Execute command1, then execute command2.

command1 && command2

• Execute command1, and if it succeeds, then execute command2.

command1 || command2

• Execute command1, and if it fails, then execute command2.

​

​

What would happen after running the following command: ls *.java | javac

​

​

Solution: This won’t work because javac does not read from stdin! Piping makes the stdout of the last program become stdin of the next.

​

XARGS

• xargs is a program that converts standard input to command line arguments (i.e. parameters).
• For example, to compile all java files in the current directory we could use the following:
• $ ls *.java | xargs javac

​

​

FIND

• find is a program for searching your filesystem for certain files.
• For example, to list all java files in the current directory and all subdirectories, recursively, we would run the following
• $ find -name “*.java”
• This is commonly used with xargs. For instance, to compile all Java files in the current directory and all subdirectories recursively
• $ find -name “*.java” | xargs javac
• Note that find has a plethora of options and flags, but we will most commonly use find with the -name and -type flags

​

​

COMMAND SUBSTITUTION

$(command)

• Another powerful tool is command substitution. It executes the given command and places that string literally into the given context.
• For example, to compile all Java files in the current directory and subdirectories recursively, we can run the following
• $ javac $(find -name “*.java”)

​

​

What is the command to remove all files listed in the file toRemove.txt?

​

toRemove.txt

CompilerErrors.java

beans.txt

​

​

xargs rm < toRemove.txt

STDERR REDIRECTION

• We’ve learned that we can redirect standard error using the 2> operator.
• Sometimes, however, we want standard error and standard out to go to the same location. We can do that with the following syntax:
• $ command > out.txt 2>&1
• To understand this command, this reads as “redirect standard out to out.txt, redirect standard error to the same place as standard out”

​

TEE

• Sometimes, we want to redirect the output of a command to both a file and to the console. Do do this, we can pipe the output of a command to tee
• $ command | tee file.txt
• To redirect both standard output and standard error to a file, and to the console, we use the following
• $ command 2>&1 | tee file.txt

​

Suppose we want to run the Java program Mystery. What would be the command to output both standard error and standard output to mystery_out.txt and print both to the console?

​

​

java Mystery.java 2>&1 | tee mystery.txt

CUT

cut -d<DELIMITER> -f<FIELD>

• cut is a simple program to split lines based on a given delimiter.
• For example, to split the string “a,b,c,d,e” on commas and get the second entry, we would use the following:
• $ echo “a,b,c,d,e” | cut -d, -f2
• Note: the echo program simply prints the given string to standard out

​

LOGS

• A common exercise in daily software development and operations is looking at log files - basically a status report of what is going on inside the program.
• We can look at the logs for all the CSE course websites by reading the file: /cse/web/courses/logs/common_log
• For example, to actively watch the log file and only look for access to our own course website, we could use the following

$ tail -f /cse/web/courses/logs/common_log | grep “391”

​

​