Beyond Bash
Shell scripting in a typed, OO language
Scala by the Bay, 15 August 2015
Slides: http://tinyurl.com/beyondbash
0.1 Who am i
Li Haoyi
Paid $ to work on dev tools @ Dropbox
Not paid $ to work on Scala.js
Using Scala professionally since… never
0.2 Agenda
0.3 Problem Statement
“How can we stop using the worst languages in the world to build our most important infrastructure?
1.1 Application Architecture
Server
Client
Client
Database
Server
Safety
Maybe Safety?
DANGER
DANGER
DANGER
DANGER
1.2 Application Architecture
Server
Client
Client
Database
Server
Safety
Maybe Safety?
Safety
Safety
1.3 Scala.js!
1.3 Scala.js!
Bad when better than worse is excellent
1.4 Application Architecture
Server
Client
Client
Database
Server
Safety
Maybe Safety?
Safety
Bash, Python, Puppet, Ruby, Vagrant...
Safety
DANGER
1.5 Danger Below!
High-performance, type-safe application code
High-performance, type-safe web front-end
Underpinned by a mix of Bash, Python, Ruby, Puppet, Vagrant, ...
1.5 Danger Below!
Hard to test!
Not typechecked!
Worst consequences for errors
1.5 Danger Below!
Server
Client
Client
Database
Server
Ok
Ok
Bash, Python, Puppet, Ruby, Vagrant...
Ok
Ok
Down
1.5 Danger Below!
Server
Client
Client
Database
Server
Ok
Ok
Ok
Ok
Down
Down
Down
Bash, Python, Puppet, Ruby, Vagrant...
Ok
1.5 Danger Below!
Server
Client
Client
Database
Server
Ok
Ok
Ok
Ok
Down
Down
Down
Down
Bash, Python, Puppet, Ruby, Vagrant...
bash$
1.6 What's wrong with Bash?
1.7 What's wrong with Bash?
# Run a script on all files with some extension
find . -name '*.ext' | while IFS=$'\n' read -r FILE; do� process "$(readlink -f "$FILE")" || echo "error processing: $FILE"�done
find . -name '*.ext' \( -exec ./some_other_script "$PWD"/{} \; -o -print \)
find . -name '*.ext' -exec ./some_other_script "$PWD"/{} \;
http://stackoverflow.com/questions/4410412/bash-processing-recursively-through-all-files-in-a-directory
It seems to work
???
Incorrect
???
Good Solution!
???
“It seems to work”
Such a high degree of confidence!
Why do people use Bash
Can we use something else?
Sample use case
Why do people use Bash
Can we use something else?
No
Bash is Better
1.9 Bash vs Scala
rm -rf folder/inner_dir
def removeAll(path: String) = {� def rec(f: File): Seq[File] =� f.listFiles� .filter(_.isDirectory)� .flatMap(rec) � .++(f.listFiles)� for(f <- rec(new File(path))){� if (!f.delete())� throw new RuntimeException()� }�}�removeAll("folder/inner_dir")
1 line 24 chars
12 lines 279 chars
1.10 Bash vs Python
rm -rf folder/inner_dir
import shutil�shutil.rmtree('folder/my_file.jpg')
1 line 24 chars
2 lines 50 chars
1.11 Bash vs Python: Round 2
import subprocess�subprocess.check_call(["git", "status"])
git status
1 line 10 chars
2 lines 60 chars
Important Bits
Dumb Noise
1.12 Bash is Better
Less syntactic ceremony cp fileB fileB
Common operations are short ls
Fewer keystrokes overall
Commands do what you want rm -rf folder
Very Important!
Ammonite-Ops
Rock-solid filesystem ops in Scala
"com.lihaoyi" %% "ammonite-ops" % "0.4.5"
2.1 Ammonite-Ops
2.2 Ammonite-Ops
git status
rm folder/my_file.jpg
%git 'status
rm! 'folder/"my_file.jpg"
1 line 10 chars
1 line 21 chars
1 line 12 chars
1 line 25 chars
2.3 A Taste of Ammonite
import ammonite.ops._
// Delete a file or folder�rm! cwd/'folder��// Make a folder named "folder"�mkdir! cwd/'folder��// Copy a file or folder�cp(cwd/'folder, cwd/'folder1)��// List the current directory�val listed = ls! cwd
Short commands that mirror Bash
That do what you want!
No ambiguity in parsing arguments
2.4 A Taste of Ammonite
// List the current directory�val listed: Seq[Path] = ls! cwd��// Commands return normal values�// you can process normally�for(path <- listed){� println(path)� // paths are proper data-structures� // with attributes, methods, etc.� if (path.ext == "tmp") rm! path�}
Values are typed, structured data
No string munging trying to do simple tasks!
2.5 Piping
things | f -> things map f�things || f -> things flatMap f�things |? f -> things filter f�things |& f -> things reduce f�things |! f -> things foreach f�things |> f -> f(things)
f! thing -> f(thing)
Traversable
Any
T => V
2.6 Putting it Together
2.7 Putting it Together
# Recursive line count of Javascript files
find ./dir -name '*.js' | xargs wc -l
�ls.rec! cwd/'dir |? (_.ext == "js") | read.lines | (_.size) sum
38 chars
64 chars
2.8 Putting it Together
# List dot-files *only*
ls -a | grep "^\."
�ls! cwd |? (_.last(0) == '.')
19 chars
30 chars
2.9 Putting it Together
# Largest 7 files in the current directory
find . -ls | sort -nrk 7 | head -7
ls.rec! cwd | (x => x.size -> x) sortBy (-_._1) take 7
35 chars
55 chars
2.10 Ammonite-Ops
2.11 This begs the question...
Can we use Ammonite-Ops + Scala-REPL as our default shell?
Let’s try contributing some changes to https://github.com/lihaoyi/demo
No
2.12 No
Ammonite-REPL
Re-inventing the Scala REPL
3.1 Ammonite-REPL
3.2 Using the Ammonite REPL
# Standalone Executable�curl -L -o amm https://git.io/v3E3V; chmod +x amm; ./amm
// SBT project�libraryDependencies += (
"com.lihaoyi" % "ammonite-repl" % "0.4.5" % "test"
cross CrossVersion.full
)�initialCommands in (Test, console) :=
"""ammonite.repl.Repl.run("")""" // sbt test/console
Live Demo
Whee!
3.3 Fun Features
3.4 Ammonite-REPL
3.5 This begs the question...
Can we use Ammonite-Ops + Ammonite-REPL as our default shell?
Let’s try contributing some changes to https://github.com/lihaoyi/wootjs
3.6 Ammonite-REPL
3.7 Work In Progress
Conclusion
WTF did we just do?
4.1 Ammonite...
Ammonite-Ops
Really-nice Filesystem Library
Ammonite-REPL
Really-nice Scala REPL
Bash Replacement?
4.2 Ammonite...
Why?
Did we need to do so many things?
4.3 Why Not...
4.4 Space of Possible Systems APIs
Danger
Verbosity
java.io
java.nio
Bash
Python
FP, Inferred Types, ...
Ammonite
Plains of Diminished Returns
Cliffs of Insanity
4.5 Problems w/ Scala as your Shell
4.6 Hopefully free improvements
5.0 Application Architecture
Server
Client
Client
Database
Server
Safety
Maybe Safety?
Bash, Python, Puppet, Ruby, Vagrant...
DANGER
DANGER
DANGER
DANGER
DANGER
DANGER
5.1 Application Architecture
Server
Client
Client
Database
Server
Safety
Maybe Safety?
Safety
Bash, Python, Puppet, Ruby, Vagrant...
DANGER
Safety
5.2 Application Architecture
Server
Client
Client
Database
Server
Safety
Maybe Safety?
Safety
Ammonite-Ops Ammonite-REPL
Safety
Safety
5.3 Beyond Bash
Additional Slides
2.5 Absolute Paths & RelPaths
case class Path(segments: Seq[String])
�case class RelPath(segments: Seq[String], ups: Int)
Absolute
Any ..s at the start of the path
2.6 Constructing Paths
> root�/��> root/'usr/'bin�/usr/bin��> 'src/'main�src/main��> up/up/'src/'main�../../src/main
Paths are constructed using / and...
2.7 Combining Paths
> val rel = 'src/'main�src/main��> val wd = root/'Users/'lihaoyi�/Users/lihaoyi��> wd/rel�/Users/lihaoyi/src/main��> wd/rel/up�/Users/lihaoyi/src
Paths can be stitched together using /
Paths are normalized at every step!
not /Users/lihaoyi/src/..
2.8 Invalid Paths
> val rel: RelPath = 'src/'main
> val abs: Path = root/'usr/'bin��> abs/rel�/usr/bin/src/main�> rel/abs�<console>:15: error: type mismatch;�> rel/rel
src/main/src/main�> abs/abs�<console>:14: error: type mismatch;
Combining Paths & RelPaths improperly is a compilation error
2.9 Invalid Paths
> rel = "???"�> abs = "???"��> abs + "/" + rel�> abs + rel�> "/" + abs + rel�> "/" + abs + "/" + rel�> rel + abs�// correct but annoying to write�> os.path.join(abs, rel)
> val rel: RelPath = 'src/'main
> val abs: Path = root/'usr/'bin��> abs/rel�/usr/bin/src/main�> rel/abs�<console>:15: error: type mismatch;�> rel/rel
src/main/src/main�> abs/abs�<console>:14: error: type mismatch;