1 of 23

Make: Build Software

Alexius Wadell

Lover of Make and all Directed Acyclic Graphs

Examples: https://tinyurl.com/jsvjhkz5

1

1

1

1

2 of 23

Build Tools: Automating Compilation

Blaze Inspired

Makefile Generation

Other Tools

2

2

2

2

3 of 23

Why Make?

Pros

  1. Lightweight
  2. Recipes are shell scripts
  3. It’s just makefiles

Cons

  1. Non-Hermetic
  2. Uses timestamps
  3. No caching of artifacts

3

3

3

3

4 of 23

But I don’t compile code…

Do you?

  • Download compressed files and then unzip them
  • Generate result files by running scripts
  • Run tests that depend on input files
  • Install dependencies with pip, conda, Pkg.jl, spack

​

​

4

4

4

4

5 of 23

Let’s Make Pie!

Target

Prerequisites

Rule

Recipe

5

5

5

5

6 of 23

Let’s Make Pie!

6

6

6

6

7 of 23

What’s going on?

  1. Build a DAG to represent dependencies
  2. Check that we have needed files
  3. Identify missing files
  4. Use recipes to build missing files

​

butter

pie_crust

flour

libbys_pumpkin

pumpkin_filling

pumpkin_pie

salt

water

7

7

7

7

8 of 23

Summarizing Data

8

8

8

8

9 of 23

Automatic Variables

Make Manual: 10.5.3 Automatic Variables

Symbol

Use

$@

Filename of the target

$<

Filename of 1st prerequisites

$(@D)

The directory of the target

9

9

9

9

10 of 23

Phony Targets

With no arguments, make builds the first target it finds

  • So far that’s what we’ve wanted
  • But can be undesirable for larger projects

​

Ever seen something like:

  • make build
  • make install
  • make clean
  • make test

Make Manual: 7.2.6 Standard Targets for Users

10

10

10

10

11 of 23

Phony Targets

Define targets that are not files

  • Avoids conflicts existing filenames
  • Indicates to make that this rule doesn’t create a file

​

Some uses

  • Submitting jobs to slurm
  • Download all data files
  • Deleting generated files

Make Manual: 4.6 Phony Targets Make Manual: 4.9 Special Built-in Target Names

11

11

11

11

12 of 23

Pattern Rules: Transform Files by Name

Make Manual: 10.5 Pattern Rules

12

12

12

12

13 of 23

Chaining Rules

all

data1.csv

data1.sum

grand.csv

data2.csv

data2.sum

grand.sum

sum.py

13

13

13

13

14 of 23

Multi-Target Pattern Rules

all

data1.csv

data1.sum

grand2.csv

data2.avg

data2.csv

grand2.avg

sum_avg.py

14

14

14

14

15 of 23

Programs without Output

Not all programs produce “output”

  • pip install package
  • spack install package

​

Possible Solutions

  • Have an install rule and run manually
  • Have an install rule and depend on it

​

15

15

15

15

16 of 23

Enter the Witness File

16

16

16

16

17 of 23

Next Steps

17

17

17

17

18 of 23

Homework

  • Work through the example exercises
  • Clone Julia and build it from scratch: https://github.com/JuliaLang/julia
  • Add a makefile to your repo (Last week’s homework) that:
    • Installs any dependencies with “make install”
    • Runs a test suite with “make test”

​

18

18

18

18

19 of 23

Resources

Documentation

​

Make in the Wild

​

​

19

19

19

19

20 of 23

Common Gotchas

20

20

20

20

21 of 23

Common Gotchas

Never use make –j to submit jobs to slurm or any other scheduler

​

Formatting:

  • Recipes are indented with 1 tab character (Not spaces)
  • Make is whitespace sensitive: Don’t put whitespace in filenames
  • Turn on “show whitespace” in your editor

​

Shell:

  • Each line in a recipe is executed in it’s own shell
  • Setting variables on one line and using later doesn’t work!
  • Adding “.ONESHELL:” to your makefile will run recipes in a single shell

21

21

21

21

22 of 23

Common Gotchas: Path Handling

Directories time stamps are not updated when contents change

  • Always touch directories after modify, if using as a target
  • Ideally track files at the file level, not the directory

​

Make matches on strings not canonical filenames

  • Recommend including trailing “/” in directory names
  • “apple//bar/foo” and “apple/bar/foo” are the same to POSTX, but not make
  • Avoid spaces in filenames (Including parent paths)

​

22

22

22

22

23 of 23

Common Gotchas: Limitations

Make is not hermetic

  • It is sensitive to your path (is “python” python 2 or 3?)
  • External changes (pip uninstall…) can mess up your build
  • bazel is hermetic, but it is a lot more work for non-standard workflows

Recursive Make Considered Harmful

  • tldr: Don’t call make from within make. Just don’t.
  • No guarantee that it will terminate

​

​

​

​

​

23

23

23

23