Data Pipeline with Threading
Ken Q Pu, Faculty of Science, Ontario Tech University
Comprehensive Guide
Ken Q Pu, Faculty of Science, Ontario Tech University
Clojure's perspective of computation
Functional programming languages including Clojure focus on data generation and transformation.
Ken Q Pu, Faculty of Science, Ontario Tech University
Clojure's perspective of computation
parse
compile
check-permission
load
run-query
query-string
data-file
user-info
query-result
Ken Q Pu, Faculty of Science, Ontario Tech University
Clojure's perspective of computation
Clojure provides syntactic support to help with building complex data pipelines.
(run-query
(check-permission
(compile (parse query-string))
(load data-file)
user-info)
(load data-file))
Ken Q Pu, Faculty of Science, Ontario Tech University
Thread-first macro form
(-> <initial-value> <form> <form> ... <form>)
result
The previous result is inserted as the first argument of the next form.
(-> 9 (inc) (- 5 2) (pow 4))
=> ?
Ken Q Pu, Faculty of Science, Ontario Tech University
Thread-first macro form
(-> 9 (inc) (- 5 2) (pow 4))
=> ?
(inc )
(- 5 2)
(pow 4)
9
9
10
3
81.0
Ken Q Pu, Faculty of Science, Ontario Tech University
Thread-last macro form
(->> <initial-value> <form> <form> ... <form>)
result
The previous result is inserted as the last argument of the next form.
(->> 9 (inc) (- 5 2) (pow 4))
=> ?
Ken Q Pu, Faculty of Science, Ontario Tech University
Thread-last macro form
(->> 9 (inc) (- 5 2) (pow 4))
=> ?
(inc )
(- 5 2 __ )
(pow 4 __ )
9
9
10
-7
6.1E-5
Ken Q Pu, Faculty of Science, Ontario Tech University
as-> threading form
Thread-first and thread-last fix the position to insert the previous value in the next form throughout the threading form.
The as-> form allows one to control where the insertion occurs throughout the threading form.
(as-> <initial-value> <symbol>
<form>
<form> ...)
The forms use the symbol to indicate where the previous result should be inserted.
Ken Q Pu, Faculty of Science, Ontario Tech University
as-> macro form
(as-> 9 x
(inc x)
(- 5 x 2)
(pow x 4))
=> ?
(inc x )
(- 5 x 2 )
(pow x 4 )
9
Ken Q Pu, Faculty of Science, Ontario Tech University
as-> macro form
(as-> 9 x
(inc x)
(- 5 x 2)
(pow x 4))
=> ?
(inc x )
(- 5 x 2 )
(pow x 4 )
9
10
-7
2401
9
Ken Q Pu, Faculty of Science, Ontario Tech University
Complex data pipelines
parse
compile
check-permission
load
run-query
query-string
data-file
user-info
query-result
(run-query
(check-permission
(compile (parse query-string))
(load data-file)
user-info)
(load data-file))
Ken Q Pu, Faculty of Science, Ontario Tech University
Complex data pipelines
parse
compile
check-permission
load
run-query
query-string
data-file
user-info
query-result
(let [data (load data-file)]
(-> query-string
(parse)
(compile)
(check-permission user-info data)
(run-query data)))
Ken Q Pu, Faculty of Science, Ontario Tech University