Open source workflow tools comparison
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
View only
 
 
Still loading...
ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
LuigiAirflowPinball
2
repohttps://github.com/spotify/luigihttps://github.com/airbnb/airflowhttps://github.com/pinterest/pinball
3
docshttp://luigi.readthedocs.orghttps://airflow.readthedocs.orgnone
4
my reviewhttp://bytepawn.com/luigi.htmlhttp://bytepawn.com/airflow.htmlhttp://bytepawn.com/pinball.html
5
github forks75034558
6
github stars40291798506
7
github watchers31916647
8
commits in last 30 dayslots of commitslots of commits3 commits
9
architecture
10
web dashboardnot really, minimalvery niceyes
11
code/dslcodecodepython dict + python code
12
files/datasetsyes, targetsnot really, as special tasks?
13
calendar schedulingno, use cronyes, LocalScheduleryes
14
datadoc'ablemaybe, doesn't really fitprobably, by conventionyes, dicts would be easy to parse
15
backfill jobsyesyes?
16
persists statekindofyes, to dbyes, to db
17
tracks historyyesyes, in dbyes, in db
18
code shippingnoyes, pickleworkflow is shipped using pickle, jobs are not?
19
prioritiesyesyes?
20
parallelismyes, workers, threads per workersyes, workers?
21
control parallelismyes, resourcesyes, pools?
22
cross-dag depsyes, using targetsyes, using sensorsyes
23
finds new deployed tasksnoyes?
24
executes dagno, have to create special sink taskyesyes
25
multiple dagsno, just oneyes, also several dag instances (dagruns)yes
26
scheduler/workers
27
starting workersusers start worker proccesesscheduler spawns workers processesusers start worker procceses
28
commsscheduler's HTTP APIminimal, in state dbthrough master module using Swift
29
workers executeworker can execute tasks that is has locallyworker reads pickled tasks from dbworker can execute tasks that is has locally?
30
contrib
31
hadoopyesyesyes
32
pigyesdoc mentions PigOperator, it's not in the sourceno
33
hiveyesyesyes
34
pgsqlyesyesno
35
mysqlyesyesno
36
redshiftyesnono
37
s3yesyesyes
38
source
39
written inpythonpythonpython
40
loc18,00021,00018,000
41
testslotsminimallots
42
maturityfairlowlow
43
other serious usersyesnot reallyno
44
pip installyesyesbroken
45
niceties-sla, xcom, variables, trigger rules, celery, chartspass data between jobs
46
does it for you
47
sync tasks to workersnoyesno
48
schedulingnoyesyes
49
monitoringnonono
50
alertingnoslas, but probably not enoughsends emails
51
dashboardsnoyesyes
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
Loading...
 
 
 
Luigi, Airflow, Pinball