1 of 75

Sell Cron, Buy Airflow!�Modern Data Pipelines in Finance

James Meickle

Site Reliability Engineer, Quantopian

Velocity New York, 10/02/2018

@jmeickle

2 of 75

About James

Site Reliability Engineer, Quantopian
Site Reliability Engineer, Harvard Center for Brain Science
Sales Engineer, AppNeta
Developer, Romney for President 2012
Formerly academia & public policy

Is this a data pipeline?

cron

@jmeickle

3 of 75

DISCLAIMER 1:

I am an SRE professional at a financial institution, not a finance professional!

@jmeickle

4 of 75

DISCLAIMER 2:

Opinions expressed in this presentation are my own, not Quantopian's!

@jmeickle

5 of 75

DISCLAIMER 3:

This presentation is not investment advice!

@jmeickle

6 of 75

SEC guidelines are extremely serious.

@jmeickle

7 of 75

PART 1:

Data pipelines and data problems

@jmeickle

8 of 75

https://www.quantopian.com/about @quantopian

@jmeickle

9 of 75

+

@jmeickle

10 of 75

data

@jmeickle

11 of 75

SRE Team

AWS Infra
Kubernetes
Architecture
Monitoring
Logging
Build
Deploy

@jmeickle

12 of 75

Black Team

Data infrastructure
Investment automation
Productizing data science

@jmeickle

13 of 75

Community data

Data from commercial data vendors
Cleaned and reconciled
Packaged into high-performance formats
Aware of lookahead bias
Portfolio risk model

@jmeickle

14 of 75

Internal data

Site analytics
Calculating contest performance
Simulating algorithm portfolios
Experimental datasets and data science
Risk management
Allocation decisions
Portfolio returns
Compliance reporting

@jmeickle

We also have internal data use cases - if we expose that much data to the end user you can imagine that we have even more ourselves! Some very similar to most companies, others quite distinct.

- We have over 200,000 users from more than 190 countries writing algorithms! We have to perform typical analyses of user growth and activity.

- Internally, we come up with rigorous quantitative definitions of successful algorithms. We don't look at the code, so we have to make these decisions based on analysis of performance characteristics.

- We also have a variety of research efforts to optimize that process, decide on allocation size, or to create new datasets for the community.

- When we do trade user algorithms, we need to manage our fund operations, such as understanding our risk exposures and generating reports for compliance purposes.

15 of 75

@jmeickle

16 of 75

@jmeickle

17 of 75

High reliability

@jmeickle

18 of 75

Unreliable data

@jmeickle

19 of 75

Brittle crontabs

@jmeickle

20 of 75

Uncoordinated logic

@jmeickle

21 of 75

SRE bottleneck

@jmeickle

22 of 75

Of course we wanted to

"fix our data pipelines", but...

@jmeickle

23 of 75

What were our business goals?

@jmeickle

24 of 75

PART 2:

Business goals and tool selection

@jmeickle

25 of 75

Reliability

@jmeickle

26 of 75

Observability

@jmeickle

27 of 75

Velocity

@jmeickle

28 of 75

Our data problems are unique, just like everyone else's.

@jmeickle

29 of 75

Democratization

@jmeickle

30 of 75

NOT big data

@jmeickle

31 of 75

NOT streaming data

@jmeickle

32 of 75

@jmeickle

33 of 75

@jmeickle

34 of 75

@jmeickle

35 of 75

@jmeickle

36 of 75

@jmeickle

37 of 75

@jmeickle

38 of 75

And the winner is...

@jmeickle

39 of 75

apache airflow

@jmeickle

40 of 75

@jmeickle

41 of 75

@jmeickle

42 of 75

Self-explaining systems are more observable

@jmeickle

43 of 75

@jmeickle

44 of 75

@jmeickle

45 of 75

import airflow

@jmeickle

46 of 75

PART 3:

The one with the code samples

@jmeickle

47 of 75

@jmeickle

48 of 75

@jmeickle

49 of 75

@jmeickle

50 of 75

@jmeickle

51 of 75

when 2 airflow??

@jmeickle

52 of 75

@jmeickle

53 of 75

@jmeickle

54 of 75

@jmeickle

55 of 75

@jmeickle

56 of 75

@jmeickle

57 of 75

@jmeickle

58 of 75

@jmeickle

59 of 75

@jmeickle

60 of 75

@jmeickle

61 of 75

@jmeickle

Something we’re still developing, but super pleased with? Kubernetes integration for our internal data science projects.

Airflow executes code in subprocesses, meaning you have to worry about worker resources and security borders.

Kubernetes has resilient cronjobs, but no dependency tracking. Half of the problems with our crontab system would still apply.

What do you get when you put them together? A system where someone can write Python code to schedule and launch their Python data science containers in a highly elastic compute environment with resource and security isolation. SOUNDS PRETTY COOL, RIGHT?

You can expect even more mainstreaming of Kubernetes integration with the KubernetesExecutor (via Bloomberg, already in 1.10) that launches all Airflow operators in pods, and the Airflow Operator from Google that installs Airflow itself in Kubernetes.

Next step: Airflow to run Helm jobs, or create temporary infrastructure.

62 of 75

@jmeickle

63 of 75

I 💖

@jmeickle

64 of 75

@jmeickle

65 of 75

Appendix 1:

Why Airflow is winning

@jmeickle

66 of 75

@jmeickle

67 of 75

@jmeickle

68 of 75

@jmeickle

69 of 75

Appendix 2:

Future Development

@jmeickle

70 of 75

@jmeickle

I think there's a tremendous untapped oppotunity for DAG testing

- DAGs are defined entirely with Python code, rather than any DSL.

- Python has extensive testing frameworks.

- Therefore we can write code to simulate failures, and see how the DAG may react.

- We can expect conditions like "we get notified if there's no file by X time", and actually confirm that this happens.

-- That could be if the previous task fails.

-- Or if this task fails.

-- Or if this task hangs.

-- Or...

- We can take historical failure modes and build them as test cases.

- We can even encode operator actions, like ensuring that an action results in expected remediation.

Airflow doesn't provide those capabilities yet - but there isn't a fundamental blocker. Some of our test suites already mock out DAG flows and "fast forward" execution times.

71 of 75

@jmeickle

72 of 75

@jmeickle

Automatic Forking for Branches Affecting Data Pipelines

You often might want to make changes to a data pipeline, and then compare the differences. But what if it causes a failure downstream? You can't just modify a node in isolation - you should compare everything about it. (Or if you expect a byte-identical output, actually add a node to check for that.) But it can be really hard to run multiple pipelines in parallel if they require manual modification each time.

This could one day be possible automatically in an Airflow DAG! You'd need to have an algorithm for resolving branch merging automatically, isolated execution tasks/containers (like Kubernetes), and isolated slug-based data outputs. All of those exist - they're just not put together yet.

End to end workflow: Test any downstream, "dirtied" tasks via a shadow DAG and understand any new branches/intersections. Like canarying, but for data processes! Allows more semver-like releases of data artifacts and have a policy on producing n versions at a time.

73 of 75

Does your team need

data engineering training?

Talk to me at the reception!

@jmeickle

74 of 75

Thank you!

@jmeickle

75 of 75

This presentation is for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation for any security; nor does it constitute an offer to provide investment advisory or other services by Quantopian, Inc. ("Quantopian"). Nothing contained herein constitutes investment advice or offers any opinion with respect to the suitability of any security, and any views expressed herein should not be taken as advice to buy, sell, or hold any security or as an endorsement of any security or company. In preparing the information contained herein, Quantopian, Inc. has not taken into account the investment needs, objectives, and financial circumstances of any particular investor. Any views expressed and data illustrated herein were prepared based upon information, believed to be reliable, available to Quantopian, Inc. at the time of publication. Quantopian makes no guarantees as to their accuracy or completeness. All information is subject to change and may quickly become unreliable for various reasons, including changes in market conditions or economic circumstances.

@jmeickle