2 of 10

Motivations

A hand full of questions in the same month:

How do I "catch" an exception and run my own exception handler on it rather than failing the task? (2 users)

I know this successful result that was checkpointed earlier is wrong, how do I ignore it?

I know this exception is permanent, I would like to checkpoint it for next run

3 of 10

common theme

working with the outcomes of tasks in Power User ways (i.e. not the defaults)

outcome = future.result() or future.exception()

4 of 10

example - DAG-brain vs PL-brain

task 1

task 2

task 3

task 4

task 1 succeeds…

… so task 2 runs …

but it happens to fail

so task 3 and 4 do not run

a = task1()

b = task2(a) # can raise exception

c = task3(b)

d = task4(b)

# see MacLisp 1972

a = task1()

try:

b = task2(a)

except:

b = 7

c = task3(b)

d = task4(b)

5 of 10

beyond DAG-brain - other ways to operate on tasks

* checkpointing - examine a task and replace it with the outcome of a different task (that we hope is actually the same…)

* join_apps - elaborate placeholders in the DAG into dynamically created subworkflows, created by running code

* retry_handler - inspect failures to decide how retrying should happen

all ad-hoc single purpose hooks to operate on tasks and their outcomes

6 of 10

prototypes:

PR #3535 - Checkpoint plugins

User policies (in Python) of when to write and read outcomes (.result() and .exception())

PR #3553 - Retry handlers can complete a failed task with a result

Historical: "compute whether we should retry this failed task, based on the exception"� Can do other stuff on task failure! But not on success.� With #3553, can "catch" exception and make the task succeed

No PR - async subworkflows - use Python async syntax for more expressivity

7 of 10

Checkpointing

Before draft PR #3535

Result goes into checkpoint db

Failures are ignored so task is retried next time - failure is assumed transient

All results are loaded from checkpoint db - results in DB assumed good

After draft PR #3535:

Power-user hook: what is checkpointed (and how, if you want own DB format)� Power-user hook: what is loaded� Policy expressed in Python

8 of 10

Retry handlers

Before draft PR #3553:

Python hook that can inspect a failed task (args, and exception) and make a decision about whether to retry or not

Can also modify task (c.f. Logan's iterative task generation that does this in a different way) that will be retried

Choice: rerun task (maybe modified), fail task with exeception

After draft PR #3553:

Choice: rerun task, fail task, complete task with result (from handler code)

Observability of exceptions => (eg) compute mean if 95% of inputs succeeded - "I don't care if a few dependencies failed"

9 of 10

@async_app - I would like to prototype

Fantasy syntax:

@async_app�def my_sub_workflow(x):� y = await task1(x)� try:� z = await task2(y)� catch FooError:� z = 7� r = await task3(z)� return r

this might look much nicer than join_app for some situations

uses Python coroutine syntax to express control flow (vs DFK dependencies) - more Pythonic control flow (try/catch, if, while)

Trades concurrency for control flow

Probably works with other app types

(Q: is this enough to implement the Parsl retry mechanism as a loop?)

(Q: does this supercede #3553 retry handlers behaviour?)

10 of 10

My quest for Pythonic abstractions

All these ad-hoc features feel a bit messy…

Rather than implementing "features" in the workflow language…

… exploit user familiarity with Python

Two themes:

1. apply some Python code to all tasks (checkpointing, retry handlers, file staging)

2. let individual tasks be richer (beyond join_app!) by expressing more Python inside a task (exceptions, loops, if, observation of intermediate results)

… in the context of this talk, for doing interesting things with task outcomes

… but more general applicability

1 of 10