impact-oriented programming

impact-oriented programming




  • Tel Aviv University
  • Ministry of Defense
  • Introduction to Information Security
  • Google
  • Magic Leap
  • SFERA, All Things Urban, and more

the motivation

  • Why make a workshop? Advanced System Design has a project bigger than most
  • Except...
  • A good friend of mine studies neuroscience here in Tel Aviv university
  • Their digital infrastructure is… bad
  • Why not help, instead of building a toy project?

The Inspiration

  • So then, the visual language is superheroes
  • You have superpowers — use them for good
  • Like the Null Pointress
  • (I put way too much effort into this workshop)
  • (I expect you to, too :-)


  • Lesson 1 — Abstracts
    • Input
    • Process
    • Output
  • Lesson 2 — Concretes
    • Workflow
    • Platforms
    • Empathy


  • 12/03, 15:00–17:00 Lesson 1
  • 19/03, 15:00–17:00 Lesson 2
  • 24 students / 6 projects, so a team of 4 students per project
  • I assembled the teams and matched the projects under various constraints
  • If two of you still want to swap, let me know
  • 22–26/03 Meet the client
  • 03–07/05 Demo day
  • 21–25/06 Handoff
  • Bi-weekly (online) status reports
  • 23/04, 15:00-17:00 Knowledge exchange


  • Communication via email and Hangouts
  • Slides, tutorials and other documents via gSuite
  • Version control via GitHub
  • Deployment via Google Cloud Platform
  • $50 cloud credit here!
  • Use your email
  • Lesson screencasts for people who can't attend


  • Effective Altruism
  • Platforms vs. Solutions
  • The Grand Abstraction

effective altruism

  • A really nerdy way to do good
  • No, seriously
  • Every dollar you donate in the US is worth less than in Africa
  • GiveWell really shines here
  • But what's more impactful than your money is your time
  • Take education, for example
  • Access ⭑ Infrastructure ⭑ Pedagogy ⭑ Management
  • Key lessons: be pragmatic and measure what matters
  • Optimizing anything but the bottleneck is suboptimal
  • 80,000 Hours is the best* career advice you'll ever get

platforms vs. solutions

  • So we're data-driven and effective; that's good
  • But that only accounts for the past; for what we already know
  • What about the future? What we don't know yet?
  • Humans are notoriously bad at predicting the future
  • Don't build a solution for an uncertain market
  • Instead, build a platform for generating new solutions
  • When the future arrives, you'll be ready
  • This doesn't come instead of a healthy, agile approach
  • The design and the strategy are complementary

Platform Design

  • How hardcoded are your logics?
  • On one hand, you can build your system as a monolith
  • On the other, you can build it as interchangeable modules
  • How coupled are your interface and your implementation?
  • Your tech is definitely good for now
  • But can your system adapt to change (e.g. scale)?
  • How hardcoded are your utilities?
  • Are they developed as part of the system and rely on it?
  • Or are they developed separately, as external libraries?

Platform Strategy

  • However, we don't want to build a platform in a vacuum
  • First, because we must deliver some solution, or die
  • Second, because who's to say we got the problem right
  • Waterfall vs. Agile development
  • Requirements → design → implementation → deployment
  • Or, small iteration thereof
  • Of course, the first iterations will be "quick and dirty"
  • MVP — minimum viable product
  • But then we iterate and calibrate according to reality

the grand abstraction

  • Most problems have a similar structure (and a similar solution)
  • Collecting some input
  • App data ⭑ sensor data ⭑ web crawling
  • Processing said input
  • Parsing ⭑ analysis ⭑ storage
  • Emitting some output
  • APIs ⭑ CLIs ⭑ GUIs
  • Key lessons: decouple and disaggregate your systems
  • Build for the future, not for the present


  • An important lesson from compilation (although in our case, there are more layers)















  • Smartphones could've been a solution; instead, it's a platform
  • The accelerometer could've been added as an app
  • Instead, if was added as a service
  • This enabled surprising use-cases such as
    step-counting apps or emergency backups
  • In general, a system providing a collection of services can be disaggregated into a bunch of separate atomic services
  • Very similar to platform design
  • Except the modules are agnostic, and expose a generic interface instead of a tailored response







shared medium










system architecture












message queue




color image

color image

color image

depth image







object saver


advanced system design :: lesson 9 :: system architecture 2


/ 91

Building for the Future

  • If we can't predict the future, how can we build for it?
  • Platform design: building for change
  • Agile development: iterating in small steps
  • Disaggregation: exposing modules as atomic services
  • But more than that — just admitting we don't know
  • This invokes empathy, for users and maintainers both
  • This in turn means better tooling
  • Kinda the difference between AirBnB and your home
  • More than that, it motivates you to stay in touch
  • Not only with your field, but with innovation in general


  • Effective Altruism: be pragmatic and measure what matters
  • Platforms vs. solutions (and design vs. strategy)
  • Platform design: monolith vs. modules ⭑ interfaces ⭑ libraries
  • Platform strategy: agile vs. waterfall
  • The Grand Abstraction: input → process → output
  • Decoupling and disaggregation: exposing modules as atomic services
  • Building for the future: caring and staying in touch
  • Bonus: the modules game

the modules game

  • Set aside 30 minutes (e.g. bus ride, doctor's appointment, etc.)
  • For 5 minutes, look for new, unfamiliar libraries
  • Google, Awesome Lists, YouTube
  • Then, select 5 of them
  • Spend 5 minutes reading each of their quickstarts
  • After a week, you'll have seen 35 new modules
  • Of them, 10 will be useful, and 2 will change your life
  • Highly recommended for this workshop
  • We're solving difficult problems, not reinventing the wheel


  • Data collection
  • Formats
  • RESTful APIs

data collection

  • The first challenge is understanding what our data model is
  • Let's say we have an analytics system
  • Is the number of visitors per month a datum?
  • How about a single view, from Israel, on a mobile device?
  • Let's say we have a system for recording physics experiments
  • One experiment tracks a spinning ball's location on a disc
  • Is the ball's location the data?
  • What if I'd tell you the raw images are the data?
  • Different experiments can feed the same system
  • Even something as simple as measuring radioactivity


  • Once we have some data, we need a data structure
  • Do we write a class, or use a dictionary?
  • Do we use a datetime object, or an integer?
  • The question is, what formats do we apply to our data model?
  • We can always have a proprietary serialization protocol
  • To dump the data to file or pass it on to another program
  • However, standard formats are language-agnostic
  • Standard libraries for serialization and deserialization
  • An already thought-through design
  • All have some trade-offs, so you need to know a few

JSON, BSON and Protobuf

  • If you're dealing with simple data, go with JSON
  • It's simple, readable, and extremely flexible
  • Mostly because it supports lists and dicts, like Python
  • However, it doesn't support bytes, datetimes and more
  • You can be creative (e.g. base64, unix timestamps)
  • Or, you can use BSON — Binary JSON
  • Just please don't use XML :[
  • And if you'd like structured data, check out Protobuf
  • You define the data model in a .proto file
  • You then auto-generate code to serialize/deserialize it

>>> obj = {'n': 1, 's': 'Hello, world!'}

>>> data = json.dumps(obj)

>>> type(data)

<class 'str'>

>>> json.loads(data)

{'n': 1, 's': 'Hello, world!'}

>>> obj['b'] = b'Hello, world!'

>>> obj['d'] =

>>> data = json.dumps(obj)

TypeError: Object of type bytes/datetime is not JSON serializable

>>> data = bson.dumps(obj)

$ cat user.protobuf

message User {

string name = 1;

repeated string email = 2;


$ protoc -I. --python_out=. user.proto

# creates

>>> from user_pb2 import User

>>> user = User(

... name = 'Dan Gittik',

... emails = [

... '',

... '',

... ],

... )

>>> data = user.SerializeToString()

>>> new_user = User()

>>> new_user.ParseFromString(data)

restful apis

  • So our data is language-agnostic, but our functions aren't
  • How can we pass this data between two programs?
  • Specifically, a local client and a remote server?
  • Again, we can develop a proprietary protocol
  • Or we can use an existing, language-agnostic solution
  • Namely, HTTP — the language of the web


  • A simple request-response architecture
  • A request specifies a method (verb) and URI (object)
  • It can also have headers (metadata) and a body (data)
  • A response specifies a status (200 OK, 404 Not Found)
  • And some headers and body as well
  • A lot of best practices we won't really cover
  • Cacheability
  • Statelessness

app = flask.Flask()

@app.route('/users', methods=['GET', 'POST'])

def users():

if flask.request.method == 'GET':

users = db.users.all()

return flask.jsonify([{'id':} for user in users])

if flask.request.method == 'POST':

data = flask.request.get_json()

user = db.users.create(name=data['name'])

return flask.jsonify({'id':})

@app.route('/users/<id>', methods=['GET', 'PATCH', 'DELETE'])

def user(id):

user = db.users.get(id)

if flask.request.method == 'GET':

return flask.jsonify({'name':})

if flask.request.method == 'PATCH': = flask.request.get_json()['name']

if flask.request.method == 'DELETE':


return ''

>>> response = request.get('http://localhost:8000/users')

>>> users = response.json()

>>> users


>>> data = {'name': 'Alice Lovelace'}

>>> response ='http://localhost:8000/users', json=data)

>>> user_id = response.json()

>>> user_id


>>> request.get('http://localhost:8000/users/1').json()

{'name': 'Alice Lovelace'}

>>> data = {'name': 'Null Pointress'}

>>> request.patch('http://localhost:8000/users/1', json=data)

>>> request.get('http://localhost:8000/users/1').json()

{'name': 'Null Pointress'}

>>> request.delete('http://localhost:8000/users/1')

>>> request.get('http://localhost:8000/users/1')

<Response [404]>


  • Define a data model
  • Choose a data format
  • Simple data works well with JSON/BSON
  • Complex data can be structured in Protobuf
  • Communicate the data via a RESTful API


  • Plugins
  • Databases
  • Cloud Services


  • Again: how hardcoded are your logics?
  • Take a web server for example
  • Some problems are rigid, like working with HTTP
  • Some problems are custom, like your routes and views
  • What does Flask do?
  • Lets you write snippets of code, and plugs them in
  • If the problem is recurring in nature — don't just solve it
  • Write an infrastructure that makes solving it easy
  • Then solve it; then solve a bunch of similar problems
  • Better yet — find an infrastructure that does that for you :]

Larry Wall

At one extreme, so-called “4th generation languages” make it easy to do some things,

but nearly impossible to do other things. At the other extreme, so-called “industrial-strength” languages make it equally difficult to do almost everything.




  • Files?
  • But this is hard to manage, search, scale…
  • Relational databases use flat tables
  • Data is normalized using relations
  • SQL — Structured Query Language
  • Powerful query language; cumbersome data model
  • Document databases use documents (dictionaries)
  • The data structure is arbitrarily complex (and nested!)
  • Querying is weirder, but you get used to it
  • Other NoSQL solutions exist (e.g. Redis, a key-value store)





Alice Lovelace

Dan Gittik










The Curious Case of the Database in 2020

  • Storage technology is so good, that for all intents and purposes, any database you choose will handle your data well
  • So the question is not which database is more efficient
  • The question is which database is most convenient
  • That's not to say we should be careless
  • For example, don't store large binary blobs in any database
  • Instead, store them on disk and keep their path
  • But you get the point: be pragmatic and measure what matters

Cloud APIs

  • You don't really need a server — you need compute
  • What if you could lease a server?
  • Better yet, what if you could lease compute?
  • Why just compute? Why not storage or other services?
  • Economies of scale kinda deal
  • Google up and solved most of your problems already
  • It's really a question of curation
  • Still — it costs money
  • Something a lot of startups tend to forget
  • So use it only where it's cost-efficient


  • Plugins: write/use an infrastructure for your solutions, then your solutions on top
  • Make the simple case simple and the hard case possible
  • Databases: relational vs. document (and many more)
  • Use whatever floats your boat, seriously (but do use it properly)
  • Cloud APIs: delegate to Google when it's cost-efficient


  • RESTful APIs, again
  • Command-line interfaces
  • Graphical user interfaces

RESTful APIs, Again

  • The processed results need to be exposed and visualized
  • Unfortunately, they are often baked into their views
  • That's why we scrape the web
  • Not very extendable, either
  • How do you add mobile or desktop apps?
  • How do you add libraries for 10 different languages?
  • Good ol' compilation lesson
  • Expose the data via a RESTful API
  • GUIs, CLIs, libraries etc. simply consume it




The Gulf of Execution

  • The gulf of execution is the difference between the intentions of the users and what the system allows them to do
  • Reducing the gulf of execution not only makes the task at hand easier, but also removes cognitive load, and shifts the cost-efficiency ratio, allowing users to innovate
  • We can't account for everything
  • So we should always expose our data to the users,
    and empower them to solve their own problems
  • Don't give them fish — give them fishing rods
  • And a way to customize their fishing rods at that

Command-Line Interfaces

  • The often overlooked art of CLIs is a great way to expose data
  • It's usually much easier to implement and to automate
  • It requires some skill — but users are not stupid
  • Make interfaces that are easy to use correctly, and hard to use incorrectly
  • מור״ק time!
  • Seriously though, have a look at click or fire
  • You can even add auto-completion!

Graphical User Interfaces

  • Of course, the holy grail is a good GUI
  • The de-facto standard is a web-based interface
  • A myriad of technologies: HTML, CSS, JS, jQuery, React...
  • All are pretty easy to grasp, at least on a basic level
  • Kind of like driving?
  • The challenge is actually good UX
  • Xenophanes: "horses would draw their god like horses"
  • Design wise, it's MVC vs. F/B
  • Model-view-controller decouples the components
  • Frontend/Backend disaggregates the UI from the API


  • The model is responsible for the data storage
  • The views are responsible for the data representation
  • The controller operates the model and the views
  • Take Django, for example
  • You define your model in
  • You define your views in
  • You bind them together in
  • Drawback: the loop is closed in one program (i.e. server)
  • Easy to start (and in many cases, this is good enough)
  • But the data is baked into the view, and the UI is limited


  • Another approach is to have the backend (server) be an API
  • So the data is easy to work with, but hard to look at
  • The frontend is then yet another client consuming it
  • There are some pretty amazing client-side technologies
  • Most of them happen in the browser and use Javascript
  • It's harder — but for a complex system, it's the right way to go


  • RESTful APIs: expose the data for CLIs, GUIs and libraries to consume
  • This reduces the gulf of execution and empowers users
  • Command-line interfaces: they're great, use 'em
  • Graphical user interfaces: MVC vs. F/B

key takeaways

  • Be pragmatic and measure what matters
  • Build platforms instead of solutions
  • The grand abstraction: input → process → output
    • Decoupling, disaggregation, building for the future
  • Define a data model and choose a data format
  • Build/use infrastructures with plugins
    • Prefer convenience* and don't reinvent the wheel
  • Expose everything through APIs
    • Write CLIs and MVC or F/B GUIs

impact-oriented programming

impact-oriented programming




  • Project structure
  • Version control
  • Testing
  • Documentation

Project Structure

  • We're working on the Foobar project in the foobar/ directory
  • In it, there's a foobar/ directory with the code
  • This is actually a python package (
  • Next to it, there's a with the project settings
  • A requirements.txt specifies the dependencies
  • A provides some documentation
  • And the tests are in the tests/ directory
  • Of course, there's a lot more
  • .git, scripts/, .tarvis.yml and more
  • To manage your environment, use pyenv and virtualenv

$ git clone

$ cd foobar/

$ ls

foobar/ requirements.txt tests/

$ pyenv install 3.8.1

$ pyenv local 3.8.1

$ virtualenv .env --prompt='[foobar] '

$ source .env/bin/activate

[foobar] $

[foobar] $ pip install -r requirements.txt

[foobar] $ pytest tests/

Version Control

  • Goal: keep the project's changes history
  • Local version control: file_old_old2.tmp; RCS
  • Problem: collaboration
  • Centralized version control: CVS, Subversion (SVN), Perforce
  • Problem: slow, single point of failure
  • Distributed version control: git, mercurial
  • Fast, and everyone has a copy of the repository


  • Developed by and for the Linux Kernel community
  • So it should be able to handle your project
  • A powerful CLI (GUIs are available, too)
  • To create a project, git init or git clone
  • Work work work
  • To save a change, git add some files and git commit it
  • Repeat, until you're ready to git push (or git pull)
  • A lot more, like checkout, branch, merges, .gitignore
  • Also, we'll be using GitHub
  • Be sure to go over the Version Control tutorial


  • Goal: asses the program's quality
  • Comes in a variety of forms and flavors
  • Unit tests, integration tests, stress tests, penetration tests
  • Actual goals (except correctness)
  • Regression: detect breaking changes
  • Documentation: effectively, code samples
  • Design: outlines the system flows and edge-cases
  • Don't insist on tests — sometimes, a script is also OK


  • My favorite Python library
  • Write a file with a test_foo function
  • Use assert to check that something is true
  • pytest will take care of the rest
  • Really cool utilities
  • A context manager that tests for exceptions
  • Fixtures: an interesting way to "ask for help"
  • For hard cases like testing FS / IO

def test_cube():

assert cube(2) == 8

assert cube(-2) == -8

def test_div():

with pytest.raises(ZeroDivisionError):

div(1, 0)

def test_capitalize(tmp_path):

f = tmp_path / 'file.txt'

f.write_text('hello, world!')


assert f.read_text() == 'HELLO, WORLD!'


def obj():

# long setup

return obj

def test(obj):

# do stuff


def obj():

# setup

yield obj

# teardown


  • Clean code
  • Be consistent; respect PEP8; use linters/Black
  • Add comments
  • But not stupid ones
  • Nobody cares about your grammar
  • But mostly, add a (it's in markdown)
  • Installation — how do I start
  • Usage — what do I do
  • Development — how do I run tests, extend it, etc.


  • Drivers
  • Plugins
  • Unix utilities


  • Decouple the interface from the implementation
  • The interface should define a clean, agnostic API
  • For example, create_foo, get_foo and delete_foo
  • This should delegate to a driver
  • Literally, self._driver.create_foo(*args, **kwargs)
  • And any implementation-agnostic code
  • Logging, argument handling, assertions
  • The driver is selected by a URL
  • So, sql://localhost:5432/db works with PostgreSQL
  • But we have memory:///, mongo://…/ and more

class Database:

def __init__(self, url):

self._driver = find_driver(url)

def create_foo(self, x):

id = self._driver.create_foo(x)

return Foo(id, x)

def find_driver(url):

for scheme, cls in drivers.items():

if url.startswith(scheme):

return cls(url)

raise ValueError(f'invalid url: {url!r}')

class MemoryDriver:

def __init__(self, url):

self.foos = {} = itertools.count()

def create_foo(self, x):

id = next(

self.foos[id] = {'x': x}

return id

drivers = {'memory://': MemoryDriver, …}


  • Write an infrastructure for plugins instead of hardcoded logic
    • Make the easy case easy and the hard case possible
  • Plugins are pieces of code that adhere to some interface
    • Decorators (like flask)
    • Aspect-oriented programming (like pytest)
  • Recurring pieces of code can be provided as utilities
  • An argument; even an optional one (like pytest)
  • If the simplest cases don't require code, introduce layers
  • Progressive disclosure: parametrization — then code
  • Don't get carried away into developing a DSL

>>> process(data, 'foo', x=1)

@algorithm('foo', x=int)

def process_foo(data, x):

return x + data['y']

algorithms = {}

def algorithm(name, **params):

def collect(f):

algorithms[name] = f, params

return f

return collect

def process(data, name, **args):

f, params = algorithms[name]

for key, cls in params.items():

if key not in args:

raise TypeError(…)

value = args[key]

if not isinstance(value, cls):

raise ValueError(…)

return f(data, **args)

>>> process(data, 'foo', x=1)


def process(data, x: int):

return x + data['y']

def import_path(path):

data = path.read_text()

module = types.ModuleType('')

exec(data, module.__dict__)

return module

def process(data, name, **args):

modules = {}

for file in root.iterdir():

module = import_path(file)

modules[] = module

f = modules[name].process

for key, cls in f.__annotations__.items():

return f(data, **args)

def foo(data):

# decrypt data — happens a lot

def foo(context, data):

data = context.decrypt(data)

# carry on

def foo(data, decrypt):

data = decrypt(data)

# carry on

def process(data, name, **args):

context = Context()

return f(context, data, **args)

def process(data, name, **args):

f = …

spec = inspect.getfullargspec(f)

for arg in spec.args:

if arg in utilities:

args[arg] = utilities[arg]

return f(data, **args)

def foo(data):

return data['x']

def foo(data):

return data['x']['y'], data['z']

def foo(data):

return data['x'] + data['y']

def foo(data):

return data['x']**2 + data['y']**2

{"foo": "x"}

{"foo": ["x.y", "z"]}

{"foo": {"op": "add", "args": ["x", "y"]}}

# highly unusual — requires code







Unix Utilities

  • Unix in general is a software design marvel
    • open(), read(), write() and close() work for files
    • But also for directories, and special files
      • /dev/null, /dev/zero, /dev/urandom
      • Drivers — screen, keyboard, speakers
  • Single responsibility principle — do one thing, but do it well
  • Plumbing vs. porcelain
    • Progressive disclosure again: people want a wheel
  • … except some people, that actually need the engine


  • APIs vs. UIs
  • Configurations
  • CLIs
  • GUIs

Henry Ford

If I had asked people what they wanted, they would have told me faster horses.



APIs vs. UIs

  • Once again: users are not stupid
  • They can and will figure out ways to solve their problem
  • Do you really want to help them? Or do you want to cram your solution down your throat?
  • Very common in architecture — "they're using it wrong!"
  • Dear sir, I beg your pardon
  • Can users parametrize or extend your functionality?
  • If not, can they at least access your (their) data?
  • And if so, is it in friendly and standard formats/protocols?
  • Also: can they use just some of your functionality?


  • It's a good practice to extract magic numbers into constants
  • It's an even better practice to extract defaults to an external configuration
  • I recommend using YAML instead of JSON
  • Use for ports, number of workers, intervals, etc.
  • But then, please, don't pass a config dictionary around
  • The configuration should be split into variables ASAP



port: 8000

backlog: 1000

>>> data = path.read_text()

>>> config = yaml.load(data)

def run_server(config):

server = socket.socket()

server.bind((config['host'], config['port']))


def run_server(host, port, backlog):

server = socket.socket()

server.bind((host, port))



  • Should you use a flat (-p/--port) or input (port: _)
  • Why not both?
  • Prompt dangerous operations (are you sure? [y/N])
  • CLIs can have colors, progress bars, ASCII art
  • It's OK to have a CLI for plumbing (in fact, it's great)
  • Meta functionality: dump / load / clean database
  • Debugging functionality (e.g. inspect)
  • Verbose logging


@click.option('-p', '--port', type=int)

def listen(port):

@click.option(…, prompt=True)

if not click.confirm('Are you sure?'):


with click.progressbar(items) as bar:

for item in bar:

# do stuff

>>> t = blessings.Terminal()

>>> print('Hello, world!'))

Hello, world!

>>> important = t.bold_white_on_red

>>> print(important('Hello, world!'))

Hello, world!


  • Imagine yourself using this UI
  • Do you want to log in with a username, email, or Google?
  • Query parameters are great for page parametrization
  • I'm talking about ?q=foo&order=bar
  • It's ugly, but it can be shared and bookmarked
  • Consider visualizations
  • Don't show a table when it should be a chart (it's easy)
  • If you required interactivity, consider Jupyter Notebook / Dash
  • Don't replace your system — just export it into them
  • And please, don't assume; A/B test on your family and friends

The End

  • Teams
  • Projects
  • Timeline


Eran Kirshenboim

Liron Levy

Lital Mandelbaum

Sahar Gavriely

Amit Sova

Dan Sterenson

Sagi Shoshan

Yaheli Cohen

Amram Nevo

Ariel Zamir

Liran Buchny

Noah Harel

Itamar Sharon
Maya Elenblum

Nofar Hadida
Reem Joron

Shira Cohen

Eran Rodrig

Gil Halal

Michael Ciubotariu

Nimrod Kider

Eden Kimelman

Lior Cohen

Maya Man

Noam Balasiano

Onni Amir




Lesson 1


Meet the Client


Lesson 2








Knowledge Exchange




Demo Day














alice lovelace :: Null Pointress
25 :: Tel Aviv

Alice was an undergrad in Computer Science, when one day she dereferenced a radioactive null pointer.

As the bug bit her, its cybertronic energy immersed her, giving her cyberkinesis — the ability to move bits from one place to another.

With great power came existential angst, major depression, and some responsibility; and so, Alice started using her superpowers for the Greater Good — fighting crypto-nazis, rewriting matlab scripts to Python, and developing RESTful JSON APIs for data otherwise only available as XML/SOAP.

A student by day, a hero by night; she does yoga, goes to parties at The Block and has a cat called hexa.

impact-oriented programming - Google Slides