1 of 30

BandSquare

Tour of a Clojure + Datomic app

Valentin Waeselynck

Mail: val.vvalval@gmail.com

Twitter: @val_waeselynck

Github: vvvvalvalval

2 of 30

About BandSquare

  • An audience engagement platform for artists
    • Artists create campaigns, publish them on their social networks, and let their fans subscribe
    • Example: https://www.bandsquare.com/campaign/ma2x-1/landing
  • Provides artists with:
    • Customizable, conversion-optimized landing pages
    • Analytics / Audience insights
    • A CRM / segmentation tool
    • Example (Ben l’Oncle Soul): https://bo.bandsquare.com/#/home/org/55e481bf702d070300d32976
  • Web & Data business

3 of 30

BandSquare’s Stack

4 of 30

BandSquare’s Stack - History

  • March 2014 - Dec 2015: the Dark Ages
    • MEAN: MongoDB, Express, AngularJs, NodeJs
    • Lightweight, comfortable to start fast, no questions asked
    • Super Hype in startups
    • No tests
    • Catastrophic as business logic grew more advanced, esp. MongoDB
  • Dec 2015 - now: Renaissance
    • Moved whole backend to Clojure + Datomic
    • 1 month migration, 11 kLoc -> 9kLoc

5 of 30

Backoffice (static site)

bandsquare.com

(static site)

Backend server

(Clojure - Java)

Analytics

Mandrill

(email delivery)

Stripe / Paypal

(online payment)

Digitick

(ticketing)

(AMQP - CloudAMQP)

AWS DynamoDB

Datomic

RedisLabs

KV Stores / Cache

AWS Elastic Beanstalk

Netlify

Netlify

AWS S3

Static storage

Scheduler

(SetCronJob)

Social Networks

URL Shorteners

etc.

6 of 30

BandSquare’s Stack - Why ?

  • To build an advanced, robust system
  • … without sacrificing productivity / agility
  • … with very little manpower

7 of 30

BandSquare’s Stack - Why ?

  • Why Datomic ?
    • Flexible schema
    • Expressive queries
    • Ease of testing
    • Programmable
  • Why Clojure ?
    • Access to Datomic
    • Data manipulation
    • Productivity / REPL

8 of 30

Datomic

9 of 30

Datomic: DB values and connections

2 notions:

  • Database value: immutable, distributed data structure, for reading
    • For reading
    • Logically equivalent to a sorted set of Datoms
  • Connection: remote, centralized, managed reference to a DB value
    • For writing, and obtaining the DB value
  • Agent analogy:

(agent #{[42 :movie/title "Gladiator"]

[43 :movie/title "Babel"]

[101 :actor/name "Russell Crowe"]

[42 :movie/starring 101]

...})

10 of 30

Datomic: architecture

11 of 30

Datomic: the database is effectively local

  • Querying happens in application process
  • Immutability => lots of caching
    • DB values share structure, and are lazily loaded in app address space
  • I/O mutualized across queries
    • Each storage access to a Datom brings along a neighborhood of this Datom
    • => no N+1 problem
  • Consequences:
    • Breaking big queries into simpler queries
    • Low latency
    • Horizontal read scalability.

12 of 30

Datomic querying: Datalog and Entities

;; Entities: find the director name of Gladiator

(def gladiator

(d/entity db [:movie/title "Gladiator"]))

(def director (get gladiator :movie/director))

(get director :director/name)

=> "Ridley Scott"

;; or more concisely

(-> gladiator :movie/director :director/name)

=> "Ridley Scott"

;; Entities have a Map-like interface

(select-keys gladiator

[:db/id :movie/title :movie/director])

=> {:db/id 42

:movie/title "Gladiator"

;; that's another Entity

:movie/director {:db/id 23}}

;; Datalog: find directors of Russell Crowe

(d/q

'[:find [?director ...]

:in $ ?actor-name :where

[?actor :actor/name ?actor-name]

[?movie :movie/starring ?actor]

[?movie :movie/director ?director]]

db "Russell Crowe")

=> #{23}

13 of 30

Datomic querying: Datalog and Entities

Datalog:

  • Query DSL
  • logic
  • Pattern-matching, joins

Entities:

  • Data structures
  • Ordinary control flow
  • Navigational
    • Like a cursor in the DB

14 of 30

Datomic querying: Datalog and Entities

42

:movie/title

"Gladiator"

23

:movie/director

:director/name

"Ridley Scott"

101

:actor/name

"Russell Crowe"

:movie/starring

15 of 30

Datomic superpowers

  • db.asOf(<time>): obtain a past version of the database
    • No information is ever lost!
  • db.with(<writes>): speculative writes, obtain a local version of the database as if some writes had been made
  • Flexible schema: specifies only attributes, not entity types
    • Attributes sharing
    • ‘Sparse tables’, graphs
  • API based on data structures (not text)
  • Non-remote querying

16 of 30

Architecture

17 of 30

Design

Design goals:

  • RPC-style API
  • transport-agnostic
  • Ease of testing
  • Interactive workflow (REPL)
  • Reproducible state
  • One codebase for all programs (server, scripts, data exploration, ...)

Constraints:

  • Legacy AngularJS client

18 of 30

Architecture: contexts

(def ^:dynamic conn ...)

(def ^:dynamic mail-sender ...)

(defn send-email! [customer data]

(mail/send-mail! mail-sender

(:customer/email customer) data)

(d/transact conn

[{:mail/id ...

:mail/to ...}]))

(defn send-email!

[{:as ctx :keys [conn mail-sender]}

customer data]

(mail/send-mail! mail-sender

(:customer/email customer) data)

(d/transact conn

[{:mail/id ...

:mail/to ...}]))

BAD

GOOD

19 of 30

Architecture: contexts

{:conn ;; Datomic Connection

#datomock.core.MockConnection{:a_db #<Atom@1a588bc3: datomic.db.Db@413e79b5>,

:a_txq #<Atom@9ef48b4: nil>}

:db ;; Datomic current database value

datomic.db.Db@413e79b5

:mail-sender ;; mail sending service

#bs.toolkit.mail.MockMailSender{:badRecipients #{"bad.email@gmail.com"},

:latency 10},

:stripe ;; Stripe (payment) client service

#bs.services.stripe.MockStripeClient{:latency 10}

:local-cache ;; in-memory caching service

#bs.toolkit.stores.AtomTtlCache{:a #<Atom@473d9abb: {}>},

:now ;; current time

#inst "2016-06-26T13:31:43.936-00:00"}

Anatomy of a context:

20 of 30

Architecture: contexts

Services declared via protocols / interfaces

(defprotocol MailSender

(send-template!* [this data])

(send-text!* [this data])

)

(defprotocol StripeClient

(call [this endpoint req]))

21 of 30

Architecture: the api multimethod

(defmulti api (fn [ctx proc-name params data]

proc-name))

22 of 30

Architecture: the api multimethod

;; registering an endpoint (implements the api multimethod)

(api/endpoint ::add-concert-to-campaign

{:middlewares [mid_user-may-access-campaign] ;; authorization middleware

:data-schema {:id sc/Str, :venueName sc/Str}} ;; Prismatic Schema

(fn handler

[{:as ctx, :keys [conn db]} ;; context - ‘Dependency Injection’

{:as params, :keys [campaignId]} ;; params: resource identification

{:as concert-data, :keys [id venueName]}] ;; data: what to write

(let [campaign (find-campaign-by-id db campaignId) ;; finder fn: (db,data) -> Entity

tx (tx_add-concert-to-campaign campaign concert-data)

{:keys [db-after]} @(d/transact conn tx)

concert (find-concert-by-id db-after id)]

(cl_concert concert) ;; clientizer fn: Entity -> serializable data structure

)))

;; ... called by HTTP route

(POST "/campaigns/:campaignId/concerts" ::concerts/add-concert-to-campaign)

23 of 30

Architecture: data flow

HTTP request

API request

Entities

conn

Data response

HTTP response

Context injection

Parsing

HTTP Routing

Handling

Finding Entities

Clientization

Serialization

Writes

Side effects

Business logic

API

24 of 30

Business logic: Entities in, Entities out

(require '[datomic.api :as d])

(defn past-concerts-of-campaign

"Given a campaign, find those concerts of the campaign which have already happened."

[now campaign]

(let [db (d/entity-db campaign)]

(->> (d/q '[:find [?concert ...] :in $ ?campaign ?now :where

[?concert :concert/campaign ?campaign]

[?concert :concert/time ?time]

[(< ?time ?now)]]

db (:db/id campaign) now)

(map #(d/entity db %))

)))

25 of 30

Business logic: Entities in, Entities out

(require '[datomic.api :as d])

(defn past-concerts-of-campaign

"Given a campaign, find those concerts of the campaign which have already happened."

[now campaign]

(->> campaign

:concert/_campaign

(filter (fn [concert]

(-> concert :concert/time (time/before? now)))))

)

26 of 30

Architecture: high-level overview

27 of 30

Contexts: the Universe as an Input

Database connection

Payment service

Email service

Payments

Manager

Signup

Manager

Application

External services (universe)

Application components

Application

Traditional, DI-style system structure

Request / Event

Response

28 of 30

Contexts: the Universe as an input

Request / Event

Response

Universe

Application

Request / Event

Universe

  • Passive
  • Configuration-free
  • Lifecycle-free
  • REPL-friendly

(a.k.a context)

side-effects

(cf Hexagonal Architecture)

29 of 30

Forking the universe

  • fork() operation on contexts / universes
    • Made possible by Datomic’s speculative writes
    • Let’s you ‘branch’ a universe off another universe, similarly to Git branches
  • Very REPL-friendly
  • Greatly simplifies system-level testing:
    • No more setup/cleanup phases
    • Instead, begin with a starting-point universe, and test by exploring a tree of possibilities
  • http://vvvvalvalval.github.io/posts/2016-01-03-architecture-datomic-branching-reality.html

30 of 30

Questions?

Valentin Waeselynck

Mail: val.vvalval@gmail.com

Twitter: @val_waeselynck

Github: vvvvalvalval