1 of 28

ANDREW THOMPSON - athompson@carto.com

SOLUTIONS ENGINEER

SERVERLESS ARCHITECTURES FOR GEO

SLIDES:�http://bit.ly/foss4g-carto-serverless

2 of 28

3 of 28

Talk Idea vs Reality

HEY! Have you all heard of serverless?

No! Wow!

Uh…�Duh!

HEY! Have you all heard of serverless?

4 of 28

Overview

A bit about what CARTO is
A bit about what Serverless and AWS Lambda are
A bit about how CARTO and Serverless are a match made in heaven
A bit about real world CARTO client-facing projects where a Serverless architecture saved the day

Twitter
City of Philadelphia

5 of 28

WHO AND WHAT IS CARTO?

Customers

PIONEERS IN LOCATION INTELLIGENCE

1,200

End-users

300K

Team members

120+

CARTO leads the World of Location Intelligence through a team of more than 100 renowned geo experts, engineers and entrepreneurs.�It actually began in 2008 as a data visualization boutique, but then turned into a product company in 2012, when we launched CartoDB. Since then our growth and impact in the market has been tremendous:��More than a 250% yearly revenue growth, 1,200 customers, 200,000 users and $30 million in funding by investors like Accel Partners (Facebook, Spotify, Kayak) or Salesforce Ventures prove the rise of location intelligence in the market with CARTO as the pioneer of this huge trend.

With headquarters in New York, we have also offices in Silicon Valley, Madrid, London, Estonia and representatives in LATAM and APAC. Our current network of more than 100 partners (Accenture, BCG…) also act as CARTO representatives in certain markets.

6 of 28

BUILDER

A self-service business user application for spatial analysis and visualization.

Built in drag and drop analytics and easy cartography
Both in the cloud and on premise
Publish interactive dashboards that update analysis and filter live

BRINGING LOCATION INTELLIGENCE TO �THE MASSES

Builder is a self service business user application for analyzing data and deploying dynamic location intelligence applications.

Available both on the cloud and on premise, Builder offers users the fastest path to delivering business outcomes
Analyze multiple layers of data and understand how they relate, impact each other, and understand how this will change in the future
Deploy location intelligence applications with drag and drop elements that update analysis, filter results, and instantly visualize multiple criteria with auto-style
Instantly share public, private, or password protected location applications that update in real time anywhere with one click

Builder quickly answers questions such as:

Where should I place my next store or distribution center?
How can I optimize my sales territories to maximize profits?
Where is the optimum location to reach my target audience?
Where are my economic outliers and high risk clusters?

7 of 28

ENGINE

POWER YOUR APPS WITH LOCATION INTELLIGENCE

The one-stop shop for developers to power location applications in their organization.

Easy-to-use, web native APIs & SDKs
Location Data Services Included
Built for developers and designers
Play nicely with other tools

8 of 28

ONE STOP SHOP BACKEND �FOR GEO-APPS

Data Importing and Storage: Import API and PostgreSQL database
Spatial Analytics: SQL and Analysis API with PostGIS
Location Data Services: Data Services API
Maps and Tiling: Maps API
User Interaction: CARTO.js�(built on Leaflet)
Native CARTO Mobile SDK

9 of 28

…unless you also need some server-side code…

10 of 28

Enter…

11 of 28

I LIKE WRITING CODE...��...BUT DEVELOPING IS FRUSTRATING

(putas dependencias!)

12 of 28

13 of 28

WHAT IS A SERVERLESS FUNCTION?

A stateless, event-driven “code bitlet”.

No provisioning, limited managment
Auto-scaling
Zip code and dependencies and upload to AWS Lambda, or edit inline, and you’re good to go

Builder is a self service business user application for analyzing data and deploying dynamic location intelligence applications.

Available both on the cloud and on premise, Builder offers users the fastest path to delivering business outcomes
Analyze multiple layers of data and understand how they relate, impact each other, and understand how this will change in the future
Deploy location intelligence applications with drag and drop elements that update analysis, filter results, and instantly visualize multiple criteria with auto-style
Instantly share public, private, or password protected location applications that update in real time anywhere with one click

Builder quickly answers questions such as:

Where should I place my next store or distribution center?
How can I optimize my sales territories to maximize profits?
Where is the optimum location to reach my target audience?
Where are my economic outliers and high risk clusters?

14 of 28

ETHNIC DIVERSITY OF THE US SHOWING ONE POINT FOR EVERY 100 PEOPLE OF A GIVEN ETHNICITY

SERVERLESS WITH CARTO, WHY?

Data needs to be “pulled” or “pushed” to CARTO to be rendered and analyzed, and sometimes pulling and pushing is complicated...�
Data needs to be analyzed, but you want to do analysis in something other than PostGIS SQL...�
CARTO can be your full geospatial backend, except when you need to run non-SQL code for authentication, API keys, etc...

15 of 28

CARTO SERVERLESS ARCHITECTURE IDEAS

16 of 28

BIG DATA INGESTION & ETL PIPELINES

For when you’re too big,

even for our Import API

17 of 28

Batch �SQL API

Data for Code Libraries

ANALYSIS & SEQUENTIAL MODELS

Iterative computation�With your own tools

Analysis In Lambda

Display & Query

18 of 28

NEW KINDS OF APIS/APPS/MIDDLEWARE USING CARTO BEHIND THE SCENES

19 of 28

REAL CLIENT PROJECTS,�REAL OPEN SOURCE CODE

20 of 28

CUSTOMER USE CASE: TWITTER EVENTS

Example links:�Super bowl 50�Super Bowl 51�Champions League ‘17

21 of 28

THE TASK:

UPLOAD MASSIVE LIVE TWITTER DATA TO CARTO

Twitter wants real-time, live maps
Important events generate tons of tweet data; GNIP API can fail; so need to grab tweets every couple minutes incrementally
Old process: Code up a Tweet-grabber script, SSH & run on our “Script Server”
Script server sometimes fails mid-event (or you worry about it doing so); disorganized, tech debt; hard to monitor, reuse, etc

22 of 28

SQL API incremental imports

GRAB LIVE GNIP DATA

Get Tweets from GNIP API

Monitoring

No Crashes!� ^__^

Source Code Here!

23 of 28

CUSTOMER USE CASE: CITY OF PHILADELPHIA

24 of 28

THE TASK:

CREATE A REVERSE PROXY BETWEEN SOCRATA AND CARTO API

Philadelphia had Socrata for many years, encouraged internal and external developers to use its API
Wanted to move from Socrata to CARTO, but vendor lock-in concerns
Need a translator between Socrata API requests existing apps would make, to new CARTO API that would store the data
Minimal maintenance, “set and forget”, don’t want a server
Wanted to control API endpoints to avoid problem in the future

25 of 28

PHILADELPHIA SERVERLESS ARCHITECTURE

CKAN OpenDataPhilly.org catalog

data.phila.gov/socrata

data.phila.gov/carto

1. Transform Socrata�2. Pass CARTO request�3. Transform response to Socrata (if needed)

Open Data Store (csv, shp, geojson, etc)�phl.carto.com account

Source Code Here!

26 of 28

USING SERVERLESS FRAMEWORK

27 of 28

EXTRA BALL

https://medium.com/@Stuart_Lynn/carto-matplotlib-amazon-aws-popup-joy-dd49fe44131d

28 of 28

ANDREW THOMPSON · athompson@carto.com

SOLUTIONS ENGINEER

THANK YOU FOSS4G BOSTON!

SLIDES:�http://bit.ly/foss4g-carto-serverless��Stop by our booth!