ANDREW THOMPSON - athompson@carto.com

SOLUTIONS ENGINEER

SERVERLESS ARCHITECTURES FOR GEO

SLIDES:
http://bit.ly/foss4g-carto-serverless

Talk Idea vs Reality

HEY! Have you all heard of serverless?

No! Wow!

Uh…
Duh!

HEY! Have you all heard of serverless?

Overview

  • A bit about what CARTO is
  • A bit about what Serverless and AWS Lambda are
  • A bit about how CARTO and Serverless are a match made in heaven
  • A bit about real world CARTO client-facing projects where a Serverless architecture saved the day
    • Twitter
    • City of Philadelphia

WHO AND WHAT IS CARTO?

Customers

PIONEERS IN LOCATION INTELLIGENCE

1,200

End-users

300K

Team members

120+

CARTO leads the World of Location Intelligence through a team of more than 100 renowned geo experts, engineers and entrepreneurs.
It actually began in 2008 as a data visualization boutique, but then turned into a product company in 2012, when we launched CartoDB. Since then our growth and impact in the market has been tremendous:

More than a 250% yearly revenue growth, 1,200 customers, 200,000 users and $30 million in funding by investors like Accel Partners (Facebook, Spotify, Kayak) or Salesforce Ventures prove the rise of location intelligence in the market with CARTO as the pioneer of this huge trend.

With headquarters in New York, we have also offices in Silicon Valley, Madrid, London, Estonia and representatives in LATAM and APAC. Our current network of more than 100 partners (Accenture, BCG…) also act as CARTO representatives in certain markets.

BUILDER

A self-service business user application for spatial analysis and visualization.

  • Built in drag and drop analytics and easy cartography
  • Both in the cloud and on premise
  • Publish interactive dashboards that update analysis and filter live

BRINGING LOCATION INTELLIGENCE TO
THE MASSES

ENGINE

POWER YOUR APPS WITH LOCATION INTELLIGENCE

The one-stop shop for developers to power location applications in their organization.

  • Easy-to-use, web native APIs & SDKs
  • Location Data Services Included
  • Built for developers and designers
  • Play nicely with other tools

ONE STOP SHOP BACKEND
FOR GEO-APPS

  • Data Importing and Storage: Import API and PostgreSQL database
  • Spatial Analytics: SQL and Analysis API with PostGIS
  • Location Data Services: Data Services API
  • Maps and Tiling: Maps API
  • User Interaction: CARTO.js
    (built on Leaflet)
  • Native CARTO Mobile SDK

…unless you also need some server-side code…

Enter…

I LIKE WRITING CODE...

...BUT DEVELOPING IS FRUSTRATING

(putas dependencias!)

WHAT IS A SERVERLESS FUNCTION?

A stateless, event-driven “code bitlet”.

  • No provisioning, limited managment
  • Auto-scaling
  • Zip code and dependencies and upload to AWS Lambda, or edit inline, and you’re good to go

ETHNIC DIVERSITY OF THE US SHOWING ONE POINT FOR EVERY 100 PEOPLE OF A GIVEN ETHNICITY

SERVERLESS WITH CARTO, WHY?

  • Data needs to be “pulled” or “pushed” to CARTO to be rendered and analyzed, and sometimes pulling and pushing is complicated...
  • Data needs to be analyzed, but you want to do analysis in something other than PostGIS SQL...
  • CARTO can be your full geospatial backend, except when you need to run non-SQL code for authentication, API keys, etc...

CARTO SERVERLESS ARCHITECTURE IDEAS

BIG DATA INGESTION & ETL PIPELINES

For when you’re too big,

even for our Import API

Batch
SQL API

Data for Code Libraries

ANALYSIS & SEQUENTIAL MODELS

Iterative computation
With your own tools

Analysis In Lambda

Display & Query

NEW KINDS OF APIS/APPS/MIDDLEWARE USING CARTO BEHIND THE SCENES

REAL CLIENT PROJECTS,
REAL OPEN SOURCE CODE

CUSTOMER USE CASE: TWITTER EVENTS

THE TASK:

UPLOAD MASSIVE LIVE TWITTER DATA TO CARTO

  • Twitter wants real-time, live maps
  • Important events generate tons of tweet data; GNIP API can fail; so need to grab tweets every couple minutes incrementally
  • Old process: Code up a Tweet-grabber script, SSH & run on our “Script Server”
  • Script server sometimes fails mid-event (or you worry about it doing so); disorganized, tech debt; hard to monitor, reuse, etc

SQL API incremental imports

GRAB LIVE GNIP DATA

Get Tweets from GNIP API

Monitoring

No Crashes!
^__^

CUSTOMER USE CASE: CITY OF PHILADELPHIA

THE TASK:

CREATE A REVERSE PROXY BETWEEN SOCRATA AND CARTO API

  • Philadelphia had Socrata for many years, encouraged internal and external developers to use its API
  • Wanted to move from Socrata to CARTO, but vendor lock-in concerns
  • Need a translator between Socrata API requests existing apps would make, to new CARTO API that would store the data
  • Minimal maintenance, “set and forget”, don’t want a server
  • Wanted to control API endpoints to avoid problem in the future

PHILADELPHIA SERVERLESS ARCHITECTURE

CKAN OpenDataPhilly.org catalog

data.phila.gov/socrata

data.phila.gov/carto

1. Transform Socrata
2. Pass CARTO request
3. Transform response to Socrata (if needed)

Open Data Store (csv, shp, geojson, etc)
phl.carto.com account

USING SERVERLESS FRAMEWORK

EXTRA BALL

ANDREW THOMPSON · athompson@carto.com

SOLUTIONS ENGINEER

THANK YOU FOSS4G BOSTON!

SLIDES:
http://bit.ly/foss4g-carto-serverless

Stop by our booth!

FOSS4G Serverless Architectures - Google Slides