Waze Github Collaboration

Michael Schnuerle, Chief Data Officer

Ed Blayney, Innovation Project Manager

​

​

Office of Civic Innovation

Louisville, KY, USA

Contents: Waze Github Collaboration

• History and Framing
• Current State
• Roadmap
• How Do We Do This?
• You Can Participate
• Use Case Collection

Office of Civic Innovation - Louisville

Mayor Greg Fischer

Created office in 2010

​

Grace Simrall

Chief of Civic Innovation

Ed Blayney

Innovation Project Manager

Michael Schnuerle

Data Officer

Mary Hampton

Senior Data Scientist

Matt Gotth-Olsen

Innovation Project Manager

Daro Mott

Performance Improvement

History and Framing

​

• History
• Internal DB limitations
• Cloud solution

​

History of Waze partnership

• Louisville Metro joined in Sept 2015
• Ed Blayney started the Civic Innovation project, data saving to cloud in June 2016
• Worked with Mike Reynolds in IT to set up processing into internal MS SQL database, finished February 2017
• Ed working with Traffic Department to find more value and benefits. Socializing across city, understanding use cases, providing access/tools/training, defining data structures, merging other data, hackathon. Data analysis by Mary Hampton
• Michael Schnuerle starts building traffic data warehouse with Amazon City on Cloud grant in August 2017, Waze first data set.
• Collaborating w/ Waze, USDOT, cities for cloud replicable solution.

Limitations of Louisville’s MSSQL Database

• Accessible only internally - can’t easily build APIs/services
• Optimization of indexes = slow
• No JDBC connections allowed (IT security)
• Limited to MSSQL Server Management Studio or Power BI access
• Cost for server, licenses, maintenance
• Replication to other cities impossible

Cloud Solution Github Collaboration

• We are creating an automated cloud Waze data processing solution that can be replicated by any CCP Partner
• You grab Terraform.io code and deploy infrastructure stack to your cloud provider
• Enter your CCP data feed URL during deployment
• Analyze, query, extract live and historic data for your city
• We are working to be serverless and inexpensive, but estimate around $200/m right now

Current Project State

​

• AWS data storage
• Terraform improvements

​

Phase Beta: AWS Cloud Formation

• Processes Waze CCP feed every 2 minutes.
• Starts saving your data to an Amazon S3 bucket
• Published November 2017
• Helped gather support for next phases - 4 cities used it

​

On Git: github.com/LouisvilleMetro/WazeCCPProcessor

Phase 1: Terraform & End-to-End RDS

• Gets the raw Waze CCP JSON data feed into a relational database.
• Base requirement for Waze data to be useful for governments.
• Can backload old JSON data.
• Done and ready to deploy.

​

On Git:

github.com/LouisvilleMetro/WazeCCPProcessor

Project Road Map

​

• Terraform
• APIs
• Data Viz
• Map

​

Switching Code to Terraform.io

• Write once. Deploy to Google, Azure, Amazon Cloud.
• Code creates reproducible cloud infrastructure.
• Open Source and widely used.

​

Upcoming (Phase 2): API Endpoints

• Allows easy access to chunks of data from the DB
• Needed for some of Interactive Map and data viz Phases
• Good for service integrations

​

On Git: github.com/LouisvilleMetro/WazeCCPProcessor/projects/3

Upcoming (Phase 3): Traffic Study Tool

• A traffic study replacement tool that allows comparison of an area before and after changes (signal retiming, road diet, reconfiguration, traffic events)
• Good for traffic departments, internal gov operations, data analytics

​

On Git: github.com/LouisvilleMetro/WazeCCPProcessor/projects/2

Upcoming (Phase 4): Interactive Map

• Overlays current/historic snapshot of alerts, jams, irregularities, with filters
• Provides a date/time selector and slider to look back in time
• Good for leadership/ mayor, business intelligence cases

​

On Git: github.com/LouisvilleMetro/WazeCCPProcessor/projects/5

How Do We Do This?

​

• Government Collaboration
• Developer Help: Slingshot
• Funding via a Sponsor
• Open Government Coalition

​

Government Collaboration

• City staff have contributed code: Louisville, Denver, Bloomington, NYC, Joinville (Brazil).
• Many more want to contribute code.
• Over 60 governments (city, state, country) have voiced support already! -->

github.com/LouisvilleMetro/WazeCCPProcessor/wiki/City-Contribution-Credits

Slingshot: Design, Dev, Cloud Shop

• Slingshot co-created the beta phase and infrastructure/database plan with Louisville for just $2K as part of a city cloud training program.
• Slingshot then continued work for free.
• Knows the Waze data and process very well now.
• Is AWS Certified, though is using Terraform to be cloud agnostic.
• Can support implementations in other govs.

www.yslingshot.com

​

​

​

Big News: Sponsorship

• Amazon has sponsored AWS Certified Partner Slingshot to develop the entire Proof of Concept.
• Provides resources for Slingshot to work on database, APIs, Map, Traffic Study Tool.
• Slingshot does the work while they collaborate with govs - this is happening now on Git (finished by Aug-Oct).
• The benefit is development moves much faster, and govs can focus on needs more than code.

​

​

Other Supporters

• Waze - supporting this project by promoting it to CCP partners, gathering use cases.
• GovEx - looking to provide virtual and on-site training for traffic departments on how to use tools developed with this project and the Waze data.
• Presidential Innovation Fellows through USDOT - Three fellows want to help with data visualizations.

Open Government Coalition (OGC)

• This Waze project is the first for the OGC.
• OGC is a network of government agencies working on open source projects together.
• Planning is happening on the OGC Slack and Git.

www.GovInTheOpen.com

You Can Participate

​

• Coding support
• Non-coding support
• Leverage Connections
• Sponsorship

​

Coding Support - Github

Use our repo’s Project and Issues area to decide how to help and discuss in detail how to implement new features and write code.

Make Pull Requests to do things like (see “help wanted” in Issues):

• Hook in your favorite data visualization tools (Tableau, R, Carto)
• Optimize the RDS: add enriched data fields, indexes
• Reduce the costs
• Split 1 DB into a real time DB and an historic serverless DB
• Optimize how files are stored or processed
• Add support for your use cases

Non-Coding Support

• Promote the project internally, externally, and with any relevant partners
• Use the code in your gov cloud infrastructure - Slingshot can help you implement
• Provide feedback on Github on issues, ideas, or improvements.

Connection Support - Find Other Sponsors

Leverage your connections to get other sponsors on board to help fund development

​

​

• 18F, USDS, xD - We would like to connect with them.
• Google - For expedited Google Cloud development.
• Microsoft - For expedited Azure Cloud development.
• Code for America - We see some opportunity to collaborate with local brigades (like Anchorage has done)
• ESRI, Koop - Andrew Stauffer has done Waze processing with their ArcGIS Open Data product
• Github, Terraform - seem like logical supporting partners

Sponsorship Support - Fund the Development

Of course you can use the code and deploy at any time for free.

How about we pool funds for:

• Road map coordination and planning
• Github code management (mergers, pull request auditing, code cleaning, QA)
• Implementing features
• Project management
• Updates as Waze data feed and the cloud services’ offerings change.
• Long-term sustainability & reliability (a Waze priority)

​

You are voicing your support for our past (and collective future) open source efforts.

Use the Github code for free or... Choose an optional annual sponsorship tier*

to sustain the project with a bucket of funding

Silver - $2K/y

• $500 AWS Credits^†
• Slingshot Implementation (one-time)
• Recognition on project website (text/small logo)

Gold - $6K/y

• $1K AWS Credits^†
• Recognition on project website (logo)
• 1 vote on roadmap features

​

Platinum - $10K/y

• $2K AWS Credits^†
• Slingshot Implementation
• Slingshot Support including all deployments, updates, DB scripts, AWS questions
• Recognition on project website (large logo)
• 2 votes on roadmap features
• Kudos from citizens and Waze, peer recognition!

​

More benefits hopefully coming soon (other cloud provider credits, trainings, conferences)

​

* details are preliminary and subject to change

^† limited pool of credits initially: first come first serve. US only. Govs not currently using AWS get priority. Agree to be a public reference.

Additional Support Information

Add-ons

Slingshot Support - around $5K/year - Implementation, plus annual support for questions/issues.

Slingshot is available for any custom work on an as-needed basis, but please feel free to use your local vendor.

Roadmap

Govs provide a bucket of potential upgrades/ use cases/ ideas/ features. Then the collaborators can vote on what is worked on next.

​

Use Case Collection

​

• Our Use Cases
• Your Use Cases

​

Current Waze Project Examples

• Traffic Studies
• Signal re-timings, road diets, configuration changes, before and after, instead of paid study
• Faulty Equipment
• Looking at traffic patterns to discover device issues
• Post Event Analysis
• Looking at traffic after events to improve for the next event
• Hot Spot Analysis
• Evaluation of the most congested areas

Your Use Cases

Waze is collecting your use cases!

​

Reach out the Waze team or on the forum to let them know what you want the tool to do.

​

Then make tasks for it on Github and we’ll add it to the project roadmap.

Contact Info:

Ed Blayney

Innovation Project Manager

edward.blayney@louisvilleky.gov

@EdBlayney

​

Michael Schnuerle

Data Officer

michael.schnuerle@louisvilleky.gov

@LouDataOfficer

​