1 of 32

�Building My Own Little World with Open Data

Presented by:

Steven Pousty

@TheSteve0 on Twitter, IRC, Ingress, SmugMug, Skype, Github, and @redhat.com

​

http://bit.ly/molws

​

​

2 of 32

Goals

  1. Share some thoughts
  2. A small small introduction to some data and software
  3. Show It In Action
  4. Talk about what I/We have learned
  5. Future Areas

3 of 32

Open Data is Now The Norm

4 of 32

Corollary:

This is not an Open Data talk.

This is about Data Knowledge

5 of 32

Cards on the Table

  1. Almost All Web Applications Should Be HTML Front-Ends Talking to REST Back Ends
  2. I like Microservices
  3. I HATE Data Portals where there is one single map or query interface and then you turn layers on and off with different queries. Single Purpose pages/apps FTW!

6 of 32

Most People Don’t Care About Data

​

They Do Have Questions About Where They Live

7 of 32

Me

I live in Santa Cruz, California, USA (similar climate to here)

I am ecologist by training

I love birding (going out spending time�and money to identify and watch birds in nature)

Weather matters to people and birds

I love exploring maps and data

8 of 32

Please note:

All source code (and some data) is at:

https://github.com/molw

​

9 of 32

What can I put together?

��E-Bird�US: http://ebird.org/content/ebird/��OZ: http://ebird.org/content/australia/

10 of 32

I live near the coast

NOAA Marine�Observations Search�https://www.ncdc.noaa.gov/cdo-web/datatools/marine

11 of 32

Almost everyone lives near streams

USGS National�Water System�https://waterdata.usgs.gov/nwis/uv/?referred_module=sw

12 of 32

Most Counties in US Share GIS Data

Santa Cruz Country GIS Data:�http://www.co.santa-cruz.ca.us/Departments/GeographicInformationSystems(GIS).aspx

13 of 32

Lessons Learned about Data

  1. Redistribution rights of Data
  2. Metadata, metadata, metadata
  3. Times without Timezones
  4. Gettings coordinates without a datum or a projection
  5. Work with small chunks of data first (‘$ head -n’ is your friend)

14 of 32

You’ve Got Data —- Now What??

15 of 32

Munge it�Python with CSV module

Get a Platform Up

OpenShift Origin on GCE

​

16 of 32

Put The Data Somewhere

PostgreSQL with PostGIS in Containers from Crunchy Data

​

Let’s take a look at the platform

17 of 32

Lesson Learned on Getting Data Somewhere Useful

  1. Moving a lot of data around takes time (and can cost you money)
  2. Try some/all of it locally first
  3. With larger data sets you do actually need to think about what and how you store stuff in the db.
    1. Location fields separately from location column
    2. Build Indices later
  4. Containers with a Platform can remove a lot of install Yak shaving

18 of 32

19 of 32

Nice! You Have Data in a DB - How Do You Analyze It??

20 of 32

Non-Spatial

Spark in Containers with Radanalytics

WIP - easy to get a cluster - not as easy to get PySpark working with JDBC in containers

Spatial

Using PostGIS Queries

21 of 32

Great, Analysis Done - How Do We Show It?

22 of 32

REST Service

Wildfly with Hibernate Spatial and JAX-RS

Map Serving

OpenStreetMap Tiles from Stamen

GeoServer in Containers to serve up County Base Layers

​

23 of 32

Front End

PatternFly (based on BootStrap) site

Leaflet for maps

​

24 of 32

What Do We Get?!?

http://bit.ly/molwb

I will talk through some of the interface

25 of 32

26 of 32

27 of 32

28 of 32

Observers

Casual Spotters

obsr297438 103 1144 days 09:00:00

obsr302446 102 141 days 00:45:00

Serious

obsr535069 1041 844 days 05:30:00

obsr187159 1028 4866 days 11:41:00

​

29 of 32

Lessons Learned On the Front End

  1. Dealing with visualizing large numbers of Points
  2. CORS for front end request to REST services

30 of 32

Larger Lessons About the Whole “Framework”

  1. Learning all the different technology - A lot of Yak Shaving
    1. Find helpful people :D
  2. Automate as much as you can
  3. Framing good questions takes time and exploration
  4. How to make an open ended framework to get citizens to be able to ask questions
  5. Monitoring the questions so you can evolve what you do
  6. How to get into a positive feedback loop with the providers of the data
  7. Give yourself more than 3 weeks to get it all working ;)

31 of 32

What I Want To Add Soon

Click on an Observation and Then Show Barometic Pressure 1 Week Before and After

Show Where Observers Don’t Go ��Show Weather Relation To Observations

Integrate Tides and Hydrology As Well (I also like to Fish ;) )

Actually Integrate Statistical Models Rather Than Just Visualization

32 of 32

�Building My Own Little World with Open Data

Presented by:

Steven Pousty

@TheSteve0 on Twitter, IRC, Ingress, SmugMug, Skype, Github, and @redhat.com��http://bit.ly/molws

​

​