1 of 40

Outline:

  • Pizza + Questions
  • Lightning Talks
    • Your Hackathon Toolbelt
  • Idea Sharing

2 of 40

Hackathon: how to get started

3 of 40

Apps Script

4 of 40

Google Apps Script

Google Apps Script is a JavaScript-based language that can be used to interact with various Google Documents including Docs, Sheets, and Forms.

​

5 of 40

Example: Pizza Form

Please complete the following important survey: https://goo.gl/forms/TUl30U15bop34jxN2

6 of 40

Example: Pizza Sheet

The responses are collected in a Google sheet, and we can add triggers that will execute program code on various actions.

7 of 40

Basic website, src-code

8 of 40

src-code

9 of 40

If This Then That (IFTTT)

10 of 40

IFTTT

IFTTT a web-based service that can be used to form applets out of simple conditional statements with web services like Gmail, Facebook, Instagram, Twitter, etc. ifttt.com/create

​

11 of 40

IFTTT

There’s hundreds of web services to easily connect with. Each service has several triggers which can be linked together in statements like:

If the weather changes, then text me a warning

​

​

If I like something on Instagram, then save it to Dropbox

12 of 40

IFTTT

If you’re looking for an example, there’s also millions of premade recipes ifttt.com/discover

13 of 40

Web APIs in Python

14 of 40

HTTP

  • HTTP (Hypertext Transfer Protocol) is how we communicate with the web!
  • We do this primarily with requests.
  • For Python we are going to use the requests library!
    • http://www.python-requests.org/en/latest/
    • Requests is a nice library, because it takes care of a lot of the dirty work for us.

15 of 40

HTTP Requests

  • 3 parts: request, header, message body
  • Header is generated automatically for us, so just worry about request and body
  • The request line is composed of a VERB + URI (uniform resource identifier)
  • The message body is usually only required for requests where we send info.
  • Once the request is made, the server will return a status code

​

16 of 40

HTTP Verbs

  • 4 main ones to know
  • GET - retrieve info
    • requests.get('URI')
  • POST - send info
    • requests.post('URI', data={'key':'value'})
  • PUT - update existing info
    • requests.put('URI', data={'key':'value'})
  • DELETE - remove existing info
    • requests.delete('URI')

​

17 of 40

Authentication

  • Sometimes we need authentication for our requests.
    • This involves special security strings.
  • For twitter and the like, we need support for OAuth 1.0A.
    • We get this from the 'requests-oauthlib' library.
  • Learn more about authentication with requests here:
  • Setting up apps is a pain, so I’ve made a key for Twitter.
  • My personal info is tied to this, so please don’t abuse it.

​

18 of 40

Twitter KEY

#!/usr/bin/env python3

from requests_oauthlib import OAuth1

consumer_key = 'GmwHQha6DItHSN07zJW7sUHNZ'

consumer_secret = 'G1rfuKfrKbxiFotMcUqFkt2jQdd43R5xCTl2vJqP9U0gqwyTCx'

oauth_token = '855487201844637696-ydUp69QwVswjCYl7JXHiIOm0AzIFaB0'

oauth_token_secret = 'Eylxy4DEj7z0GmBrOjgJZ1cia8mZLGks4VQPDKFpoYXPm'

auth = OAuth1(consumer_key, consumer_secret, oauth_token, oauth_token_secret)

​

19 of 40

Status Codes

  • 1xx: info
  • 2xx: success
    • 200: okay
  • 3xx: redirect
    • 300: multiple choices (list of links)
    • 301: moved permanently
    • 302: page moved to new URL
    • 303: page found at different URL
  • 4xx: client error (bad request)
  • 5xx: server error (server couldn't respond)

​

20 of 40

Let’s make a GET request!

​

​

​

​

​

​

​

​

​

Source: https://dev.twitter.com/rest/public/search

21 of 40

Let’s make a GET request!

Source: https://dev.twitter.com/rest/reference/get/search/tweets

22 of 40

Let’s make a GET request!

23 of 40

Handling JSON Data

24 of 40

Handling JSON Data!

​

  • First, read the docs: https://dev.twitter.com/rest/reference/get/search/tweets
  • Next, read the docs again (just in case you missed something).
  • Most of the things that you need are probably in the docs.
  • This will tell you about the nature of the information that you are retrieving.
  • The .json() function in requests converts the json data into series of nested dictionaries, lists, and strings
  • You can make sense of this by using the built-in type() and dict.keys() functions.
  • Alternatively, Python has a built-in json library. Read the docs to learn how to use this.

25 of 40

Handling JSON Data!

26 of 40

27 of 40

Twitter Docs / Other Resources

28 of 40

Visualization tools

29 of 40

Carto.com

carto.com/builder

CARTO Builder is a web-based drag and drop analysis tool for analysts and business users to discover and predict key insights from location data.

30 of 40

ArcGIS API for Javascript

https://developers.arcgis.com/labs/

​

Develop powerful 2D and 3D web mapping applications.

​

Get started here

31 of 40

Data Sources and APIs

32 of 40

Google BigQuery Pubic Datasets

  • GDELT book corpus
  • GitHub data
  • Hacker News
  • IRS 990
  • Major League Baseball
  • Medicare
  • NOAA GHCN/GSOD Weather
  • NYC 311 Service Requests
  • NYC Citi Bike Trips
  • NYC Tree Census
  • NYPD Motor Vehicle Collisions
  • Open Images
  • Stack Overflow
  • USA Bureau of Labor Statistics
  • USA Disease Data
  • USA Names Data

cloud.google.com/bigquery/public-data/

33 of 40

World Bank Datasets

  • Agriculture
  • Aid Effectiveness
  • Climate Change
  • Economy/Growth
  • Education
  • Energy/MIning
  • Environment
  • External Debt
  • Financial Sector
  • Gender

data.worldbank.org/indicator

  • Health
  • Infrastructure
  • Poverty
  • Private Sector
  • Public Sector
  • Science/Technology,
  • Social Development,
  • Social Protection/Labor
  • Trade
  • Urban Development

34 of 40

US Government data

  • catalog.data.gov/dataset

​

  • Housing affordability data system
  • College Scorecard
  • Crimes (2001-present)
  • Student �Loan data

35 of 40

Map-related data

  • ARCGIS OpenData 60,000 datasets from 3,855 organizations worldwide
  • Knoema Atlas World and regional statistics, national data, maps, rankings

36 of 40

Social Networking APIs

37 of 40

Google

  • Google APIs
    • Calendar, Docs,�Gmail, Maps, �Search, Youtube
  • Google Developer Tools

38 of 40

Good places to look for more data

39 of 40

  • Form teams (2-5 people)
  • Choose an idea
  • Get started!

​

  • Presentations @ Saturday 8pm

Please upload code + presentation to devpost BEFORE 8pm

40 of 40

Have an idea? Stand up and share!