1 of 19

Data Generation in Python

Web Scraping and APIs

Big Analytics with Python, AIMS 2023

Dunstan Matekenya, PhD

1

7/18/2017

Big Data Program-Lighting Talks

​

2 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Expected Learning Outcomes

At the end of the session today, learners should:

Learners should know which libraries to use for different data science tasks and use cases. For example, how to process spatial data or how to handle large scale data

BE FAMILIAR WITH ESSENTIAL PYTHON LIBRARIES FOR DATA SCIENCE

How you can generate, pull and assemble data using from websites and APIs using Python packages such as requests and BeatifulSoup.

UNDERSTAND KEY CONCEPTS IN WEB SCRAPING AND ACCESSING APIS

Be able to handle spatial data in Python

APPRECIATE HOW TO PROCESS SATELLITE IMAGES IN PYTHON

01

03

02

2

7/18/2017

Big Data Program-Lighting Talks

​

3 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Lecture Outline

1

3

4

Why Bother About Getting Data from Websites?

Web Scraping Primer

Why scrape websites, HTML, tools, key steps

2

Introduction to APIs

API requests, Status codes, Qeuery parameters, Content type, JSON data

Following Etiquette When Accessing Data Online

3

7/18/2017

Big Data Program-Lighting Talks

​

4 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Why Gather Data From The Web

​

  1. A LOT of data is on the Web

​

  1. Some Data is only on the Web (e.g., Social Network Data, E-Commerce websites)

​

4

7/18/2017

Big Data Program-Lighting Talks

​

5 of 19

A Brief Introduction

To APIs

​

5

7/18/2017

Big Data Program-Lighting Talks

​

6 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

What is an API?

​

Application Program Interfaces (APIs), are used to retrieve data from remote websites. Sites like Twitter and Facebook all offer certain data through their APIs. To use an API, you make a request to a remote web server, and retrieve the data you need.

6

7/18/2017

Big Data Program-Lighting Talks

​

7 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

API Requests

​

  • APIs work the same way as browsers but instead of your web browser asking for a webpage, your program asks for the data. Data is usually returned in JSON format

​

  • The process: make a request to a webserver, the server then replies with our data

​

  • Tools
    • Requests: a package for accessing websites
    • Special packages from the service provider or third party (e.g., for twitter, Facebook and other organizations) they are specific Python packages for accessing the APIs which you have to install�

About API Requests

Type of API Requests

  • GET requests are the most common and widely used methods in APIs and websites.

​

  • The GET method is used to retrieve data from a server at the specified resource. For example, if you have an API with a /users endpoint. Making a GET request to that endpoint should return a list of all available users.

7

7/18/2017

Big Data Program-Lighting Talks

​

8 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Status Codes

​

Status codes are returned with every request that is made to a web server. Status codes indicate information about what happened with a request. See https://restfulapi.net/http-status-codes/ for a complete list of status codes.

Common Status Codes

How to Ensure You Get 200 Code

​

  • Make sure you have credentials if required

​

  • Hit the right endpoint by checking the API documentation

​

  • Put correct query parameters

​

  • Follow the rules (requests limits etc)

8

7/18/2017

Big Data Program-Lighting Talks

​

9 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Working With JSON Data

Recall the dictionary data type in Python: JSON is simply a nested dictionary

JSON is the primary format in which data is passed back and forth to APIs, and most API servers will send their responses in JSON format.

  • Python has the built-in json package to handle JSON data

​

  • The package allows different conversions, for instance, lists and dictionaries to JSON

​

  • The json package has two main methods:
    • dumps -- takes in a Python object, and converts it to a string.
    • loads -- takes a JSON string, and converts it to a Python object.

9

7/18/2017

Big Data Program-Lighting Talks

​

10 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Getting Content From API Requests

  • Getting the JSON: use the  .json() method on the response.

​

​

  • Metadata: the response also comes with metadata containing information on how the data was generated and how to decode it. In Python, we can access this with the headers property of a response object.

​

10

7/18/2017

Big Data Program-Lighting Talks

​

11 of 19

A Brief Introduction

Web Scraping

​

11

7/18/2017

Big Data Program-Lighting Talks

​

12 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

What We Will Cover

WEB CRAWLING is essentially what companies such as Google, Yahoo, MSN and others do, gathering any kinds of information. They will view an entire webpage and index it. Bots used by these search engines will go through every page and link on a website and literally pull out ANY information which is included.

WEB SCRAPING  is generally more targeted at certain websites, looking for specific information, e.g. GTINs/EANs for e-commerce merchants or for price comparison, so that the methodology and software are different.

The two terms can be used interchangeably and of course web scraping involves some limited web crawling

12

7/18/2017

Big Data Program-Lighting Talks

​

13 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Why Scrape?

​

  • USUALLY WHEN THERE IS NO API AVAILABLE, YOU HAVE NO OPTION BUT TO SCRAPE TO GET THE DATA

​

  • IN SOME CASES (E-COMMERCE ETC), SCRAPING IS PART OF THE GAME

​

  • WEB SCRAPING ISNT ALWAYS LEGAL/ALLOWED, SOME WEBSITES PROHIBIT

​

13

7/18/2017

Big Data Program-Lighting Talks

​

14 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Main Languages for the Web

​

HTML, CSS and Javascript

  • HTML gives structure to content (text, images, etc.)

​

  • CSS gives appearance to content (font colors, image borders, etc.)

​

  • JAVASCRIPT makes content interactive (a clock that ticks in real time, sparkly things that follow your mouse cursor)

14

7/18/2017

Big Data Program-Lighting Talks

​

15 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

HTML Structure

A sample HTML document structure

  • You can check HTML document structure by viewing the page source

​

  • For more detailed inspection, use developer tools provided by the browser

​

ITS CRUCIAL TO UNDERSTAND THE HTML STRUCTURE OF THE WEB PAGE OF INTEREST TO SUCCESSFULLY SCRAPE IT

15

7/18/2017

Big Data Program-Lighting Talks

​

16 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Python Tools for Web Scraping

Requests, Beautifulsoup, Scrapy

Depending on your level of programming and needs, the most common tools include:

  • Requests: A base library for sending requests to web servers including APIs

​

  • BeatifulSoup: Parsing HTML documents to extract useful data

​

  • Scrapy: A a more sophisticated framework for web scraping. See details here: https://docs.scrapy.org/en/latest/intro/overview.html

16

7/18/2017

Big Data Program-Lighting Talks

​

17 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Main Steps in Web Scraping

A Summary of Main Steps

Requests, BeatifulSoup, Scrapy

Ensure you have the required

tools installed

Using developer tools in your browser identify the structure of the pages you are trying to scrape

Understand the structure

of the HTML

Based on what data you are looking for. If the site has an API, please use it

Find the URL(s) you want

to Scrape

Write code, get and save the data

17

7/18/2017

Big Data Program-Lighting Talks

​

18 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Notes About Accessing Websites

  • Before scraping, check if there is an API available

​

  • When using API read the documentation carefully

​

  • Before scraping check legality of scraping the website of interest

​

  • Don’t send too many requests or you can get banned

​

  • When using API stick to the requests limits

​

​

Check out this blog which talks about some of the legal issues:

https://benbernardblog.com/web-scraping-and-crawling-are-perfectly-legal-right/

18

7/18/2017

Big Data Program-Lighting Talks

​

19 of 19

Big Data Analytics with Python, AIMS 2022 by Dunstan Matekenya

Web scraping is tedious and hard BUT there is money to be made on Freelancing sites

19

7/18/2017

Big Data Program-Lighting Talks

​