Chocolate Scraping

Web Scraping and Data Analysis Project

Fırat Olçum

06-07-2022

Table Of Contents

1. What is Web Scraping
2. Rules of Scraping
3. Chocolate Scraping
4. Data Wrangling
5. Data Analysis

1. What is Web Scraping

​

Web Scraping

• Often times you’ll find the perfect website that has all the data you need, but there’s no way to download it. This is where BeautifulSoup comes in handy to scrape the HTML.
• If we find the data we want to analyze online, we can use BeautifulSoup to grab it and turn it into a structure we can understand.
• This Python library, which takes its name from a song in Alice Wonderland, allows us to easily and quickly take information from a website and put it into a DataFrame.

2. Rules of Scraping

​

Web Scraping Rules

1. Always check a website’s Terms and Conditions before scraping. Read the statement on the legal use of data. Usually, the data you scrape should not be used for commercial purposes.
2. Do not spam the website with a ton of requests. A large number of requests can break a website that is unprepared for that level of traffic. As a general rule of good practice, make one request to one webpage per second.
3. If the layout of the website changes, you will have to change your scraping code to follow the new structure of the site.

3. Chocolate Scraping

​

3.1 Website Structure

• When we’re telling our Python script what HTML tags to grab, we need to know the structure of the website and what we’re looking for.

​

• Many browsers, including Chrome, Firefox, and Safari, have Developer Tools that help you inspect a webpage and see what HTML elements it is composed of.

​

• Then, when you’re preparing to scrape a website, first inspect the HTML to see where the info you are looking for is located on the page.

​

• Let's review the website before we start:

https://content.codecademy.com/courses/beautifulsoup/cacao/index.html

3.2 Requests

In order to get the HTML of the website, we need to make a request to get the content of the webpage.

--Python Code

​

import requests

​

url = "https://content.codecademy.com/courses/beautifulsoup/cacao/index.html"

​

data = requests.get(url).text

​

​

3.3 The BeautiflSoup Object

If we print out all of that HTML from our request, it seems pretty long and messy.

BeautifulSoup is a Python library that makes it easy for us to traverse an HTML page and pull out the parts we’re interested in.

Then, all we have to do is convert the HTML document to a BeautifulSoup object! "html5lib" is one option for parsers we could use.

There are other options, like "lxml" and "htmlparser" that have different advantages and disadvantages, but for our purposes we will be using "html5lib" throughout.

--Python Code

​

from bs4 import BeautifulSoup

​

soup = BeautifulSoup(data, "html5lib")

3.4 Finding Table

Beautiful Soup offers two methods for traversing the HTML tags on a webpage, .find() and .find_all(). Both methods can take just a tag name as a parameter but will return slightly different information.

--Python Code

​

from bs4 import BeautifulSoup

​

soup = BeautifulSoup(data, "html5lib")

​

table = soup.find_all("table")[1]

3.5 Reading Text

When we use BeautifulSoup to select HTML elements, we often want to grab the text inside of the element, so that we can analyze it. We can use .get_text() to retrieve the text inside of whatever tag we want to call it on.

--Python Code

​

companies = []

for row in table.find_all("tr")[1:]:

companies.append(row.find_all("td")[0].get_text())

​

review_date = []

for row in table.find_all("tr")[1:]:

review_date.append(row.find_all("td")[3].get_text())

​

cocoa_percent = []

for row in table.find_all("tr")[1:]:

cocoa_percent.append(row.find_all("td")[4].get_text())

​

company_location = []

for row in table.find_all("tr")[1:]:

company_location.append(row.find_all("td")[5].get_text())

​

rating = []

for row in table.find_all("tr")[1:]:

rating.append(row.find_all("td")[6].get_text())

3.6 Creating Dataframe

After creating the dictionary from the lists, we can convert this dictionary to dataframe using pandas.

--Python Code

​

data_dict = {"Companies":companies,

"ReviewDate": review_date,

"CocoaPercent": cocoa_percent, "CompanyLocation":company_location,

"Rating":rating}

​

df = pd.DataFrame(data_dict)

​

df.head()

​

4. Data Wrangling

​

4.1 Checking Data Types

We can check the dataframe using the .info() method. We understand that we need to change the data type of the ReviewDate, CocoaPercent and Rating columns.

--Python Code

​

df.info()

4.2 Cleaning and Converting Data Types

We cleaned the columns of the dataframe and converted it to the appropriate data type.

--Python Code

​

df["ReviewDate"] = df["ReviewDate"].astype("int")

​

df["CocoaPercent"] = df["CocoaPercent"].astype("str").str.rstrip("%").astype("float")

​

df["Rating"] = df["Rating"].astype("float")

​

df.head()

5. Data Analysis

​

5.1 The 3 Most Voted Companies and Their Number of Votes

You can see the number of votes of the companies. We see that the most voted models by participants are ‘Soma’, ‘Bonnat’, and ‘Fresco’ models.

​

--Python Code

​

ReviewCount = df[["Companies"]].value_counts()[:3].to_frame(name="Count").reset_index()

​

ax = sns.barplot(data=ReviewCount, x="Companies", y="Count")

​

for container in ax.containers:

ax.bar_label(container)

​

plt.title("3 Companies with The Most Votes and Their Number of Votes");

5.2 Review Averages of The 3 Most Voted Companies

You can see the average votes of the companies. We see that the company with the highest review average is Soma.

--Python Code

​

Top3ReviewCount = df[df["Companies"].isin(df["Companies"].value_counts()[:3].index)]

​

Ratings = Top3ReviewCount.groupby("Companies")[["Rating"]].mean().reset_index().sort_values(by="Rating", ascending=False)

​

ax = sns.barplot(data=Ratings, x="Companies", y="Rating")

​

for container in ax.containers:

ax.bar_label(container, fmt="%.2f")

​

plt.title("3 Most Rated Companies and Their Review Averages");

5.3 Change of Review Averages by Years of The 3 Company

When we look at the graph, we can say that in general, all 3 companies made changes in the year following the bad review average and increased their review averages.

--Python Code

​

df3 = Top3ReviewCount.groupby(["Companies", "ReviewDate"])["Rating"].mean().to_frame().reset_index()

​

plt.figure(figsize=(8,5))

​

sns.lineplot(data=df3, x="ReviewDate", y="Rating", hue="Companies")

​

plt.title("Change of Review Averages by Years");

5.4 The Effect of Cocoa Ratio Used in Chocolates on Reviews

When you look at the chart You can easily see that there is no relationship between the cocoa ratio used and the review.

--Python Code

​

plt.figure(figsize=(8,5))

​

sns.scatterplot(data=df, y="Rating", x="CocoaPercent")

​

plt.title("Cocoa Percentages and Review Relationship");

Thank you for your time and attention.