1 of 29

Accessing and analysing your own social media data

Caitlin Macleod

@caitelatte

c aitelatte.com

2 of 29

Caitlin Macleod (@caitelatte)

Signed up to Facebook in 2007 and has been in 1107 Messenger conversations
Joined Twitter in 2011 and tweeted more than 1000 times in the first year
Started Python in high school
Learnt a lot of software things at the Australian National University
Currently write a lot of Python and command-line magic at work

3 of 29

Personal data

Who did I interact with?

What did I say/search/look at?

When did something happen?

Where was I?

How was I feeling?

"The data subject shall have the right to receive the personal data concerning him or her, which he or she has provided to a controller, in a structured, commonly used and machine-readable format [...]"

Right to data portability, Art. 20, General Data Protection Regulation (see https://gdpr-info.eu/art-20-gdpr/)

4 of 29

Shared personal data

Be careful with others' personal data!
Messages are often assumed to be a shared secret.
Be careful about the possibility of sharing a friend's information with someone they may have blocked

So the thing with using personal data from a social media site is that the interactions we're tracking often involve more than yourself. You may be handling sensitive data from your friends or people you've interacted with.

Think about this before downloading your social media archives.

Don't leave the archives on unprotected USBs lying around
don't release your friend's messages onto the internet
Be careful even about basic information from others that you may share; it may reach those who shouldn't usually have any visibility of it.

But also! This can be great for helping you come up with project ideas! I was talking to Jack Massey over lunch who did a real cool thing! He took his facebook conversation with his girlfriend and made an infographic bookmark out of key analytic takeaways! So cool and exciting! But also be careful about that also, some people might not appreciate it.

5 of 29

Downloading from Facebook

Facebook: https://www.facebook.com/settings?tab=your_facebook_information

6 of 29

Downloading from Facebook - a direct link from settings to https://www.facebook.com/your_information

7 of 29

Downloading from Facebook

Facebook: https://facebook.com/your_information

A highlighted button in the settings menu. A user can choose what date selection and aspects they want to include, as well as HTML vs JSON formats.
The downloaded zip contains 25 different folders with different aspects of the platform, incl json and actual picture or media content.

8 of 29

Downloading from Twitter

Twitter: https://twitter.com/settings/your_twitter_data

9 of 29

Downloading from Twitter - find the menu from https://twitter.com/settings/account

10 of 29

Downloading from Twitter - interesting fields https://twitter.com/settings/your_twitter_data

11 of 29

Downloading from Twitter - get the download! https://twitter.com/settings/your_twitter_data

12 of 29

Downloading from Twitter

Twitter: https://twitter.com/settings/your_twitter_data

A page showing:

Personal information they've inferred or asked for
Advertising targeting information!
A button inviting you to download your Twitter data

The downloaded archive consists of: 1 file named tweets.csv

No images from your tweets
No information about follows or messages

13 of 29

Social media structures

14 of 29

How do data models shape platforms?

Twitter: Users post unchangeable tweets. Relationships are follows and can be actioned by mentioning, replying, retweeting and liking.

Facebook: Users can post on their own or others' timelines, and comment on posts. Relationships are friendships and can be actioned by reacting, commenting, sharing, tagging or posting.

Messages: Users can message others. Relationships are actioned by messaging and reacting, whether in a private message or a group.

Understanding the underlying data structure of these sites isn't just useful for understanding their culture, it'll help when we get to looking at the data too!

First off, let's look at the original ideas and framework behind Twitter! Users post unchangeable tweets. These are public by default, though they added extra controls around this later. Relationships are represented by "following" and you can mention, reply to, reweet and like tweets from other users.

Twitter is pretty simple at its base, and a great target for data analysis for this reason!

Facebook, on the other hand, has so many features it has a (really useful) wikipedia article about features and when they were added! Let's focus on the basics for now:

posts on your own timelines, friendships, reactions, comments, sharing, tagging and posting on others' timelines.

Messaging structures are pretty similar across both platforms, so they're defined here. The basic difference is in the definitions of relationships, since Twitter focuses on following instead of friendships.

I'll bring these back up when we look at the platforms again, but now I get to talk about analysing it!

15 of 29

Analysing!

16 of 29

How to look at this data?

What tools did I find and look at?

Anaconda is a distribution of Python and R with data science and machine learning tools included.
Jupyter Notebooks intermingle runnable code with Markdown annotations
Natural Language Toolkit (NLTK) is a Python tool for natural language processing.
Matplotlib is a 2D plotting library in Python which is great for statistical representation.

17 of 29

Twitter

Structure: Users post unchangeable tweets. Relationships are follows and can be actioned by mentioning, replying, retweeting and liking.

Archive: a single comma-separated values file of your own tweets and an HTML page to scroll through it. Same format of tweets as presented by their API.

18 of 29

Load tweets from tweets.csv into a list of OrderedDicts

19 of 29

Example 1: Histogram of tweets!

Goal: a frequency histogram with the number of tweets per a given time period.

datetime.datetime.strptime is a function to parse a formatted string and return a datetime.datetime
pandas.date_range(start=first_time, end=last_time, freq=timediff)
n, bins, patches = ax.hist(tweet_mpldates, stacked=True, color='teal')
matplotlib.dates uses a representation of dates using numbers, which are very easy/fast to do calculation on! Use ax.xaxis.set_major_locator and ax.xaxis.set_major_formatter to format the numbers into dates!

20 of 29

Example 1: Histogram of Tweets! Latest 200 tweets on left, all time on right.

21 of 29

Example 2: Mentions of different people

Goal: a representation of all the different people I've mentioned over time.

Steps:

Identify data structure goal: list of tuples of usernames and timestamps
Extract the necessary data
Present the data

22 of 29

Example 2: Mentions of different people

23 of 29

Example 2: Mentions of different people

24 of 29

Example 3: Sentiment analysis!

Goal: Track my own feelings over the course of several years, relate them to different hashtags. This should include tweets related to conferences :D #livetweeting #conflife

Notes: I haven't done this. This is a stretch goal :D

25 of 29

Analysing Facebook

26 of 29

Facebook

Facebook: Users can post on their own or others' timelines, and comment on posts. Relationships are friendships and can be actioned by reacting, commenting, sharing, tagging or posting.

Messages: Users can message others. Relationships are actioned by messaging and reacting, whether in a private message or a group.

Archive: 25 different folders with different aspects of the platform, with either json or HTML contents. User can choose what date selection and aspects they want to include.

27 of 29

Output 1: Reactions! 💖

Goal: Show a bar chart of how many reacts of each type per month.

28 of 29

Download from everywhere!

Facebook
Twitter
Google
Instagram
Myspace (have to log into Myspace classic)
Snapchat

Tumblr (via yahoo oauth?)
Slack
Uber (scraper required)
LinkedIn

I'm sure there's more! Not just social media sites, either. Think of anything you've got an account on.

29 of 29

Sources

Social Media Structures - Geoff Hinchcliffe & Mitchell Whitelaw - DESN1001 Lecture 2017-03
Matplotlib Pyplot Text example https://matplotlib.org/gallery/pyplots/pyplot_text.html#sphx-glr-gallery-pyplots-pyplot-text-py