Besides Blocks: Twitter

Connecting the World Together

Table of Contents:

Goals:

References:

Intro to Unix:

Python In the Terminal

Getting Up and Running

What’s an Interactive Interpreter?

Working with Files

Connecting the World Together

What is an API?

Web APIs

What to do when you get stuck:

Twitter!

Getting Started:

Simple APIs

Background

Your Own Twitter Account

Getting Your Followers

Following a New Person

Posting a Tweet

Wordcloud

(dictionary may be in a different order)

Going Beyond

How Many Tweets Should You See?

Getting your own API Key


Goals:


References:


Intro to Unix:

In today’s lab, we’ll be working with Python in the Terminal application on your computer. Before we get into Python, you’ll need to learn a few terminal commands.

How files are organized on a computer:

As you probably already know, your python files, and any file for that matter, can be found within a particular folder on your computer.  This folder may in turn be nested inside of other folders to keep things organized.  In the terminal, files are organized in this exact same way, except we simply refer to folders as “directories.”  We can imagine that each directory has a parent directory and perhaps several subdirectories or files that are located inside of it (just like folders).  By moving in and out of these directories, we can easily organize and access our files using the following commands (note: some of the following commands might be different for windows):

Let’s use a few of these commands to get set up to do the lab.

Begin by opening up the terminal and entering the ls command into the prompt.  You should see a list of files and directories that are contained within your current directory.  You can navigate to any directory you like using the cd command.  Once you are ready, type the command mkdir pythonLab2 to create a subdirectory called pythonLab2 (name it something else if you’d like).  Then cd into this directory (Pro Tip: pressing [TAB] while typing the beginning of a file name will make the terminal try to auto-complete the name for you).  Once inside of this directory type ls and see what happens.  You should see nothing printed out since this directory is empty!  Now that you can navigate to a particular file, let’s try creating some python files to fill your empty directory.

        

Python In the Terminal

Getting Up and Running

Recall that in the previous Python lab, you primarily worked within edX, in your browser. While this method is useful for simple purposes, it’s much more common for programmers to run Python directly on their computers, usually through a Terminal application. The simplest way to start Python is to open the Terminal application, type python3, and press [Return]. This will drop you into the interactive interpreter.

The Terminal application is in the folder (/Applications/Utilities/) on the Macs in lab. You can also find it by searching “Terminal” using the spotlight feature (magnifying glass in the top-right corner). On Windows, you need to open the Command Prompt, which you can do by pressing the “windows key” and the r button.  Then type cmd into the “run” prompt that appears (you can also search for “cmd” in the start menu).

What’s an Interactive Interpreter?

The interpreter is a way of running Python that allows you to quickly and easily test small pieces of code. The interpreter is well-suited for this purpose because it instantaneously provides feedback after you type lines of code. You will find this to be quite helpful for debugging and trying things out.

By default, when you type python3 you’re dropped into the interactive interpreter. You’ll see some information which prints out what version of Python you’re using as well as >>>.  Whenever you see >>> (known as “the prompt”), this means you can simply start typing code and the interpreter will evaluate each line as you enter it. Try typing 4 + 5 and then press [Return]. You should see the number 9 displayed, and then be returned to the prompt.

Lists

Recall that lists in Python are very similar to lists in Snap! We can add and remove items from them, as well as look up their contents. Let’s practice with lists in the interpreter.

List syntax:

To create an empty list, assign a name to a set of brackets:

>>> my_list = []

Some important list functions (sometimes called “methods”) include:

-Append, which inserts an “item” at the end of a list. This works like “add ( ) to list” in Python.

>>> my_list.append(5)

>>> my_list

[5]

>>> my_list.append(1)

>>> my_list.append(0)

>>> my_list.append(5)

>>> my_list.append(0)

>>> my_list

[5, 1, 0, 5, 0]

- To get a value at a certain index, place your desired index between two brackets like this (remember indexing in Python begins at 0). It’s like the “item ( ) of list” block in Snap!:

>>> my_list[0]

5

>>> my_list[2]

0

-Remove: removes the first instance of “item” in the list.

>>> my_list.remove(0)

>>> my_list

[5, 1, 5, 0]

-Pop: removes and returns the last “item” in the list.

>>> my_list.pop()

0

>>> my_list

[5, 1, 5]

>>> my_list.pop()

5

>>> my_list

[5, 1]

There’s a vast number of functions for lists which can be found in the Python documentation, here.

Working with Files

Though the interpreter is quite useful, it will become cumbersome when you begin constructing larger and more complex programs. Python files (a.k.a. “scripts”) are signified by the file extension “.py” (i.e. my_python_script.py). The general process for writing longer Python scripts goes something like this:

        

1. Edit using text editor

        2. Save File

        3. Run In Terminal

        4. Repeat as necessary

Let’s imagine we are working on a file called “script.py”. The first thing we will do is create the file (or if it already exists, open it) in a text editor. Some good text editors for programming include Sublime, Text Mate, gedit, Notepad++, Emacs, and Vim. Most of these can be downloaded for free. You will find Sublime and TextMate already installed on the lab computers.

Let’s assume script.py does not exist yet. To create it, open your text editor and choose “File -> New.” Then, choose “File -> Save As” and type “script.py” (after choosing the desired directory, of course!). Make sure to type the “.py”, or else Python and your text editor won’t handle your file correctly. Now it’s time to write some code!

Try typing the following two lines into your file:

print(“Hello world!”)

1 + 2

Now that we have a bit of code, let’s run it. Make sure to save your file, then make your way to the terminal. To run the file, simply type python3 script.py. At this point, you may encounter an unfriendly error saying something like “no such file.” This most likely means that you are not in the same directory as your script.py file. Remember that cd will help you get to the relevant directory and ls let’s you view the files in your current directory. Once in the correct directory, try running python3 script.py again. You should see Hello world! appear in the terminal. You may be wondering why 3 (the result of 1 + 2) wasn’t printed. This is because when running .py files, Python doesn’t display the results of computation automatically. You need to explicitly use print if you want to display the results. Change the line in script.py to print(1 + 2) and verify that it displays 3 when you run the file.

Now, let’s write a couple of list functions. In your script.py file, define the following two functions:

# a pound sign means that the rest of the line is a comment

# and not code

# return items 0, 2, 4, … in a new list

every_other_item(my_list)

>>> every_other_item([“a”, “b”, “c”, “d”])

[“a”, “c”]

# Returns the smallest item in the list.

# Assume the items are all positive numbers

minimum(my_list)

>>> minimum([2, 1, 3, 4, 6])

1

After defining one or both of the functions, try running the file by typing python3 script.py. If you have written the functions without errors, you should find that nothing really happens when you run the file. This is because you have just defined the functions and have yet to call them. You can call them in two ways:

1. Below the function definitions, call one of the functions with inputs. You’ll need to print it to get any output. For example:

def minimum(my_list):

         # your definition goes here

         # …

print(minimum([2, 3, 1]))

Now, assuming you’ve written minimum correctly, 1 should be printed when you run the file.

2. Call the functions from within the interpreter in the following way:

It’d be nice if we could test our function more easily. Fortunately, Python provides an option which combines the convenience of writing Python scripts in a text editor with the flexibility of the interactive interpreter. We can access this option by typing python3 -i script.py (or any other .py file). What this will do is first run the file, and then put you into an interactive interpreter with all of the variables and functions from the file available to use. Now, try running your file with python3 -i and then test the functions we have defined with various inputs.

Dictionaries

Dictionaries are similar to lists, except instead of using an index to insert and retrieve an “item”, you use a key and value. For example, if you think of a dictionary as a phone book, the “key” would be a person’s name, and the “value” would be their phone number. How do you find someone’s phone number? You use their name to look it up! But dictionaries are NOT sorted. What makes dictionaries useful is that we assign a key to each value, instead of a numerical index like we do for a list. You can also use a dictionary to represent things like a menu (meal (key) → price (value)), birthdays, (name (key) → date of birth (value)), etc.

To create an empty dictionary, assign to a set of braces:

>>> grades = {}

To add an entry to your dictionary, use this syntax (remember if you are using a string for your keys, they must be in quotes):

>>> grades[“Bob”] = 85

>>> grades

{“Bob”: 85}

>>> grades[“Alice”] = 90

>>> grades

{“Bob”: 85, “Alice”: 90}

>>> grades[“Alice”]

90

To remove an entry:

>>> grades.pop(“Bob”) # just like .pop() for lists

85

>>> grades

{“Alice”: 90}

Search online to find more helpful dictionary methods!  Here is the python documentation:

https://docs.python.org/3.4/tutorial/datastructures.html#dictionaries

Now let’s use a dictionary to find out something interesting about a person.  A person’s information can be represented by a dictionary that has keys corresponding to age, height, weight, and friendsList. For example, person[“weight”] would return a number representing the person’s weight, while person[“friendsList”] would return a list of their friends (notice that dictionary values can be anything, such as lists and even other dictionaries).  First, create a dictionary that represents your imaginary (or real) person, which we will test our functions on. Then, write a function that returns True if the given person is an adult (assume an adult is someone of age 18+).  The previous exercise should only require one dictionary access.  Now let’s write another function to compute the Body Mass Index (BMI) of a person using the formula given below:

                                BMI = (703 x weight) / (height)2

Once you have these functions written, test them out by loading your file into the interpreter using the methods outlined in the previous section.

Extra for experience: Write a function that returns True if a person has more than one friend.

Connecting the World Together

So far, most of the programs you have written have been pretty self-contained. However, many of the most popular applications rely on information from other services. Facebook can connect to Twitter, or Yelp can display the location of a restaurant on a map. These sites use “APIs” to connect to each other. On your computer, applications uses

What is an API?

An API, or Application Programming Interface, is merely an agreement for how to share information. On the web, APIs exist in the form of special web addresses. Instead of displaying a webpage, they return the requested information in some easy to use format. Programmers also create their own libraries of code, which are APIs that make creating other applications easier. Imagine if every time someone wanted to write an application that opened and saved files, they had to rewrite the entire user interface from scratch. APIs exist to make applications easy to write, and provide some consistency for users.

Web APIs

While web APIs are essentially just special URL’s, the process of sharing information can quickly get complex. Different APIs will return data in a variety of different formats, and the process of authenticating users may be tricky. Don’t worry if you get stuck or see some term you’re not familiar with. Often times, programming involves working with new concepts. The trick is figuring out how to get help.

What to do when you get stuck:

 When you’re stuck there’s a number of places to look for help:

Twitter!

Twitter is a service that lets you share short messages with your friends. We will be using the Twitter API to create a very simple Twitter application which will display a stream of tweets and let you post your own tweets.

For this lab, we have created an account @pythonblocks that you will be able to use to view and post tweets which are visible to the world. Throughout this lab, we will work through some common things you might want to do with Twitter, but we encourage you to explore the API for yourself, or post tweets to your personal account if you have one!

Getting Started:

The first thing to do, is grab the starter files we will be working with. If you are working in lab, open up Terminal and run the following command:

cp -r ~cs10/twitter ~/Desktop/

This will copy all the necessary files to a Twitter folder on your Desktop. If you are

working from home, download the files HERE and copy them somewhere convenient.

Twython: Twython is a Python library for working with the Twitter API. It makes setting up authentication of each Twitter account easy and provides some easy to use functions. If you are curious, take a look at the Twython folder inside the files you downloaded. Most of the code is not terribly complex, but it does include concepts you have yet to see, so don’t worry if you don’t understand it.

        

The Twython docs can be found here.

The Twitter API is a set of URLs which lets you access nearly all of the data associated with a Twitter account. Take a quick look at the documentation, here.

        

Take a look at the documentation for the home_timeline method. Notice that there are three basic things to look for on the page. These three things will appear on almost every API page (even for other applications) and are a good guide for learning how the API works.

1. Description. This gives a basic overview of what the method does. It will also tell you the type of the method. home_timeline is a “GET” method which means that it will return data to you. Another common type is “POST” which is generally used for methods which you will be providing some information to upload, such as the content of a message or an image.

2. How to Use: The middle section usually gives a description of how you should call the method. In this case, there is a section for Parameters. These parameters give more control over what is returned from the Twitter servers. In this case, all the parameters are optional, but many methods require some parameters such as the ID of a tweet to delete.

3. Example: The final section is usually an example of what an API call looks like and what is returned. Take a close look at the example data! (You might notice a Cal reference.)

This data is technically in a format called JSON, but it should look very similar to the Python dictionaries we used earlier. In fact, Twython (and many other libraries) will automatically convert to a dictionary.

Simple APIs

Background

We’ve provided a set of files. To use these files, you’ll want to make sure you’re in the correct directory in Terminal. If you ran the command above, try doing:

        cd ~/Desktop/twitter

To run the file, you’ll simply do:

        python3 twitter.py

However, we really recommend trying to play around in the interactive interpreter as well! All you need to do is:

        python3 -i twitter.py

Your Own Twitter Account

While we’ve setup the @pythonblocks account, you can post tweets to your own Twitter account. To do this open the twitter.py file and  change the following line:

Change personal_account = False to  personal_account = True

Now, try running twitter.py by typing python3 twitter.py in the Terminal. The first time you do this, a web browser should open displaying a pin code. Copy this pin code into the terminal, and you should be signed into your own account. After the first time you sign in with Twitter, a set of ‘keys’ will be created so you can be automatically signed in next time you use this Twitter application.

Building a Timeline

The timeline in Twitter is a page with posts from a user and the people that he/she follows. To get your timeline, simply call twitter.get_home_timeline(). This will return a list of tweets (each of which is represented as a dictionary). Each tweet has a number of fields. To get an idea of what fields a tweet contains, use .keys(). For example:

>>> tweets = twitter.get_home_timeline()

>>> tweets[0].keys()

dict_keys(['contributors', 'entities', 'retweeted', 'retweet_count', 'in_reply_to_user_id_str', 'lang', 'created_at', 'truncated', 'text', 'in_reply_to_status_id_str', 'favorited', 'in_reply_to_status_id', 'in_reply_to_user_id', 'favorite_count', 'id_str', 'place', 'source', 'geo', 'coordinates', 'in_reply_to_screen_name', 'id', 'possibly_sensitive', 'user'])

>>> tweets[0][“text”]

‘The content of a tweet’

For this exercise, you’re going to build the function displayTimeline(), which displays the top of your timeline in the terminal. Try the function out by typing python3 -i twitter.py. Then, in the Python interpreter, type displayTimeline(). You should see a printout of screen-names and the string “YOUR CODE HERE.” Your job is to make this display more detailed (and actually display the content of the users’ tweets). First, fill in the functions under #BASIC FUNCTIONS in the file twitter.py. After doing so, you should be able to run the code again and see the users’ tweets. Then, edit the displayTimeline function to display more information, such as the timestamp of the tweet.

Getting Your Followers

There is an easy to use function provided by twitter for getting who you are following, called a “friendship” in Twitter-API-speak.

Take a look at the getFollowers() function we’ve provided. It’s not very pretty, so it could definitely use a bit of work. Try fixing the function so it looks like this:

>>> getFollowers()

@user1

@user2

@user3

...

[and so on]

Following a New Person

To follow a new person, we need to find someone to follow! Let’s try searching for a user. Twython provides some search functionality for Twitter, so try help(twitter) to see the functions associated with the Twitter account we have set up.

To follow a new person, we need to know who to follow! Fortunately, there’s a search_users()function which works just like it sounds. However, it needs to know the correct search terms.

The Twython documentation (from help()) will show something like this:

 |  search_users(self, **params)

 |      Provides a simple, relevance-based search interface to public user accounts on Twitter.

 |      

 |      Docs: https://dev.twitter.com/docs/api/1.1/get/users/search

This is telling you a few things:

  1. This function accepts a variety of special parameters, shown by **params.
  2. To find out what these parameters are, we should head over to the Twitter documentation (which is linked above).
  3. The Twitter documentation will tell us, that a parameter ‘q’ (short for query) which is what we want to search for. To call the function with the appropriate parameter we should do this:

(Remember to put quotes around your search term!)

>>> twitter.search_users(q=”cycomachead”)

[ … a long list of user data]

Twitter will return a long list of user “objects”. We have already written a function that displays the screen name for each user. Try something like this:

>>> getScreenName(twitter.search_users(q=”cycomachead”)[0])

Now that we know who we should follow, we need to find the user ID for the particular account. Fortunately, we have provided a getUserID() method which you can use. Notice that most of these functions are very simple! And that’s the point - we can build up a pretty complex application from a bunch of simple command.

The function we can use to follow someone is called create_friendship(). The documentation is here. Notice that it requires a parameter of whom to befriend. This can be either a user ID or a screen name. In the interpreter, try following someone:

>>> twitter.create_friendship(param_name=value)

This will display infomation about whom you just followed, but if you’d like, simply head over to twitter.com to see that it actually worked!

Posting a Tweet

No Twitter application would be complete without the ability to post a tweet! Fortunately, posting a tweet is actually quite easy. Play around with the function

twitter.update_status() or take a look at postStatus() function we’ve written.

Wordcloud

Now that we’ve followed a user, let’s take a look at some of their tweets using a wordcloud. If you’ve never seen a wordcloud before, it is a graphic that depicts the relative frequency of words using their size. Here’s an example, using data about common passwords. We’ve given you the code to draw a basic wordcloud. Your task is to build a dictionary that keeps track of the frequency of words in a set of tweets.

First, write a function join(tweets) which takes a list of tweets and puts all of the words in one big list (repeats are allowed). It’ll be helpful to use the getText(tweet) function inside your block definition. To get a user’s tweets, you can use the get_user_timeline function. See below for an example.

Next, write a function buildCount(words) that takes a list of (possibly repeated) words and returns a dictionary with their frequencies. E.g.:

>>> buildCount([“hello”, “bye”, “hello”, “my”, “name”, “is”, “John”, “this”, “is”, “fun”])

{‘hello’: 2, ‘bye’: 1, ‘my’: 1, ‘John’: 1, ‘is’: 2, ‘name’: 1, ‘this’: 1, ‘fun’: 1}

(dictionary may be in a different order)

This will essentially entail building a dictionary, looping over all of the words, and updating the dictionary of counts every time you see an instance of a particular word.

Finally, create the wordcloud by passing in the result of buildCount to the function wordcloud.drawCloud, like so:

>>> tweets = twitter.get_user_timeline(screen

>>> freq = buildCount(join(tweets))

>>> wordcloud.drawCloud(freq)

Going Beyond

What’s given here is a good start at a basic Twitter application, however there are millions of ways you can continue to build out what you can do with Twitter. Here are a few simple ideas.

How Many Tweets Should You See?

By default, Twitter returns the most recent 20 tweets. However, sometimes you might want to see more. There are some optional parameters you supply to change what you see. Modify the displayTimeline() function to show more than 20 tweets.

Sending Direct Messages

Retweeting What’s ‘Popular’

#Hashtags

Finding Trending Topics

Getting your own API Key

While we’ve done some of the work to setup our own Twitter application, you might want to build your own! Twitter makes it pretty each for anyone to create a new application. All you need is a Twitter account. Once you’ve signed in, head over to Twitter’s developer site to create your own application:

https://apps.twitter.com/app/new

This will give you a set of “keys” sometimes also referred to as “tokens”. These are what help Twitter verify the identity of the person connecting to their services. Take a look at the setup.py file to see some of the code we’ve written to handle the sign in process.