Plotting Exercises, Part 2

Wealth and Democracy

Let’s now pivot from working with example data to real data. Load the World Development Indicator data you worked with over the summer. This is country-level data that includes information on both countries’ GDP per capita (a measure of wealth) and the Polity IV scores (a measure of how democratic a country is – countries with higher scores are liberal democracies, countries with low scores are autocratic.). Use the code below to download the data.

[1]:
import pandas as pd
wdi = pd.read_csv('https://raw.githubusercontent.com/nickeubank/practicaldatascience/master/Example_Data/world-small.csv')

Your data should look like this:

[2]:
wdi.head()
[2]:
country region gdppcap08 polityIV
0 Albania C&E Europe 7715 17.8
1 Algeria Africa 8033 10.0
2 Angola Africa 5899 8.0
3 Argentina S. America 14333 18.0
4 Armenia C&E Europe 6070 15.0

Exercise 1

Let’s being analyzing this data by estimating a simple linear model (“ordinary least squares”) of the relationship between GDP per capita (gdppcap08) and democracy scores (polityIV). We will do so using the statsmodel package, which we’ll discuss in detail later is this course. For the momement, just use this code:

import statsmodels.formula.api as smf
results = smf.ols('polityIV ~ gdppcap08',
                   data=wdi).fit()
print(results.summary())

Exercise 2

Based on the results of this analysis, what would you conclude about about the relationship between gdppcap08 and polityIV?

(If you aren’t familiar with Linear Models and aren’t sure how to interprete this, you can also just look at the simple correlation between these two variables using wdi[['polityIV', 'gdppcap08']].corr().)

Write down your conclusions.

Exercise 3

Now let’s plot the relationship you just estimated statistically. First, use plotnine to create a scatter plot of polityIV and gdppcap08.

Exercise 4

Now overlay the linear model you just estimated. You can do this by adding a geom_smooth() layer, where the method argument is set to 'lm' (for linear model).

Exercise 5

Does it seem like the linear model you estimated fits the data well?

Exercise 6

Linear models impose a very strict functional form on the model they use: they try to draw a straight line through the data, no matter what. Let’s consider a more flexible functional form. Change the method in your geom_smooth to "lowess". This is a form of local polynomial regression that is designed to be flexible in how it fits the data.

Exercise 7

This does seem to fit the data better, but there’s clearly this HUGE outlier in the bottom right. Who is that? Using geom_text(), label the points on your graph with country names.

Exercise 8

Interesting. It seems that there’s are a lot of rich, undemocratic countries that all have something in common: they’re oil-rich, small, Middle Eastern countries.

Let’s see what happens if we exclude the ten countries with the highest per-capita oil production from our data: Qatar, Kuwait, Equatorial Guinea, United Arab Emirates, Norway, Saudi Arabia, Libya, Oman, Gabon, and Angola. (Note this was in 2007, and excludes very small countries!)

What does the relationship between Polity and GDP per capita look like for non-natural resource producers?

Exercise 9

Let’s make sure that you accurately identified all 10 of the oil producers. Write a line of code to count up how many big producers you have identified. If you do not get 10, can you figure out what you did wrong?

Exercise 10

How does the relationship between GDP per capita and Polity look for the oil producers we dropped above?

Exercise 11

Look back to your answer for Exercise 2. Do you still believe the result of your linear model? What did you learn from plotting. Write down your answers with your partner.

Exercise 12

Pick one of your plots and make it look good – label your axes, add a title, try to make the county labels readable.

Take-aways

One of our main jobs as data scientists is to summarize data. In fact, its such an obvious part of our jobs we often don’t think about it very much. In reality, however, this is one of the most difficult things we do.

Summarization means taking rich, complex data and trying to tell readers about what is going on in that data using simple statistics. In the process of summarization, therefore, we must necessarily throw away much of the richness of the original data. When done well, this simplification makes data easier to understand, but only if we throw away the right data. You can always calulate the average value of a variable, or fit a linear model, but whether doing so generates a summary statistic that properly represents the essence of the data being studied depends on the data itself.

Plotting is one fo the best tools we have as data scientists for evaluating whether we are throwing away the right data. As we learned from Part 1 of this exercise, just looking at means and standard deviations can mask tremendous variation. Each of our example datasets looked the same when we examined our summary statistics, but they were all radically different when plotted.

Similarly, a simple linear model would “tell” us that if GDP per capita increases by $10,000, we would expect Polity scores to increase by about 1 (i.e. the coefficent on the linear model was 9.602e-05). But when we plot the data, not only can we that the data is definitely not linear (and so that slope doesn’t really mean anything), but we can also see that oil producing countries seem to defy the overall trend, and so should maybe be studied separately.

Moreover, we can see that if we just look at oil producers, there is no clear story: some are rich and democratic, while others are rich and autocratic (indeed, this observation is the foundation of some great research on the political consequences of resource wealth!)

So remember this: tools for summarizing data will always give you an answer, but it’s up to you as a data scientist to make sure that the summaries you pass on to other people properly represent the data you’re using. And there is perhaps no better way to do this than with plotting!

Want more practice?

Great!

First, if you haven’t done so already, do the “Do you want more practice” exercise from the first set of plotting exercises (looking at the gender wage gap and racial age gap across age cohorts).

If you’ve don’t that, then load the ACS data that we used for our first DataFrames exercise and create a plot that illustrates the sheepskin effect. Note that because the “sheepskin” effect is fundamentally about discontinuities, you won’t be able to illustrate it using a single geom_smooth object. Rather, you’ll probably want to plot a linear or non-parametric fit for years of education below finishing high school, then plot the value of finishing high school above that.

As a second extension, you can also plot out the income consequences of different levels of college achievement.

Absolutely positively need the solutions?

Don’t use this link until you’ve really, really spent time struggling with your code! Doing so only results in you cheating yourself.

Link