Plotting Exercises, Part 2

Wealth and Democracy

Let’s now pivot from working with example data to real data. Load the World Development Indicator data you worked with over the summer. This is country-level data that includes information on both countries’ GDP per capita (a measure of wealth) and the Polity IV scores (a measure of how democratic a country is – countries with higher scores are liberal democracies, countries with low scores are autocratic.). Use the code below to download the data.

[77]:
wdi = pd.read_csv('https://raw.githubusercontent.com/nickeubank/practicaldatascience/master/Example_Data/world-small.csv')

Your data should look like this:

[78]:
wdi.head()
[78]:
country region gdppcap08 polityIV
0 Albania C&E Europe 7715 17.8
1 Algeria Africa 8033 10.0
2 Angola Africa 5899 8.0
3 Argentina S. America 14333 18.0
4 Armenia C&E Europe 6070 15.0

Exercise 1

Let’s being analyzing this data by estimating a simple linear model (“ordinary least squares”) of the relationship between GDP per capita (gdppcap08) and democracy scores (polityIV). We will do so using the statsmodel package, which we’ll discuss in detail later is this course. For the momement, just use this code:

import statsmodels.formula.api as smf
results = smf.ols('polityIV ~ gdppcap08',
                   data=wdi).fit()
print(results.summary())

Exercise 2

Based on the results of this analysis, what would you conclude about about the relationship between gdppcap08 and polityIV?

(If you aren’t familiar with Linear Models and aren’t sure how to interprete this, you can also just look at the simple correlation between these two variables using wdi[['polityIV', 'gdppcap08']].corr().)

Write down your conclusions.

Exercise 3

Now let’s plot the relationship you just estimated statistically. First, use plotnine to create a scatter plot of polityIV and gdppcap08.

Exercise 4

Now overlay the linear model you just estimated. You can do this by adding a geom_smooth() layer, where the method argument is set to 'lm' (for linear model).

Exercise 5

Does it seem like the linear model you estimated fits the data well?

Exercise 6

Linear models impose a very strict functional form on the model they use: they try to draw a straight line through the data, no matter what. Let’s consider a more flexible functional form. Change the method in your geom_smooth to "lowess". This is a form of local polynomial regression that is designed to be flexible in how it fits the data.

Exercise 7

This does seem to fit the data better, but there’s clearly this HUGE outlier in the bottom right. Who is that? Using geom_text(), label the points on your graph with country names.

Exercise 8

Interesting. It seems that there’s are a lot of rich, undemocratic countries that all have something in common: they’re oil-rich, small, Middle Eastern countries.

Let’s see what happens if we exclude the ten countries with the highest per-capita oil production from our data: Qatar, Kuwait, Equatorial Guinea, United Arab Emirates (UAE), Norway, Saudi Arabia, Libya, Oman, Gabon, and Angola. (Note this was in 2007, and excludes very small countries!)

To do this, I would recomment creating a new variable called bigproducer that is True if country matches a name in that list, and False otherwise. If you’re struggling to find a good way to do this, googling “How to check if a value is in the list in selection from pandas data frame?” might help you out…

Exercise 9

Let’s make sure that you accurately identified all 10 of the oil producers. Write a line of code to count up how many big producers you have identified. If you do not get 10, can you figure out what you did wrong?

Exercise 10

How does the relationship between GDP per capita and Polity look without those oil producers? Does it look the same as it did without the oil producers?

Exercise 11

Now that we’ve gotten a good sense of the relationship between wealth and democracy for non-oil producers, can we draw any conclusions about the relationship between polity scores and wealth for the oil producers? Plot the polity / GDP per capita relationship just for the oil producers.

Exercise 12

Look back to your answer for Exercise 2. Do you still believe the result of your linear model? What did you learn from plotting. Write down your answers with your partner.

Take-aways

One of our main jobs as data scientists is to summarize data. In fact, its such an obvious part of our jobs we often don’t think about it very much. In reality, however, this is one of the most difficult things we do.

Summarization means taking rich, complex data and trying to tell readers about what is going on in that data using simple statistics. In the process of summarization, therefore, we must necessarily throw away much of the richness of the original data. When done well, this simplification makes data easier to understand, but only if we throw away the right data. You can always calulate the average value of a variable, or fit a linear model, but whether doing so generates a summary statistic that properly represents the essence of the data being studied depends on the data itself.

Plotting is one fo the best tools we have as data scientists for evaluating whether we are throwing away the right data. As we learned from Part 1 of this exercise, just looking at means and standard deviations can mask tremendous variation. Each of our example datasets looked the same when we examined our summary statistics, but they were all radically different when plotted.

Similarly, a simple linear model would “tell” us that if GDP per capita increases by $10,000, we would expect Polity scores to increase by about 1 (i.e. the coefficent on the linear model was 9.602e-05). But when we plot the data, not only can we that the data is definitely not linear (and so that slope doesn’t really mean anything), but we can also see that oil producing countries seem to defy the overall trend, and so should maybe be studied separately.

Moreover, we can see that if we just look at oil producers, there is no clear story: some are rich and democratic, while others are rich and autocratic (indeed, this observation is the foundation of some great research on the political consequences of resource wealth!)

So remember this: tools for summarizing data will always give you an answer, but it’s up to you as a data scientist to make sure that the summaries you pass on to other people properly represent the data you’re using. And there is perhaps no better way to do this than with plotting!

[ ]: