Plotting Exercises, Part 2¶
Wealth and Democracy¶
Let’s now pivot from working with example data to real data. Load the World Development Indicator data you worked with over the summer. This is country-level data that includes information on both countries’ GDP per capita (a measure of wealth) and the Polity IV scores (a measure of how democratic a country is – countries with higher scores are liberal democracies, countries with low scores are autocratic.). Use the code below to download the data.
import pandas as pd wdi = pd.read_csv( "https://raw.githubusercontent.com/nickeubank/practicaldatascience" "/master/Example_Data/world-small.csv" )
Your data should look like this:
Let’s being analyzing this data by estimating a simple linear model (“ordinary least squares”) of the relationship between GDP per capita (
gdppcap08) and democracy scores (
polityIV). We will do so using the
statsmodel package, which we’ll discuss in detail later is this course. For the momement, just use this code:
import statsmodels.formula.api as smf results = smf.ols('polityIV ~ gdppcap08', data=wdi).fit() print(results.summary())
Based on the results of this analysis, what would you conclude about about the relationship between
(If you aren’t familiar with Linear Models and aren’t sure how to interprete this, you can also just look at the simple correlation between these two variables using
Write down your conclusions.
Now let’s plot the relationship you just estimated statistically. First, use Altair to create a scatter plot of
Now overlay the linear model you just estimated.
Does it seem like the linear model you estimated fits the data well?
Linear models impose a very strict functional form on the model they use: they try to draw a straight line through the data, no matter what.
Can you think of a transform for your data that would make the data a little more sane?
Once you’ve applied that transformation, let’s re-fit our model. Rather than imposing linearity this time, however, let’s fit a model with a flexible functional form. Use
transform_loess() to fit an updated model over your data. This is a form of local polynomial regression that is designed to be flexible in how it fits the data.
This does seem to fit the data better, but there’s clearly this HUGE outlier in the bottom right. Who is that? Add text labels to the points on your graph with country names.
Interesting. It seems that there’s are a lot of rich, undemocratic countries that all have something in common: they’re oil-rich, small, Middle Eastern countries.
Let’s see what happens if we exclude the ten countries with the highest per-capita oil production from our data: Qatar, Kuwait, Equatorial Guinea, United Arab Emirates, Norway, Saudi Arabia, Libya, Oman, Gabon, and Angola. (Note this was in 2007, and excludes very small countries!)
What does the relationship between Polity and GDP per capita look like for non-natural resource producers?
Let’s make sure that you accurately identified all 10 of the oil producers. Write a line of code to count up how many big producers you have identified. If you do not get 10, can you figure out what you did wrong?
How does the relationship between GDP per capita and Polity look for the oil producers we dropped above?
Look back to your answer for Exercise 2. Do you still believe the result of your linear model? What did you learn from plotting. Write down your answers with your partner.
Finally, let’s make an interactive chart to help readers explore this relationship themselves! Create an interactive chart with a dropdown menu to select big oil producers / non-big oil producers, and add a mouseover to allow users to check the names of countries.
(You might want to review the interactive chart directions)
One of our main jobs as data scientists is to summarize data. In fact, its such an obvious part of our jobs we often don’t think about it very much. In reality, however, this is one of the most difficult things we do.
Summarization means taking rich, complex data and trying to tell readers about what is going on in that data using simple statistics. In the process of summarization, therefore, we must necessarily throw away much of the richness of the original data. When done well, this simplification makes data easier to understand, but only if we throw away the right data. You can always calulate the average value of a variable, or fit a linear model, but whether doing so generates a summary statistic that properly represents the essence of the data being studied depends on the data itself.
Plotting is one fo the best tools we have as data scientists for evaluating whether we are throwing away the right data. As we learned from Part 1 of this exercise, just looking at means and standard deviations can mask tremendous variation. Each of our example datasets looked the same when we examined our summary statistics, but they were all radically different when plotted.
Similarly, a simple linear model would “tell” us that if GDP per capita increases by $10,000, we would expect Polity scores to increase by about 1 (i.e. the coefficent on the linear model was 9.602e-05). But when we plot the data, not only can we that the data is definitely not linear (and so that slope doesn’t really mean anything), but we can also see that oil producing countries seem to defy the overall trend, and so should maybe be studied separately.
Moreover, we can see that if we just look at oil producers, there is no clear story: some are rich and democratic, while others are rich and autocratic (indeed, this observation is the foundation of some great research on the political consequences of resource wealth!)
So remember this: tools for summarizing data will always give you an answer, but it’s up to you as a data scientist to make sure that the summaries you pass on to other people properly represent the data you’re using. And there is perhaps no better way to do this than with plotting!