Estimating Labor Market Returns to Education¶

In this exercise, we’re going to use data from the American Communities Survey (ACS) to study the relationship betwen educational attainment and wages. The ACS is a survey conducted by the United States Census Bureau (though it is not “The Census,” which is a counting of every person in the United States that takes place every 10 years) to measure numerous features of the US population. The data we will be working with includes about 100 variables from the 2017 ACS survey, and is a 10% sample of the ACS (which itself is a 1% sample of the US population, so we’re working with about a 0.1% sample of the United States).

This data comes from IPUMS, which provides a very useful tool for getting subsets of major survey datasets, not just from the US, but from government statistical agencies the world over.

This is real data, meaning that you are being provided the data as it is provided by IPUMS. Documentation for all variables used in this data can be found here (you can either search by variable name to figure out the meaning of a variable in this data, or search for something you want to see if a variable with the right name is in this data).

Within this data is information on both the educational background and current earnings of a representative sample of Americans. We will now use this data to estimate the labor-market returns to graduating high school and college, and to learn something about the meaning of an educational degree.

Gradescope Autograding¶

Please follow all standard guidance for submitting this assignment to the Gradescope autograder, including storing your solutions in a dictionary called results and ensuring your notebook runs from the start to completion without any errors.

For this assignment, please name your file exercise_dataframes.ipynb before uploading.

You can check that you have answers for all questions in your results dictionary with this code:

assert set(results.keys()) == {
    "ex2_num_obs",
    "ex3_num_vars",
    "ex8_updated_num_obs",
    "ex9_updated_num_obs",
    "ex11_grade12_income",
    "ex12_college_income",
    "ex12_college_income_pct",
    "ex14_high_school_dropout",
    "ex15_grade_9",
    "ex15_grade_10",
    "ex15_grade_11",
    "ex15_grade_12",
    "ex15_4_years_of_college",
    "ex15_graduate",
}

Submission Limits¶

Please remember that you are only allowed three submissions to the autograder. Your last submission (if you submit 3 or fewer times), or your third submission (if you submit more than 3 times) will determine your grade Submissions that error out will not count against this total.

Exercises¶

Exercise 1¶

Data for these exercises can be found here.

Import US_ACS_2017_10pct_sample.dta into a pandas DataFrame (read it directly from a URL to help the autograder, please).

This can be done with the command pd.read_stata, which will read in files created in the program Stata (and which uses the file suffix .dta). This is a format commonly used by social scientists.

Getting to Know Your Data¶

When you get a new dataset like this, it’s good to start by trying to get a feel for its contents and organization. Toy datasets you sometimes get in classes are often very small, and easy to look at, but this is a pretty large dataset, so you can’t just open it up and get a good sense of it. Here are some ways to get to know your data.

Exercise 2¶

How many observations are in your data? Store the answer in your results dictionary with the key "ex2_num_obs".

Exercise 3¶

How many variables are in your data? Store the answer in your results dictionary with the key "ex3_num_vars".

Exercise 4¶

Let’s see what variables are in this dataset. First, try to see them all using the command:

acs.columns

As you will see, python doesn’t like to print out all the different variables when there are this many in a dataset.

To get everything printed out, we can loop over all the columns and print them one at a time with the command:

for c in acs.columns: print(c)

It’s definitely a bit of a hack, but honestly a pretty useful one!

Exercise 5¶

That’s a lot of variables, and definitely more than we need. In general, life is easier when working with these kinds of huge datasets if you can narrow down the number of variables a little. In this exercise, we will be looking at the relationship between education and wages, we need variables for:

Age
Income
Education
Employment status (is the person actually working)

These quantities of interest correspond to the following variables in our data: age, inctot, educ, and empstat.

Subset your data to just those variables.

Exercise 6¶

Now that we have a more manageable number of variables, it’s often very useful to look at a handful of rows of your data. The easiest way to do this is probably the .head() method (which will show you the first five rows), or the tail() method, which will show you the last five rows.

But to get a good sense of your data, it’s often better to use the sample() command, which returns a random set of rows. As the first and last rows are sometimes not representative, a random set of rows can be very helpful. Try looking at a random sample of 20 rows (note: you don’t have to run .sample() ten times to get ten rows. Look at the .sample help file if you’re stuck.

Exercise 7¶

Do you see any immediate problems? What issues do you see? (Please do answer in markdown)

Exercise 8¶

One problem is that many people seem to have incomes of $9,999,999. Moreover, people with those incomes seem to be very young children.

What you are seeing is one method (a relatively old one) for representing missing data. In this case, the value 9999999 is being used as a sentinel value — a way to denote missing data that was used back in the day when there was no way to add a special data type for mossing data. In this case, it identifies observations where the person is too young to work, so their income value is missing.

So let’s begin by dropping anyone who has inctot equal to 9999999.

After dropping, how many observations do you have? Save your answer in your results dictionary under the key "ex8_updated_num_obs"

Exercise 9¶

OK, the other potential problem is that our data includes lots of people who are unemployed and people who are not in the labor force (this means they not only don’t have a job, but also aren’t looking for a job). For this analysis, we want to focus on the wages of people who are currently employed. So subset the dataset for the people for whom empstat is equal to “employed”.

Note that our decision to only look at people who are employed impacts how we should interpret the relationship we estimate between education and income. Because we are only looking at employed people, we will be estimating the relationship between education and income for people who are employed. That means that if education affects the likelihood someone is employed, we won’t capture that in this analysis.

(You might also want to run .sample() after this just to make sure you were successful in your subsetting).

After this subsetting, how many observations do you have? Save your answer in your results dictionary under the key "ex9_updated_num_obs"

Exercise 10¶

Now let’s turn to education. The educ variable seems to have a lot of discrete values. Let’s see what values exist, and their distribution, using the value_counts() method. This is an extremely useful tool you’ll use a lot! Try the following code (modified for the name of your dataset, of course):

acs["educ"].value_counts()

Exercise 11¶

There are a lot of values in here, so let’s just check a couple. What is the average value of inctot for people whose highest grade level is “grade 12” (in the US, that is someone who has graduated high school)?

Save your answer in your results dictionary under the key "ex11_grade12_income".

Exercise 12¶

What is the average income of someone who has completed an undergraduate degree but not done any postgraduate education (“4 years of college”)?

Save your answer in your results dictionary under the key "ex12_college_income".

In percentage terms, how much does an employed college graduate earn as compared to someone who is only a high school graduate? Use the reference category that gives an answer above 100.

Store your answer in "ex12_college_income_pct". Put your answer in percentage terms (so 100 implies they earn the same amount).

Make sure to interpret your result in words when you print it out!

Exercise 13¶

What does that suggest is the value of getting a college degree after graduating high school?

Exercise 14¶

What is the average income for someone who has not finished high school? What does that suggest is the value of a high school diploma? (Treat n/a or no schooling as having no formal schooling, not as missing).

Hint: You may find the .isin() method to be really helpful here.

Save your answer in your results dictionary under the key "ex14_high_school_dropout".

Exercise 15¶

Complete the following table (storing values under the provided keys where listed):

Average income for someone who only completed 9th grade (ex15_grade_9): _________
Average income for someone who only completed 10th grade (ex15_grade_10): _________
Average income for someone who only completed 11th grade (ex15_grade_11): _________
Average income for someone who finished high school (12th grade) but never started college (ex15_grade_12): _________
Average income for someone who completed 4 year of college (in the US, this corresponds to getting an undergraduate degree), but has no post-graduate education (no more than 4 years, ex15_4_years_of_college): _________
Average income for someone who has some graduate education (more than 4 years, ex15_graduate): _________

Exercise 16¶

Why do you think there is no benefit from moving from grade 9 to grade 10, or grade 10 to grade 11, but there is a huge benefit to moving from grade 11 to graduating high school (grade 12)?

(Think carefully before reading ahead!)

Take-aways¶

Congratulations! You just discovered “the sheepskin effect!”: people with degrees tend to earn substantially more than people who have almost as much education, but don’t have an actual degree.

In economics, this is viewed as evidence that the reason employers pay people with high school degrees more than those without degree is not that they think those who graduated high school have learned specific, useful skills. If that were the case, we would expect employee earnings to rise with every year of high school, since in each year of high school we learn more.

Instead, this suggests employees pay high school graduates more because they think the kind of people who can finish high school are the kind of people who are likely to succeed at their jobs. Finishing high school, in other words, isn’t about accumulating specific knowledge; it’s about showing that you are the kind of person who can rise to the challenge of finishing high school, also suggesting you are also the kind of person who can succeed as an employee.

(Obviously, this does not tell us whether that is an accurate inference, just that that seems to be how employeers think.)

In other words, in the eyes of employers, a high school degree is a signal about the kind of person you are, not certification that you’ve learned a specific set of skills (an idea that earned Michael Spence a Nobel Prize in Economics).