{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Missing Values Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradescope Autograding\n",
    "\n",
    "Please follow [all standard guidance](https://www.practicaldatascience.org/html/autograder_guidelines.html) for submitting this assignment to the Gradescope autograder, including storing your solutions in a dictionary called `results` and ensuring your notebook runs from the start to completion without any errors.\n",
    "\n",
    "For this assignment, please name your file `exercise_missing.ipynb` before uploading.\n",
    "\n",
    "You can check that you have answers for all questions in your `results` dictionary with this code:\n",
    "\n",
    "\n",
    "```python\n",
    "assert set(results.keys()) == {\n",
    "    \"ex2_avg_income\",\n",
    "    \"ex3_share_making_9999999\",\n",
    "    \"ex3_share_making_zero\",\n",
    "    \"ex5_avg_income\",\n",
    "    \"ex8_avg_income_black\",\n",
    "    \"ex8_avg_income_white\",\n",
    "    \"ex8_racial_difference\",\n",
    "    \"ex9_avg_income_black\",\n",
    "    \"ex9_avg_income_white\",\n",
    "    \"ex10_wage_gap\",\n",
    "}\n",
    "```\n",
    "\n",
    "### Submission Limits\n",
    "\n",
    "Please remember that you are **only allowed three submissions to the autograder.** Your last submission (if you submit 3 or fewer times), or your third submission (if you submit more than 3 times) will determine your grade Submissions that error out will **not** count against this total.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Today, we will be using the ACS data we used during out first `pandas` exercise to examine the US income distribution, and how it varies by race. Note that because the US income distribution has a very small number of people with *extremely* high incomes, and the ACS is just a sample of Americans, the far right tail of the distribution will not be very well estimated. However, this data should suffice for helping to understand wealth inequality in the United States. \n",
    "\n",
    "To begin, load the ACS Data we used in our first pandas exercise. That [data can be found here](https://github.com/nickeubank/MIDS_Data/tree/master/US_AmericanCommunitySurvey). We'll be working with `US_ACS_2017_10pct_sample.dta`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "\n",
    "Let's begin by calculating the mean US incomes from this data (recall that income is stored in the `inctot` variable). Store the answer in `results` under the key `\"ex2_avg_income\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "\n",
    "Hmmm... That doesn't look right. The average American is definitely not earning that much a year! Let's look at the values of `inctot` using `value_counts()`. Do you see a problem?\n",
    "\n",
    "Now use `value_counts()` with the argument `normalize=True` to see proportions of the sample that report each value instead of the count of people in each category. What percentage of our sample has an income of 9,999,999? Store that proportion (between 0 and 1) as `\"ex3_share_making_9999999\"`. What percentage has an income of 0? Store that proportion as `\"ex3_share_making_zero\"`.\n",
    "\n",
    "(Recall `.value_counts()` returns a Series, so you can pull values out with our usual pandas tools.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "As we discussed before, the ACS uses a value of 9999999 to denote that income information is not available for someone. The problem with using this kind of \"sentinel value\" is that pandas doesn't understand that this is supposed to denote missing data, and so when it averages the variable, it doesn't know to ignore 9999999. \n",
    "\n",
    "To help out `pandas`, use the `replace` command to replace all values of 9999999 with `np.nan`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5\n",
    "\n",
    "Now that we've properly labeled our missing data as `np.nan`, let's calculate the average US income once more. Store the answer in `results` under the key `\"ex5_avg_income\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6\n",
    "\n",
    "OK, now we've been able to get a reasonable average income number. As we can see, a major advantage of using `np.nan` is that `pandas` knows that `np.nan` observations should just be ignored when we are calculating means. \n",
    "\n",
    "But it's not enough to just get rid of the people who had `inctot` values of 9999999. We also need to know why those values were missing. Suppose, for example, that the value of 9999999 was used for anyone who made more than 100,000 dollars: if we just dropped those people, then our estimate of average income wouldn't mean much, would it?\n",
    "\n",
    "So let's make sure we understand *why* data is missing for some people. If you recall from our last exercise, it seemed to be the case that most of the people who had incomes of 9999999 were children. Let's make sure that's true by looking at the distribution of the variable `age` for people for whom `inctot` is missing (i.e. subset the data to people with `inctot` missing, then look at the values of `age` with `value_counts()`).\n",
    "\n",
    "Then do the opposite: look at the distribution of the `age` variable for people who whom `inctot` is *not* missing. \n",
    "\n",
    "Can you determine when 9999999 was being used? Is it ok we're excluding those people from our analysis?\n",
    "\n",
    "Note: In this data, Python doesn't understand `age` is a number; it thinks it is a string because the original data has categories like \"90 (90+ in 1980 and 1990)\" and \"less than 1 year old\". So you can't just use `min()` or `max()`. We'll discuss converting string variables into numbers in a future class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 7\n",
    "\n",
    "Great, so now we know why those people had missing data, and we're ok with excluding them. \n",
    "\n",
    "But as we previously noted, there are also a lot of observations of zero income in our data, and it's not clear that we want everyone with a zero-income *should* be included in this average, since those may be people who are retired, or in school. \n",
    "\n",
    "Let's limit our attention to people who are currently working by subsetting to only employed respondents. We can do this using `empstat`. Remember you can use `value_counts()` to see what values of `empstat` are in the data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 8\n",
    "\n",
    "Now let's estimate the racial income gap in the United States. What is the average salary for employed Black Americans, and what is the average salary for employed White Americans? In percentage terms, how much more does the average White American make than the average Black American?\n",
    "\n",
    "**Note:** these values are not quite accurate estimates. As we'll discuss in later lessons, to get completely accurate estimates from the ACS we have to take into account how people were selected to be interviewed. But you get pretty good estimates in most cases even without weights—your estimate of the racial wage gap without weights is within 5\\% of the corrected value. \n",
    "\n",
    "**Note:** This is actually an underestimate of the wage gap. The US Census treats Hispanic respondents as a sub-category of \"White.\" While all ethnic distinctions are socially constructed, and so on some level these distinctions are all deeply problematic, this coding is inconsistent with what most Americans think of when they hear the term \"White,\" a term *most* Americans think of as a category that is mutually exclusive of being Hispanic or Latino (categories which are also usually conflated in American popular discussion). With that in mind, most researchers working with US Census data split \"White\" into \"White, Hispanic\" and \"White, Non-Hispanic\" using `race` *and* `hispan`. But for the moment, just identify \"White\" respondents using the value in `race`.\n",
    "\n",
    "Store your results in `results` under the keys `\"ex8_avg_income_black\"`, `\"ex8_avg_income_white\"`, and the percentage difference as `ex8_racial_difference`. Please note the wording above when calculating the percentage difference to ensure you get the reference category correct, and interpret your result as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 9\n",
    "\n",
    "\n",
    "As noted above, these estimates are not actually *quite* correct because we aren't using survey weights. To calculate a weighted average that takes into account survey weights, you need to use the following formula:\n",
    "\n",
    "$$weighted\\_mean\\_of\\_x = \\frac{\\sum_i x_i * weight_i}{\\sum_i weight_i}$$\n",
    "\n",
    "(As you can see, when $weight_i$ is constant for all observations, this just simplifies to our normal formula for mean values. It is only when weights vary across individuals that weights must be explicitly addressed).\n",
    "\n",
    "In this data, weights are stored in the variable `perwt`, which is the number of people for which each observation is a stand-in (the inverse of that observations sampling probability). \n",
    "\n",
    "Using the formula, re-calculate the *weighted* average income for both populations and store them as `ex9_avg_income_white` and `ex9_avg_income_black`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 10\n",
    "\n",
    "Now calculate the weighted average income gap between *non-Hispanic* White Americans and Black Americans. What percentage more do employed White non-Hispanic Americans earn than employed Black Americans? Store as `\"ex10_wage_gap\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 11\n",
    "\n",
    "Is that greater or less than the difference you found in Exercise 8? Why do you think that's the case?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "718fed28bf9f8c7851519acf2fb923cd655120b36de3b67253eeb0428bd33d2d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}