{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Coding Your Own Linear Regression Model\n",
"\n",
"One task that you will almost certainly be required to do other data science courses (especially if you are a MIDS student) is to write up some of your statistical / machine learning models from scratch. This can be a very valuable exercise, as it ensures that you understand what is actually going on behind the scenes of the models you use ever day, and that you don't just think of them as \"black boxes\". \n",
"\n",
"To get a little practice doing this, today you will be coding up your own linear regression model! \n",
"\n",
"(If you are using this site but aren't actually in this class, you are welcome to skip this exercise if you'd like -- this is more about practicing Python in anticipation of the requirements of other courses than developing your applied data science skills.) \n",
"\n",
"There are, broadly speaking, two approaches you can take to coding up your own model: \n",
"\n",
"1. you can write the model by defining a new function, or \n",
"2. you can write the model by defining a new class with associated methods (making a model that works the way a model works in `scikit-learn`). \n",
"\n",
"Whether you do 1 or 2 is very much a matter of choice and style. Approach one, for example, is more consistent with what is called a *functional* style of programming, while approach two is more consistent with an *object-oriented* style of programming. Python can readily support both approaches, so either would work fine. \n",
"\n",
"In these exercises, however, I will ask you to use approach number 2 as this *tends* to be the more difficult approach, and so practicing approach 2 will be extra useful in preparing you for other classes (HA! Pun...). In particular, our goal is to implement a linear regression model that has the same \"initialize-fit-predict-score\" API (application programming interface -- a fancy name for the methods a class supports) as `scikit-learn` models. That means your model should be able to do all of the following:\n",
"\n",
"1. **Initialize** a new model. \n",
"2. **Fit** a linear model when given a numpy vector (`y`) and a numpy matrix (`X`) with the syntax `my_model.fit(X, y)`. \n",
"3. **Predict** values when given a new `numpy` matrix (`X_test`) with the syntax `my_model.predict(X_test)`. \n",
"4. Return the **model coefficients** through the property `my_model.coefficients` (not quite what is used in `sklearn`, but let's use that interface). \n",
"\n",
"Also, bear in mind that throughout these exercises, we'll be working in `numpy` instead of `pandas`, just as we do in `scikit-learn`. So assume that before using your model, your user has already converted their data from `pandas` into `numpy` arrays. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(1)** Define a new Class called `MyLinearModel` with methods for `__init__`, `fit`, `predict`, and an attribute for `coefficients`. For now, we don't need any initialization *arguments*, just an `__init__` function. \n",
"\n",
"As you get your code outline going, start by just having each method `pass`:\n",
"\n",
"```python\n",
"def my_method(self):\n",
" pass\n",
"```\n",
"\n",
"This will allow your methods to run without errors (they just don't do anything). Then we can double back to each method to get them working one by one."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(2)** Now define your `fit` method. This is the method that should actually run your linear regression. In case you've forgotten your linear algebra, remember that for linear regressions, $\\beta = (X'X)^{-1}X'Y$, a fact you can see explained in detail on page four [here](https://www.stat.purdue.edu/~boli/stat512/lectures/topic3.pdf).\n",
"\n",
"Note that once you have written the code to do a linear regression, you'll need to put your outputs (your coefficents) somewhere. I recommend making an attribute for your class where you can store your coefficients. \n",
"\n",
"(As a reminder: the normal multiply operator (`*`) in `numpy` implies scalar multiplication. Use `@` for matrix multiplication). \n",
"\n",
"**HINT:** Remember that linear regressions usually include a constant term. As in most packages, you should assume that users want this included, even if there isn't a vector of 1s in their `X`. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(3)** As you write code, it is good to test your code as you work. With that in mind, let's create some toy data. First, create a 100 x 2 matrix where each column is normally distributed. Then create a vector `y` that is a linear combination of those two columns plus a vector of normally distributed noise. \n",
"\n",
"In other words, we want to create data where we *know* exactly what coefficients we should see so when we test our regression, we know if the results are accurate!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(4)** Now test whether you `fit` method generates the correct coefficients. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(5)** Now let's make the statisticians proud, and in addition to storing the coefficients, let's store the standard errors for our estimated coefficients as another attribute. Recall that the simplest method of calculating the variance covariance matrix for $\\beta$ is using the formula $\\sigma^2 (X'X)^{-1}$, where $\\sigma^2$ is the variance of the error terms of your regression. See page six [here](https://www.stat.purdue.edu/~boli/stat512/lectures/topic3.pdf) for a full derivation. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(6)** Now let's also add an R-squarded attribute to the model."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(7)** Now we'll go ahead and cheat a little. Use `statsmodels` to see if your standard errors and r-squared are correct!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(8)** Now implement `predict`! Then test it against your original `X` data -- do you get back something very close to your true `y`?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**(9)** Finally, create the *option* of fitting the model with or without a constant term. As in `scikit-learn`, make this an option you set during initialization. "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}