PDS (MIDS) (IDS 720)

A one semester version of Practical Data Science specifically tailored to Masters of Interdisciplinary Data Science (MIDS) students. As all MIDS students complete a mandatory, 4-week, in-person, intensive summer Python programming bootcamp in August before the start of classes, this class assumes a strong foundational understanding of the Python standard library. Because this class is a one-semester class and MIDS students all take a full NLP class, it skips some topics covered in Practical Data Science II, like NLP and GIS.

Syllabus

Course syllabus can be found here.

Office Hours and Links

Nick (Zoom or in Gross 231): Thursday 8:45 am - 9:45 am. Please email me in advance if you plan to attend. I’ll usually be there regardless, but knowing people are coming is useful if conflicts arise.

Diego (Zoom or Gross Hall 2nd Floor, 230K): Wednesday 2:00 - 3:00 pm

Meron (Zoom or Gross Hall 2nd Floor, 230N): Tuesday 12:15 - 1:15pm.

Class Schedule

Class will be Tuesdays and Thursdays from 1:25pm-2:40pm (EDT).

Date, Rm	Topic	Do Before Class	In-Class Exercise
Tues, Aug 26	Class Introduction Welcome to VS Code	SETUP 1: Installing Python and miniconda SETUP 2: Installing and Configuring VS Code chatGPT and You
Thurs, Aug 28	Command Line Basics	SETUP 3: Setup Augmented Command Line Command Line Basics Read and sign syllabus, submit on gradescope Register with IPUMS Register for DataCamp	Ex
Tues, Sep 2	Advanced Command Line Packages Jupyter	Advanced Command Line Python packages SETUP 4: Jupyter in VS Code OPTIONAL SETUP: R Jupyter Notebooks	Ex 1 Ex 2
Thurs, Sep 4	Python Debugger R / Python Differences Packages	What are Tracebacks? Debugging Tools Python Debugger in VS Code variables v objects Ergonomics Python v. R	Ex 1 Ex 2
Tues, Sep 9	Git	Git and Github Create an ssh key and register with ssh-agent (you don’t need to create a passphrase) If you don’t have a github account, make one. Register a passkey or at least two factor authentication. Add your ssh key to your github account Visualize Git Branches	Ex 1
Thurs, Sep 11	Numpy Basics	Under the numpy heading, read everything under Vectors EXCEPT Speed/Ease of Use Tradeoff and Vectorization Under the numpy heading, also read everything under Subsetting Vectors Numbers in Computers Writing Good Notebooks	Ex 1 Ex 2
Tues, Sep 16	Numpy Arrays	More Numpy Concepts: Views and Copies When Do I Get a View Variables, Objects, and Numpy Views and Copies Recap Matrices: Matrices — From Welcome through Review of Matrices (see navigation bar on left) ND Arrays: Arrays — From Welcome through Color Images as Arrays (see navigation bar on left)	Ex 1 (Finish other numpy)
Thurs, Sep 18	Pandas: Series	Speed / Ease of Use Tradeoff Vectorization Under Pandas, Read from From Working with Tabular Data through Subsetting and Indexing with Single Square Brackets	Link
Tues, Sep 23	Pandas: DataFrames	Object Datatypes Pandas DataFrame readings from Working with tabular data through Dataframes through Subsetting Tricks and Gotchas Categorical Datatype PyArrow Backend	Link
Thurs, Sep 25	Pandas: Indices & Missing	Indices Views and Copies in Pandas through the View/Copy Headache without CoW Reading and Writing Data: Read from Ways To Store and Read Data through ways to Store and Read Binary Data Missing Data	Link 1 Link 2
Tues, Sep 30	Pandas: Cleaning Tracebacks	Data Cleaning from Identifying Problems through Cleaning Data Types f-strings Datacamp Class “Regular Expressions for Pattern Matching,” Chapter 3 (Introduction to regular expressions)	Link
Thurs, Oct 2	Plotting	Plotting Principles Matplotlib Basics Matplotlib Vocabulary Explicit v Implict Saving Plots	Link
Tues, Oct 7	Plotting	Plotting in Python through seaborn and Grammer of Graphics seaborn recipes Customizing matplotlib Making Pretty Plots 1 Making Pretty Plots 2 Pandas Plotting
Thurs, Oct 9	Merging	Combining, Grouping and Querying Concatenating Merging/Joins Merging Part 2 Merging Example Merge Validation
Tues, Oct 14 (Break)	FALL BREAK
Thurs, Oct 16 (Break)	FALL BREAK (Not technically, but take the week)
Tues, Oct 21	Statistics with statsmodels	Read all readings under the Statistical Modelling heading, starting with Inference and Prediction through Beyond the Basic Model
Thurs, Oct 23	Big Data: What is it, how do I work with it?	What is Big Data? Strategies for Big Data Download the dataset linked at the top of the linked exercise before class. Parquet Format
Tues, Oct 28	Defensive Programming Workflow Backwards Design Getting Help Online Git and Github	Workflow Management Backwards Design Getting Help Code Reviews
Thurs, Oct 30	Defensive Programming Groupby / Split-Apply-Combine	Defensive Programming: from first reading through Iceberg Principle Groupby
Tues, Nov 4	Pandas: Reshaping Pandas: Categoricals	Wes McKinney’s Python for Data Analysis, Third Edition, Chapter 8, Section 3. Available online through duke library free Tidy Data Format: the goal of reshaping Pandas reshaping (with pics!)
Thurs, Nov 6	Speed and Performance in Python	Understanding Performance Solving Performance Problems
Tues, Nov 11	Machine Learning with sckikit-learn	JVP Chapter 5 up to but not including Hyperparameters and Model Validation Section
Thurs, Nov 13	Defining Your Own Estimators	Writing to Stakeholders Review linear regression as matrix manipulations. Here’s a nice review. Review how to define classes Broadcasting Gotchyas
Tues, Nov 18	Parallelism Distributed Computing	conda Environments Parallel Computing Distributed Computing with dask (Note reading includes a 45 minute video to watch)
Thurs, Nov 20	LAST CLASS	ROUGH DRAFT OF OPIOID PROJECT DUE, extendable to Morning of Mon Nov 26 but feedback may be later
Wed Dec 12		Final Project Report Due