{ "cells": [ { "cell_type": "code", "execution_count": 121, "metadata": { "id": "7dZbT3ijl0Qh" }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import geopandas as gpd\n", "from shapely import wkt\n", "from shapely.geometry import Point\n", "\n", "%matplotlib inline\n", "import matplotlib.pyplot as plt\n", "plt.style.use('fivethirtyeight')\n", "import warnings\n", "warnings.simplefilter('ignore', FutureWarning)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Project for DATA 118 - Winter 2025\n", "\n", "Welcome to your project for DATA 118. This is not a group project. You may discuss the project with other students but you should code and write the report independently. You should acknowledge any help in writing (including any use of generative AI, but AI should not be used for general coding in this class). The score will be based on:\n", "- Clarity and soundness of the arguments and conclusions;\n", "- Use of data to back up arguments and analysis quality; \n", "- Insightfulness of the results;\n", "- Quality of the data vizualizations, summaries used, and overall presentation. Make sure you use headings, captions for figures and tables etc. When you interpret a graph or data from a table, you should clearly specify which figure/table you refer to.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "### The Data" ] }, { "cell_type": "markdown", "metadata": { "id": "RdiytFTml0Ql" }, "source": [ "Option 1: You are welcome to find your own data set for this project, but keep the goals in mind. Here are some suggested sources.\n", "\n", "Suggestions for datasets \n", "- Github Collection of publicly available data: https://github.com/awesomedata/awesome-public-datasets\n", "- Google Dataset search: https://datasetsearch.research.google.com/\n", "- CDC: https://data.cdc.gov/browse\n", "- 500 cities: https://www.cdc.gov/500cities/index.htm\n", "- UN: http://data.un.org/\n", "- Kaggle: https://www.kaggle.com/datasets\n", "- FEC: https://www.fec.gov/\n", "- FiveThirtyEight: https://github.com/fivethirtyeight/data\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Option 2:\n", "\n", "Use data from the \"National Wellbeing Survey, United States, 2021\",\n", "\n", "*\"The National Wellbeing Survey (NWS) is a population-based survey on the wellbeing of adults aged 18 to 64 in the United States. Specific survey domains include psychological well-being, social relationships and support, physical health, mental health, health behaviors, COVID-19 experiences and impacts, socioeconomic measures, political orientation, and demographic measures.\"*\n", "\n", "The data can be downloaded from the link below\n", "https://www.icpsr.umich.edu/web/ICPSR/studies/38879/\n", "\n", "\n", " First, Click on 'Download' and then 'Delimited'.\n", "\n", " Next, you will need to make an account and agree to the corresponding terms and conditions to have access to the data. Note you will get a folder called \"ICPSR_38879\". This folder contains the codebook that describes the study and the variables in detail. It has information on the survey that was administered. This folder is also how you will get the data itself.\n", "\n", "A few notes, hints, that may be helpful as you are working with this data.\n", "1. Take a look at the codebook. It has a lot of information about the survey itself and the variable names. We have put together a spreadsheet of selected variables to make it easier to see the variables present.\n", "2. The variable outcomes are encoded as numeric values. To see the corresponding outputs, you will need to look at the codebook, including how missing data was handled in the survey.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Tl4X-SRBl0Qn" }, "source": [ "\n", "### The Report \n", "\n", "You must submit four files broken into 2 parts:\n", "\n", "Part 1: **Due Sunday February 16 at 11:59pm**\n", "\n", "1. The file you plan to use for your project. \n", "\n", "2. The introduction of the report. \n", "\n", " A. **Introduce the dataset**. Describe the data. What is the being studied? Where does it come from? Why was it collected (what are the researchers interested in studying)?Was it an experiment? A retrospective observational study? A prospective observational study? How many variables are there in the full dataset? List a few. How many observations (i.e. rows)? Describe the sampling process. Using what you have learned about data collection is this a biased or unbiased sample? Why?\n", "\n", "\n", "Part 2: **Due Saturday March 8 at 11:59pm**\n", "\n", "1. The Jupyter Notebook that contains all the code you use for the analysis broken into the parts listed below as necessary. Make sure to RUN ALL CELLS IN YOUR NOTEBOOK before submitting.\n", "\n", "2. A report of your findings. **This report should be at most 4 pages (12 pt font 1.5 spaced) long including references.** Use data visualization and data summaries to justify your conclusions. Note that the page limitation means you will not show all analyses you will make - select carefully what you think is most relevant.\n", "\n", " The report should address the following points broken into parts:\n", "\n", " A. Same as above, but edited based on feedback.\n", "\n", " B. **Characteristics of sample**. Describe the unit (people, airbnbs, specific animals, etc) sampled in the dataset. Summarize the distributions or ranges of 3 or more of the characteristics (variables). You should include a graph or table for each distribution. You should create at least one graph and at least one table (DataFrame).\n", "\n", " C. **Relationships between variables.** Now, shift focus from distributions of single variables to relationships between variables. Include two or more graphs or tables here that highlight interesting relationships between variables.\n", "\n", " During your exploration you should create at least one new column in the DataFrame based on existing variables in the dataset. To do this, you need to write a function (or more than one if you like) and apply it to one or more columns of the dataset. Make sure to clearly mark in your code notebook where your function is located.\n", "\n", " D. **Provide context and set up a Hypothesis Test.** To the best of your knowledge, what do you think the relationships you discovered imply?\n", "\n", " Based on supporting evidence from your investigation, set up a hypothesis test centering around a question of interest. Formulate the hypothesis test by **clearly stating the null and alternate hypothesis, and the corresponding test statistic.** You are just setting up the hypothesis test here, **you do not need to carry out the hypothesis test**.\n", "\n", " E. **Conclusion/Self-Assessment.** What did you learn from exploring this dataset? If you were to continue this project, what is one question you would want to continue to investigate?\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Qs8jgh62l0Qo" }, "source": [ "Your submission is to be uploaded to Canvas and Gradescope. **If you do not submit all four files you will automatically recieve a deduction of 20 points from your submission.**\n", "There are some minimum requirements for your submission:\n", "\n", "1. Upload your data to Canvas\n", "2. Upload Part A: Introdcution to the data set as a PDF to Canvas\n", "3. Upload your final report as a PDF to Canvas\n", "4. Upload the jupyter notebook containing your analysis code to Gradescope, in .ipynb format.\n", "5. The report should be 4 pages maximum, , contain no code, and including tables and figures.\n", "6. You must have a bibliography citing the dataset. This can be on a separate page.\n", "7. If your report uses additional outside results and/or data, proper citations must be provided.\n", " " ] }, { "cell_type": "markdown", "metadata": { "id": "Josxqm3MeiSu" }, "source": [ "You may use the following code to help you load in the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 162 }, "executionInfo": { "elapsed": 415, "status": "error", "timestamp": 1739136430632, "user": { "displayName": "Kelly Smalenberger", "userId": "17455113318685349574" }, "user_tz": 360 }, "id": "eDNtjpSBeiSu", "outputId": "c5c93771-390f-42d0-8ceb-1f1306d95958" }, "outputs": [], "source": [ "crimes_no_zip = pd.read_csv(\"chicagocrime.csv\")\n", "zipcodes_df = gpd.read_file(\"chizipcodes.csv\")\n", "income = pd.read_excel(\"illinoisincome.xlsx\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "data": { "application/vnd.microsoft.datawrangler.viewer.v0+json": { "columns": [ { "name": "index", "rawType": "int64", "type": "integer" }, { "name": "ID", "rawType": "int64", "type": "integer" }, { "name": "Case Number", "rawType": "object", "type": "string" }, { "name": "Date", "rawType": "object", "type": "string" }, { "name": "Block", "rawType": "object", "type": "string" }, { "name": "IUCR", "rawType": "object", "type": "string" }, { "name": "Primary Type", "rawType": "object", "type": "string" }, { "name": "Description", "rawType": "object", "type": "string" }, { "name": "Location Description", "rawType": "object", "type": "string" }, { "name": "Arrest", "rawType": "bool", "type": "boolean" }, { "name": "Domestic", "rawType": "bool", "type": "boolean" }, { "name": "Beat", "rawType": "int64", "type": "integer" }, { "name": "District", "rawType": "int64", "type": "integer" }, { "name": "Ward", "rawType": "float64", "type": "float" }, { "name": "Community Area", "rawType": "int64", "type": "integer" }, { "name": "FBI Code", "rawType": "object", "type": "string" }, { "name": "X Coordinate", "rawType": "float64", "type": "float" }, { "name": "Y Coordinate", "rawType": "float64", "type": "float" }, { "name": "Year", "rawType": "int64", "type": "integer" }, { "name": "Updated On", "rawType": "object", "type": "string" }, { "name": "Latitude", "rawType": "float64", "type": "float" }, { "name": "Longitude", "rawType": "float64", "type": "float" }, { "name": "Location", "rawType": "object", "type": "unknown" } ], "conversionMethod": "pd.DataFrame", "ref": "53232b7b-935b-4aba-8646-a5a1898849c7", "rows": [ [ "0", "11938228", "JD100017", "12/31/2019 11:55:00 PM", "0000X W 69TH ST", "143A", "WEAPONS VIOLATION", "UNLAWFUL POSS OF HANDGUN", "STREET", "True", "False", "731", "7", "6.0", "69", "15", "1176896.0", "1859260.0", "2019", "01/07/2020 06:57:13 PM", "41.769150218", "-87.627136786", "(41.769150218, -87.627136786)" ], [ "1", "11940078", "JD100016", "12/31/2019 11:54:00 PM", "063XX S MAY ST", "0420", "BATTERY", "AGGRAVATED:KNIFE/CUTTING INSTR", "SIDEWALK", "False", "False", "724", "7", "16.0", "68", "04B", "1169736.0", "1862855.0", "2019", "01/08/2020 03:45:22 PM", "41.779173667", "-87.653277703", "(41.779173667, -87.653277703)" ], [ "2", "11938857", "JD100599", "12/31/2019 11:50:00 PM", "004XX N Ashland ave", "0820", "THEFT", "$500 AND UNDER", "BAR OR TAVERN", "False", "False", "1215", "12", "27.0", "24", "06", null, null, "2019", "01/07/2020 06:57:13 PM", null, null, null ], [ "3", "11938240", "JD100002", "12/31/2019 11:48:00 PM", "004XX S CICERO AVE", "143A", "WEAPONS VIOLATION", "UNLAWFUL POSS OF HANDGUN", "VEHICLE NON-COMMERCIAL", "True", "False", "1533", "15", "29.0", "25", "15", "1144466.0", "1897452.0", "2019", "01/07/2020 06:57:13 PM", "41.874623951", "-87.745052647", "(41.874623951, -87.745052647)" ], [ "4", "11937967", "JC567053", "12/31/2019 11:46:00 PM", "034XX W JACKSON BLVD", "143A", "WEAPONS VIOLATION", "UNLAWFUL POSS OF HANDGUN", "STREET", "False", "False", "1133", "11", "28.0", "27", "15", "1153587.0", "1898480.0", "2019", "01/07/2020 06:57:13 PM", "41.877268465", "-87.711536692", "(41.877268465, -87.711536692)" ] ], "shape": { "columns": 22, "rows": 5 } }, "text/html": [ "
\n", " | ID | \n", "Case Number | \n", "Date | \n", "Block | \n", "IUCR | \n", "Primary Type | \n", "Description | \n", "Location Description | \n", "Arrest | \n", "Domestic | \n", "... | \n", "Ward | \n", "Community Area | \n", "FBI Code | \n", "X Coordinate | \n", "Y Coordinate | \n", "Year | \n", "Updated On | \n", "Latitude | \n", "Longitude | \n", "Location | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "11938228 | \n", "JD100017 | \n", "12/31/2019 11:55:00 PM | \n", "0000X W 69TH ST | \n", "143A | \n", "WEAPONS VIOLATION | \n", "UNLAWFUL POSS OF HANDGUN | \n", "STREET | \n", "True | \n", "False | \n", "... | \n", "6.0 | \n", "69 | \n", "15 | \n", "1176896.0 | \n", "1859260.0 | \n", "2019 | \n", "01/07/2020 06:57:13 PM | \n", "41.769150 | \n", "-87.627137 | \n", "(41.769150218, -87.627136786) | \n", "
1 | \n", "11940078 | \n", "JD100016 | \n", "12/31/2019 11:54:00 PM | \n", "063XX S MAY ST | \n", "0420 | \n", "BATTERY | \n", "AGGRAVATED:KNIFE/CUTTING INSTR | \n", "SIDEWALK | \n", "False | \n", "False | \n", "... | \n", "16.0 | \n", "68 | \n", "04B | \n", "1169736.0 | \n", "1862855.0 | \n", "2019 | \n", "01/08/2020 03:45:22 PM | \n", "41.779174 | \n", "-87.653278 | \n", "(41.779173667, -87.653277703) | \n", "
2 | \n", "11938857 | \n", "JD100599 | \n", "12/31/2019 11:50:00 PM | \n", "004XX N Ashland ave | \n", "0820 | \n", "THEFT | \n", "$500 AND UNDER | \n", "BAR OR TAVERN | \n", "False | \n", "False | \n", "... | \n", "27.0 | \n", "24 | \n", "06 | \n", "NaN | \n", "NaN | \n", "2019 | \n", "01/07/2020 06:57:13 PM | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
3 | \n", "11938240 | \n", "JD100002 | \n", "12/31/2019 11:48:00 PM | \n", "004XX S CICERO AVE | \n", "143A | \n", "WEAPONS VIOLATION | \n", "UNLAWFUL POSS OF HANDGUN | \n", "VEHICLE NON-COMMERCIAL | \n", "True | \n", "False | \n", "... | \n", "29.0 | \n", "25 | \n", "15 | \n", "1144466.0 | \n", "1897452.0 | \n", "2019 | \n", "01/07/2020 06:57:13 PM | \n", "41.874624 | \n", "-87.745053 | \n", "(41.874623951, -87.745052647) | \n", "
4 | \n", "11937967 | \n", "JC567053 | \n", "12/31/2019 11:46:00 PM | \n", "034XX W JACKSON BLVD | \n", "143A | \n", "WEAPONS VIOLATION | \n", "UNLAWFUL POSS OF HANDGUN | \n", "STREET | \n", "False | \n", "False | \n", "... | \n", "28.0 | \n", "27 | \n", "15 | \n", "1153587.0 | \n", "1898480.0 | \n", "2019 | \n", "01/07/2020 06:57:13 PM | \n", "41.877268 | \n", "-87.711537 | \n", "(41.877268465, -87.711536692) | \n", "
5 rows × 22 columns
\n", "\n", " | the_geom | \n", "OBJECTID | \n", "ZIP | \n", "SHAPE_AREA | \n", "SHAPE_LEN | \n", "geometry | \n", "
---|---|---|---|---|---|---|
0 | \n", "MULTIPOLYGON (((-87.67762151065281 41.91775780... | \n", "33 | \n", "60647 | \n", "106052287.488 | \n", "42720.0444058 | \n", "MULTIPOLYGON (((-87.67762151065281 41.91775780... | \n", "
1 | \n", "MULTIPOLYGON (((-87.72683253163021 41.92264626... | \n", "34 | \n", "60639 | \n", "127476050.762 | \n", "48103.7827213 | \n", "MULTIPOLYGON (((-87.72683253163021 41.92264626... | \n", "
2 | \n", "MULTIPOLYGON (((-87.78500237831095 41.90914785... | \n", "35 | \n", "60707 | \n", "45069038.4783 | \n", "27288.6096123 | \n", "MULTIPOLYGON (((-87.78500237831095 41.90914785... | \n", "
3 | \n", "MULTIPOLYGON (((-87.6670686895295 41.888851884... | \n", "36 | \n", "60622 | \n", "70853834.3797 | \n", "42527.9896789 | \n", "MULTIPOLYGON (((-87.6670686895295 41.888851884... | \n", "
4 | \n", "MULTIPOLYGON (((-87.70655631674127 41.89555340... | \n", "37 | \n", "60651 | \n", "99039621.2518 | \n", "47970.1401531 | \n", "MULTIPOLYGON (((-87.70655631674127 41.89555340... | \n", "
\n", " | ILLINOIS | \n", "Unnamed: 1 | \n", "Unnamed: 2 | \n", "Unnamed: 3 | \n", "Unnamed: 4 | \n", "Unnamed: 5 | \n", "Unnamed: 6 | \n", "Unnamed: 7 | \n", "Unnamed: 8 | \n", "Unnamed: 9 | \n", "... | \n", "Unnamed: 140 | \n", "Unnamed: 141 | \n", "Unnamed: 142 | \n", "Unnamed: 143 | \n", "Unnamed: 144 | \n", "Unnamed: 145 | \n", "Unnamed: 146 | \n", "Unnamed: 147 | \n", "Unnamed: 148 | \n", "Unnamed: 149 | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "Individual Income Tax Returns: \\nSelected Inco... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
1 | \n", "[Money amounts are in thousands of dollars] | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
2 | \n", "ZIP\\ncode [1] | \n", "Size of adjusted gross income | \n", "Number of returns [2] | \n", "Number of single returns | \n", "Number of joint returns | \n", "Number of head of household returns | \n", "Number of electronically filed returns | \n", "Number of computer prepared paper returns | \n", "Number with paid preparer's signature | \n", "Number of returns with direct deposit | \n", "... | \n", "Net investment income tax | \n", "NaN | \n", "Tax due at time of filing [15] | \n", "NaN | \n", "Total overpayments | \n", "NaN | \n", "Overpayments refunded [16] | \n", "NaN | \n", "Credited to next year's estimated tax | \n", "NaN | \n", "
3 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "Number of returns | \n", "Amount | \n", "Number of returns | \n", "Amount | \n", "Number of returns | \n", "Amount | \n", "Number of returns | \n", "Amount | \n", "Number of returns | \n", "Amount | \n", "
4 | \n", "NaN | \n", "NaN | \n", "-1 | \n", "-2 | \n", "-3 | \n", "-4 | \n", "-5 | \n", "-6 | \n", "-7 | \n", "-8 | \n", "... | \n", "-139 | \n", "-140 | \n", "-141 | \n", "-142 | \n", "-143 | \n", "-144 | \n", "-145 | \n", "-146 | \n", "-147 | \n", "-148 | \n", "
5 rows × 150 columns
\n", "