Solving Hypothesis Testing Problems.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import ttest_ind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis Testing\n",
    "\n",
    "Definitions:\n",
    "* A _quarter_ is a specific three month period, Q1 is January through March, Q2 is April through June, Q3 is July through September, Q4 is October through December.\n",
    "* A _recession_ is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.\n",
    "* A _recession bottom_ is the quarter within a recession which had the lowest GDP.\n",
    "* A _university town_ is a city which has a high percentage of university students compared to the total population of the city.\n",
    "\n",
    "**Hypothesis**: University towns have their mean housing prices less effected by recessions. Run a t-test to compare the ratio of the mean price of houses in university towns the quarter before the recession starts compared to the recession bottom. (`price_ratio=quarter_before_recession/recession_bottom`)\n",
    "\n",
    "The following data files are available:\n",
    "* From the [Zillow research data site](http://www.zillow.com/research/data/) there is housing data for the United States. In particular the datafile for [all homes at a city level](http://files.zillowstatic.com/research/public/City/City_Zhvi_AllHomes.csv), ```City_Zhvi_AllHomes.csv```, has median home sale prices at a fine grained level.\n",
    "* From the Wikipedia page on college towns is a list of [university towns in the United States](https://en.wikipedia.org/wiki/List_of_college_towns#College_towns_in_the_United_States) which has been copy and pasted into the file ```university_towns.txt```.\n",
    "* From Bureau of Economic Analysis, US Department of Commerce, the [GDP over time](http://www.bea.gov/national/index.htm#gdp) of the United States in current dollars (use the chained value in 2009 dollars), in quarterly intervals, in the file ```gdplev.xls```. For this assignment, only look at GDP data from the first quarter of 2000 onward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this dictionary to map state names to two letter acronyms\n",
    "states = {'OH': 'Ohio', 'KY': 'Kentucky', 'AS': 'American Samoa', 'NV': 'Nevada', 'WY': 'Wyoming', 'NA': 'National', 'AL': 'Alabama', 'MD': 'Maryland', 'AK': 'Alaska', 'UT': 'Utah', 'OR': 'Oregon', 'MT': 'Montana', 'IL': 'Illinois', 'TN': 'Tennessee', 'DC': 'District of Columbia', 'VT': 'Vermont', 'ID': 'Idaho', 'AR': 'Arkansas', 'ME': 'Maine', 'WA': 'Washington', 'HI': 'Hawaii', 'WI': 'Wisconsin', 'MI': 'Michigan', 'IN': 'Indiana', 'NJ': 'New Jersey', 'AZ': 'Arizona', 'GU': 'Guam', 'MS': 'Mississippi', 'PR': 'Puerto Rico', 'NC': 'North Carolina', 'TX': 'Texas', 'SD': 'South Dakota', 'MP': 'Northern Mariana Islands', 'IA': 'Iowa', 'MO': 'Missouri', 'CT': 'Connecticut', 'WV': 'West Virginia', 'SC': 'South Carolina', 'LA': 'Louisiana', 'KS': 'Kansas', 'NY': 'New York', 'NE': 'Nebraska', 'OK': 'Oklahoma', 'FL': 'Florida', 'CA': 'California', 'CO': 'Colorado', 'PA': 'Pennsylvania', 'DE': 'Delaware', 'NM': 'New Mexico', 'RI': 'Rhode Island', 'MN': 'Minnesota', 'VI': 'Virgin Islands', 'NH': 'New Hampshire', 'MA': 'Massachusetts', 'GA': 'Georgia', 'ND': 'North Dakota', 'VA': 'Virginia'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "umich_part_id": "021",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def get_list_of_university_towns():\n",
    "    \n",
    "    '''Returns a DataFrame of towns and the states they are in from the \n",
    "    university_towns.txt list. The format of the DataFrame should be:\n",
    "    DataFrame( [ [\"Michigan\", \"Ann Arbor\"], [\"Michigan\", \"Yipsilanti\"] ], \n",
    "    columns=[\"State\", \"RegionName\"]  )\n",
    "    \n",
    "    The following cleaning needs to be done:\n",
    "\n",
    "    1. For \"State\", removing characters from \"[\" to the end.\n",
    "    2. For \"RegionName\", when applicable, removing every character from \" (\" to the end.\n",
    "    3. Depending on how you read the data, you may need to remove newline character '\\n'. '''\n",
    "    \n",
    "    with open('university_towns.txt') as file:\n",
    "        data = []\n",
    "        for line in file:\n",
    "            data.append(line[:-1])\n",
    "    state_town = []\n",
    "    for line in data:\n",
    "        if line[-6:] == '[edit]':\n",
    "            state = line[:-6]\n",
    "        elif '(' in line:\n",
    "            town = line[:line.index('(')-1]\n",
    "            state_town.append([state,town])        \n",
    "        else:\n",
    "            town = line\n",
    "            state_town.append([state,town])\n",
    "    state_college_df = pd.DataFrame(state_town,columns = ['State','RegionName'])\n",
    "    \n",
    "    return state_college_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>RegionName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>Auburn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>Florence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>Jacksonville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>Livingston</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>Alabama</td>\n",
       "      <td>Montevallo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>512</td>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>River Falls</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>513</td>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>Stevens Point</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>514</td>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>Waukesha</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>515</td>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>Whitewater</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>516</td>\n",
       "      <td>Wyoming</td>\n",
       "      <td>Laramie</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>517 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         State     RegionName\n",
       "0      Alabama         Auburn\n",
       "1      Alabama       Florence\n",
       "2      Alabama   Jacksonville\n",
       "3      Alabama     Livingston\n",
       "4      Alabama     Montevallo\n",
       "..         ...            ...\n",
       "512  Wisconsin    River Falls\n",
       "513  Wisconsin  Stevens Point\n",
       "514  Wisconsin       Waukesha\n",
       "515  Wisconsin     Whitewater\n",
       "516    Wyoming        Laramie\n",
       "\n",
       "[517 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_list_of_university_towns()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "umich_part_id": "022",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def get_recession_start():\n",
    "    \n",
    "    '''Returns the year and quarter of the recession start time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    \n",
    "    gdp = pd.read_excel('gdplev.xls', skiprows = 7)\n",
    "    gdp = gdp[['Unnamed: 4', 'Unnamed: 5']]\n",
    "    gdp = gdp.loc[212:]\n",
    "    gdp.columns = ['Quarter','GDP']\n",
    "    gdp['GDP'] = pd.to_numeric(gdp['GDP'])\n",
    "    quarters = []\n",
    "    for i in range(len(gdp) - 2):\n",
    "        if (gdp.iloc[i][1] > gdp.iloc[i+1][1]) & (gdp.iloc[i+1][1] > gdp.iloc[i+2][1]):\n",
    "            quarters.append(gdp.iloc[i][0])\n",
    "            \n",
    "    return quarters[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2008q3'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recession_start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "gdp = pd.read_excel('gdplev.xls', skiprows = 7)\n",
    "gdp = gdp[['Unnamed: 4', 'Unnamed: 5']]\n",
    "gdp = gdp.loc[212:]\n",
    "gdp.columns = ['Quarter','GDP']\n",
    "gdp['GDP'] = pd.to_numeric(gdp['GDP'])\n",
    "quarters = []\n",
    "for i in range(len(gdp) - 2):\n",
    "    if (gdp.iloc[i][1] > gdp.iloc[i+1][1]) & (gdp.iloc[i+1][1] > gdp.iloc[i+2][1]):\n",
    "        quarters.append(gdp.iloc[i][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "umich_part_id": "023",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def get_recession_end():\n",
    "    '''Returns the year and quarter of the recession end time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    \n",
    "    gdp2 = gdp.loc[245:]\n",
    "    recession_end = []\n",
    "    for i in range(len(gdp2)- 2):\n",
    "        if (gdp2.iloc[i+2][1] > gdp2.iloc[i+1][1])  & (gdp2.iloc[i+1][1] > gdp2.iloc[i][1]):\n",
    "            recession_end.append(gdp2.iloc[i+2][0])\n",
    "            \n",
    "    return recession_end[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2009q4'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recession_end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "recession_period = gdp.loc[245:]\n",
    "recession_min = recession_period[recession_period['GDP'] == recession_period['GDP'].min()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "umich_part_id": "024",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def get_recession_bottom():\n",
    "    \n",
    "    '''Returns the year and quarter of the recession bottom time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    \n",
    "    recession_period = gdp.loc[245:]\n",
    "    recession_min = recession_period[recession_period['GDP'] == recession_period['GDP'].min()]\n",
    "    \n",
    "    return recession_min.values[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2009q2'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recession_bottom()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "umich_part_id": "025",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def convert_housing_data_to_quarters():\n",
    "    \n",
    "    '''Converts the housing data to quarters and returns it as mean \n",
    "    values in a dataframe. This dataframe should be a dataframe with\n",
    "    columns for 2000q1 through 2016q3, and should have a multi-index\n",
    "    in the shape of [\"State\",\"RegionName\"].\n",
    "    \n",
    "    Note: Quarters are defined in the assignment description, they are\n",
    "    not arbitrary three month periods.\n",
    "    \n",
    "    The resulting dataframe should have 67 columns, and 10,730 rows.\n",
    "    '''\n",
    "    housingdata_df = pd.read_csv('City_Zhvi_AllHomes.csv')\n",
    "    \n",
    "    #convert two-letter-state to full name of state\n",
    "    housingdata_df['State'] = housingdata_df['State'].map(states)\n",
    "    \n",
    "    #set index to state, regionname\n",
    "    housingdata_df.set_index([\"State\",\"RegionName\"], inplace=True)\n",
    "    \n",
    "    #filter columns by year, only want 2000 to 2016\n",
    "    housingdata_df = housingdata_df.filter(regex='^20', axis=1)\n",
    "    \n",
    "    #group select columns by quarter, calculates average per quarter\n",
    "    housingdata_df = housingdata_df.groupby(pd.PeriodIndex(housingdata_df.columns, freq='Q'), axis=1).mean()\n",
    "    \n",
    "    return housingdata_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>2000Q1</th>\n",
       "      <th>2000Q2</th>\n",
       "      <th>2000Q3</th>\n",
       "      <th>2000Q4</th>\n",
       "      <th>2001Q1</th>\n",
       "      <th>2001Q2</th>\n",
       "      <th>2001Q3</th>\n",
       "      <th>2001Q4</th>\n",
       "      <th>2002Q1</th>\n",
       "      <th>2002Q2</th>\n",
       "      <th>...</th>\n",
       "      <th>2014Q2</th>\n",
       "      <th>2014Q3</th>\n",
       "      <th>2014Q4</th>\n",
       "      <th>2015Q1</th>\n",
       "      <th>2015Q2</th>\n",
       "      <th>2015Q3</th>\n",
       "      <th>2015Q4</th>\n",
       "      <th>2016Q1</th>\n",
       "      <th>2016Q2</th>\n",
       "      <th>2016Q3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>State</th>\n",
       "      <th>RegionName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>New York</td>\n",
       "      <td>New York</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>515466.666667</td>\n",
       "      <td>522800.000000</td>\n",
       "      <td>528066.666667</td>\n",
       "      <td>532266.666667</td>\n",
       "      <td>540800.000000</td>\n",
       "      <td>557200.000000</td>\n",
       "      <td>572833.333333</td>\n",
       "      <td>582866.666667</td>\n",
       "      <td>591633.333333</td>\n",
       "      <td>587200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>California</td>\n",
       "      <td>Los Angeles</td>\n",
       "      <td>207066.666667</td>\n",
       "      <td>214466.666667</td>\n",
       "      <td>220966.666667</td>\n",
       "      <td>226166.666667</td>\n",
       "      <td>233000.000000</td>\n",
       "      <td>239100.000000</td>\n",
       "      <td>245066.666667</td>\n",
       "      <td>253033.333333</td>\n",
       "      <td>261966.666667</td>\n",
       "      <td>272700.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>498033.333333</td>\n",
       "      <td>509066.666667</td>\n",
       "      <td>518866.666667</td>\n",
       "      <td>528800.000000</td>\n",
       "      <td>538166.666667</td>\n",
       "      <td>547266.666667</td>\n",
       "      <td>557733.333333</td>\n",
       "      <td>566033.333333</td>\n",
       "      <td>577466.666667</td>\n",
       "      <td>584050.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Illinois</td>\n",
       "      <td>Chicago</td>\n",
       "      <td>138400.000000</td>\n",
       "      <td>143633.333333</td>\n",
       "      <td>147866.666667</td>\n",
       "      <td>152133.333333</td>\n",
       "      <td>156933.333333</td>\n",
       "      <td>161800.000000</td>\n",
       "      <td>166400.000000</td>\n",
       "      <td>170433.333333</td>\n",
       "      <td>175500.000000</td>\n",
       "      <td>177566.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>192633.333333</td>\n",
       "      <td>195766.666667</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>201066.666667</td>\n",
       "      <td>206033.333333</td>\n",
       "      <td>208300.000000</td>\n",
       "      <td>207900.000000</td>\n",
       "      <td>206066.666667</td>\n",
       "      <td>208200.000000</td>\n",
       "      <td>212000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>Philadelphia</td>\n",
       "      <td>53000.000000</td>\n",
       "      <td>53633.333333</td>\n",
       "      <td>54133.333333</td>\n",
       "      <td>54700.000000</td>\n",
       "      <td>55333.333333</td>\n",
       "      <td>55533.333333</td>\n",
       "      <td>56266.666667</td>\n",
       "      <td>57533.333333</td>\n",
       "      <td>59133.333333</td>\n",
       "      <td>60733.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>113733.333333</td>\n",
       "      <td>115300.000000</td>\n",
       "      <td>115666.666667</td>\n",
       "      <td>116200.000000</td>\n",
       "      <td>117966.666667</td>\n",
       "      <td>121233.333333</td>\n",
       "      <td>122200.000000</td>\n",
       "      <td>123433.333333</td>\n",
       "      <td>126933.333333</td>\n",
       "      <td>128700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Arizona</td>\n",
       "      <td>Phoenix</td>\n",
       "      <td>111833.333333</td>\n",
       "      <td>114366.666667</td>\n",
       "      <td>116000.000000</td>\n",
       "      <td>117400.000000</td>\n",
       "      <td>119600.000000</td>\n",
       "      <td>121566.666667</td>\n",
       "      <td>122700.000000</td>\n",
       "      <td>124300.000000</td>\n",
       "      <td>126533.333333</td>\n",
       "      <td>128366.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>164266.666667</td>\n",
       "      <td>165366.666667</td>\n",
       "      <td>168500.000000</td>\n",
       "      <td>171533.333333</td>\n",
       "      <td>174166.666667</td>\n",
       "      <td>179066.666667</td>\n",
       "      <td>183833.333333</td>\n",
       "      <td>187900.000000</td>\n",
       "      <td>191433.333333</td>\n",
       "      <td>195200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>Town of Wrightstown</td>\n",
       "      <td>101766.666667</td>\n",
       "      <td>105400.000000</td>\n",
       "      <td>111366.666667</td>\n",
       "      <td>114866.666667</td>\n",
       "      <td>125966.666667</td>\n",
       "      <td>129900.000000</td>\n",
       "      <td>129900.000000</td>\n",
       "      <td>129433.333333</td>\n",
       "      <td>131900.000000</td>\n",
       "      <td>134200.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>144866.666667</td>\n",
       "      <td>146866.666667</td>\n",
       "      <td>149233.333333</td>\n",
       "      <td>148666.666667</td>\n",
       "      <td>149333.333333</td>\n",
       "      <td>149866.666667</td>\n",
       "      <td>149933.333333</td>\n",
       "      <td>149833.333333</td>\n",
       "      <td>151266.666667</td>\n",
       "      <td>155000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>New York</td>\n",
       "      <td>Urbana</td>\n",
       "      <td>79200.000000</td>\n",
       "      <td>81666.666667</td>\n",
       "      <td>91700.000000</td>\n",
       "      <td>98366.666667</td>\n",
       "      <td>94866.666667</td>\n",
       "      <td>98533.333333</td>\n",
       "      <td>102966.666667</td>\n",
       "      <td>98033.333333</td>\n",
       "      <td>93966.666667</td>\n",
       "      <td>94600.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>132133.333333</td>\n",
       "      <td>137033.333333</td>\n",
       "      <td>140066.666667</td>\n",
       "      <td>141700.000000</td>\n",
       "      <td>137866.666667</td>\n",
       "      <td>136466.666667</td>\n",
       "      <td>136166.666667</td>\n",
       "      <td>138966.666667</td>\n",
       "      <td>144200.000000</td>\n",
       "      <td>143000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>New Denmark</td>\n",
       "      <td>114566.666667</td>\n",
       "      <td>119266.666667</td>\n",
       "      <td>126066.666667</td>\n",
       "      <td>131966.666667</td>\n",
       "      <td>143800.000000</td>\n",
       "      <td>146966.666667</td>\n",
       "      <td>148366.666667</td>\n",
       "      <td>149166.666667</td>\n",
       "      <td>153133.333333</td>\n",
       "      <td>156733.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>174566.666667</td>\n",
       "      <td>181166.666667</td>\n",
       "      <td>186166.666667</td>\n",
       "      <td>187600.000000</td>\n",
       "      <td>188666.666667</td>\n",
       "      <td>188433.333333</td>\n",
       "      <td>188933.333333</td>\n",
       "      <td>191066.666667</td>\n",
       "      <td>192833.333333</td>\n",
       "      <td>197600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>California</td>\n",
       "      <td>Angels</td>\n",
       "      <td>151000.000000</td>\n",
       "      <td>155900.000000</td>\n",
       "      <td>158100.000000</td>\n",
       "      <td>167466.666667</td>\n",
       "      <td>176833.333333</td>\n",
       "      <td>183766.666667</td>\n",
       "      <td>190233.333333</td>\n",
       "      <td>184566.666667</td>\n",
       "      <td>184033.333333</td>\n",
       "      <td>186133.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>244466.666667</td>\n",
       "      <td>254066.666667</td>\n",
       "      <td>259933.333333</td>\n",
       "      <td>260100.000000</td>\n",
       "      <td>250633.333333</td>\n",
       "      <td>263500.000000</td>\n",
       "      <td>279500.000000</td>\n",
       "      <td>276533.333333</td>\n",
       "      <td>271600.000000</td>\n",
       "      <td>269950.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>Holland</td>\n",
       "      <td>151033.333333</td>\n",
       "      <td>150500.000000</td>\n",
       "      <td>153233.333333</td>\n",
       "      <td>155833.333333</td>\n",
       "      <td>161866.666667</td>\n",
       "      <td>165733.333333</td>\n",
       "      <td>168033.333333</td>\n",
       "      <td>167400.000000</td>\n",
       "      <td>165766.666667</td>\n",
       "      <td>161966.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>201566.666667</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>206000.000000</td>\n",
       "      <td>207600.000000</td>\n",
       "      <td>212866.666667</td>\n",
       "      <td>217833.333333</td>\n",
       "      <td>221966.666667</td>\n",
       "      <td>228033.333333</td>\n",
       "      <td>234950.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10730 rows × 67 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         2000Q1         2000Q2         2000Q3  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York                       NaN            NaN            NaN   \n",
       "California   Los Angeles          207066.666667  214466.666667  220966.666667   \n",
       "Illinois     Chicago              138400.000000  143633.333333  147866.666667   \n",
       "Pennsylvania Philadelphia          53000.000000   53633.333333   54133.333333   \n",
       "Arizona      Phoenix              111833.333333  114366.666667  116000.000000   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  101766.666667  105400.000000  111366.666667   \n",
       "New York     Urbana                79200.000000   81666.666667   91700.000000   \n",
       "Wisconsin    New Denmark          114566.666667  119266.666667  126066.666667   \n",
       "California   Angels               151000.000000  155900.000000  158100.000000   \n",
       "Wisconsin    Holland              151033.333333  150500.000000  153233.333333   \n",
       "\n",
       "                                         2000Q4         2001Q1         2001Q2  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York                       NaN            NaN            NaN   \n",
       "California   Los Angeles          226166.666667  233000.000000  239100.000000   \n",
       "Illinois     Chicago              152133.333333  156933.333333  161800.000000   \n",
       "Pennsylvania Philadelphia          54700.000000   55333.333333   55533.333333   \n",
       "Arizona      Phoenix              117400.000000  119600.000000  121566.666667   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  114866.666667  125966.666667  129900.000000   \n",
       "New York     Urbana                98366.666667   94866.666667   98533.333333   \n",
       "Wisconsin    New Denmark          131966.666667  143800.000000  146966.666667   \n",
       "California   Angels               167466.666667  176833.333333  183766.666667   \n",
       "Wisconsin    Holland              155833.333333  161866.666667  165733.333333   \n",
       "\n",
       "                                         2001Q3         2001Q4         2002Q1  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York                       NaN            NaN            NaN   \n",
       "California   Los Angeles          245066.666667  253033.333333  261966.666667   \n",
       "Illinois     Chicago              166400.000000  170433.333333  175500.000000   \n",
       "Pennsylvania Philadelphia          56266.666667   57533.333333   59133.333333   \n",
       "Arizona      Phoenix              122700.000000  124300.000000  126533.333333   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  129900.000000  129433.333333  131900.000000   \n",
       "New York     Urbana               102966.666667   98033.333333   93966.666667   \n",
       "Wisconsin    New Denmark          148366.666667  149166.666667  153133.333333   \n",
       "California   Angels               190233.333333  184566.666667  184033.333333   \n",
       "Wisconsin    Holland              168033.333333  167400.000000  165766.666667   \n",
       "\n",
       "                                         2002Q2  ...         2014Q2  \\\n",
       "State        RegionName                          ...                  \n",
       "New York     New York                       NaN  ...  515466.666667   \n",
       "California   Los Angeles          272700.000000  ...  498033.333333   \n",
       "Illinois     Chicago              177566.666667  ...  192633.333333   \n",
       "Pennsylvania Philadelphia          60733.333333  ...  113733.333333   \n",
       "Arizona      Phoenix              128366.666667  ...  164266.666667   \n",
       "...                                         ...  ...            ...   \n",
       "Wisconsin    Town of Wrightstown  134200.000000  ...  144866.666667   \n",
       "New York     Urbana                94600.000000  ...  132133.333333   \n",
       "Wisconsin    New Denmark          156733.333333  ...  174566.666667   \n",
       "California   Angels               186133.333333  ...  244466.666667   \n",
       "Wisconsin    Holland              161966.666667  ...  201266.666667   \n",
       "\n",
       "                                         2014Q3         2014Q4         2015Q1  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York             522800.000000  528066.666667  532266.666667   \n",
       "California   Los Angeles          509066.666667  518866.666667  528800.000000   \n",
       "Illinois     Chicago              195766.666667  201266.666667  201066.666667   \n",
       "Pennsylvania Philadelphia         115300.000000  115666.666667  116200.000000   \n",
       "Arizona      Phoenix              165366.666667  168500.000000  171533.333333   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  146866.666667  149233.333333  148666.666667   \n",
       "New York     Urbana               137033.333333  140066.666667  141700.000000   \n",
       "Wisconsin    New Denmark          181166.666667  186166.666667  187600.000000   \n",
       "California   Angels               254066.666667  259933.333333  260100.000000   \n",
       "Wisconsin    Holland              201566.666667  201266.666667  206000.000000   \n",
       "\n",
       "                                         2015Q2         2015Q3         2015Q4  \\\n",
       "State        RegionName                                                         \n",
       "New York     New York             540800.000000  557200.000000  572833.333333   \n",
       "California   Los Angeles          538166.666667  547266.666667  557733.333333   \n",
       "Illinois     Chicago              206033.333333  208300.000000  207900.000000   \n",
       "Pennsylvania Philadelphia         117966.666667  121233.333333  122200.000000   \n",
       "Arizona      Phoenix              174166.666667  179066.666667  183833.333333   \n",
       "...                                         ...            ...            ...   \n",
       "Wisconsin    Town of Wrightstown  149333.333333  149866.666667  149933.333333   \n",
       "New York     Urbana               137866.666667  136466.666667  136166.666667   \n",
       "Wisconsin    New Denmark          188666.666667  188433.333333  188933.333333   \n",
       "California   Angels               250633.333333  263500.000000  279500.000000   \n",
       "Wisconsin    Holland              207600.000000  212866.666667  217833.333333   \n",
       "\n",
       "                                         2016Q1         2016Q2    2016Q3  \n",
       "State        RegionName                                                   \n",
       "New York     New York             582866.666667  591633.333333  587200.0  \n",
       "California   Los Angeles          566033.333333  577466.666667  584050.0  \n",
       "Illinois     Chicago              206066.666667  208200.000000  212000.0  \n",
       "Pennsylvania Philadelphia         123433.333333  126933.333333  128700.0  \n",
       "Arizona      Phoenix              187900.000000  191433.333333  195200.0  \n",
       "...                                         ...            ...       ...  \n",
       "Wisconsin    Town of Wrightstown  149833.333333  151266.666667  155000.0  \n",
       "New York     Urbana               138966.666667  144200.000000  143000.0  \n",
       "Wisconsin    New Denmark          191066.666667  192833.333333  197600.0  \n",
       "California   Angels               276533.333333  271600.000000  269950.0  \n",
       "Wisconsin    Holland              221966.666667  228033.333333  234950.0  \n",
       "\n",
       "[10730 rows x 67 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "convert_housing_data_to_quarters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "umich_part_id": "026",
    "umich_partlist_id": "004"
   },
   "outputs": [],
   "source": [
    "def run_ttest():\n",
    "    \n",
    "    '''First creates new data showing the decline or growth of housing prices\n",
    "    between the recession start and the recession bottom. Then runs a ttest\n",
    "    comparing the university town values to the non-university towns values, \n",
    "    return whether the alternative hypothesis (that the two groups are the same)\n",
    "    is true or not as well as the p-value of the confidence. \n",
    "    \n",
    "    Return the tuple (different, p, better) where different=True if the t-test is\n",
    "    True at a p<0.01 (we reject the null hypothesis), or different=False if \n",
    "    otherwise (we cannot reject the null hypothesis). The variable p should\n",
    "    be equal to the exact p value returned from scipy.stats.ttest_ind(). The\n",
    "    value for better should be either \"university town\" or \"non-university town\"\n",
    "    depending on which has a lower mean price ratio (which is equivilent to a\n",
    "    reduced market loss).'''\n",
    "    \n",
    "    housingdata_df = convert_housing_data_to_quarters().dropna()\n",
    "    hdf = housingdata_df.copy()\n",
    "    \n",
    "    recession_start = get_recession_start()\n",
    "    recession_bottom = get_recession_bottom()\n",
    "    university_towns = get_list_of_university_towns()\n",
    "    housingdata_df = convert_housing_data_to_quarters().dropna()\n",
    "    hdf = housingdata_df.copy()\n",
    "    \n",
    "    #create ratio of housing prices from recession start to recession bottom\n",
    "    ratio = pd.DataFrame({'ratio': hdf[recession_start].div(hdf[recession_bottom])})\n",
    "    hdf.columns = hdf.columns.to_series().astype(str)\n",
    "    hdf = pd.concat([hdf, ratio], axis=1)\n",
    "    hdf = pd.DataFrame(hdf)\n",
    "    hdf.reset_index(['State','RegionName'], inplace = True)\n",
    "    \n",
    "    #spliced dataframe to unitown and non-unitown, calculated ratios for each, dropped NaN values\n",
    "    unitown_priceratio = hdf.loc[list(university_towns.index)]['ratio'].dropna()\n",
    "    nonunitown_priceratio_index = set(hdf.index) - set(unitown_priceratio)\n",
    "    nonunitown_priceratio = hdf.loc[list(nonunitown_priceratio_index),:]['ratio'].dropna()\n",
    "    \n",
    "    #run t-test comparing university town values to non-university town values\n",
    "    tstat, p = tuple(ttest_ind(unitown_priceratio, nonunitown_priceratio))\n",
    "    \n",
    "    #return tuple where different = True or False, return p-value, and whether university-town is better or not\n",
    "    different = p < 0.05\n",
    "    result = tstat < 0\n",
    "    better = [\"university town\", \"non-university town\"]\n",
    "    \n",
    "    return (different, p, better[result])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:39: DeprecationWarning: In future, it will be an error for 'np.bool_' scalars to be interpreted as an index\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(True, 1.200820031703725e-36, 'university town')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_ttest()"
   ]
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-data-analysis",
   "graded_item_id": "Il9Fx",
   "launcher_item_id": "TeDW0",
   "part_id": "WGlun"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "umich": {
   "id": "Assignment 4",
   "version": "1.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}