{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"color:purple\"> Pandas Notebook 2, ATM350 Spring 2023 </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivating Science Questions:\n",
    "1. What was the daily temperature and precipitation at Albany last year?\n",
    "2. What were the the days with the most precipitation?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivating Technical Question:\n",
    "1. How can we use Pandas to do some basic statistical analyses of our data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We'll start by repeating some of the same steps we did in the first Pandas notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file = '/spare11/atm350/common/data/climo_alb_2022.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the first five lines of this file using Python's built-in `readline` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileObj = open(file)\n",
    "nLines = 5\n",
    "for n in range(nLines):\n",
    "    line = fileObj.readline()\n",
    "    print(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(file, dtype='string')\n",
    "\n",
    "nRows = df.shape[0]\n",
    "print (\"Number of rows = %d\" % nRows )\n",
    "nCols = df.shape[1]\n",
    "print (\"Number of columns = %d\" % nCols)\n",
    "\n",
    "date = df['DATE']\n",
    "date = pd.to_datetime(date,format=\"%Y-%m-%d\")\n",
    "\n",
    "maxT = df['MAX'].astype(\"float32\")\n",
    "minT = df['MIN'].astype(\"float32\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's generate the final timeseries we made in our first Pandas notebook, with all the \"bells and whistles\" included."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib.dates import DateFormatter, AutoDateLocator,HourLocator,DayLocator,MonthLocator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the year so we don't have to edit the string labels every year!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "year = 2022"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, maxT, color='red',label = \"Max T\")\n",
    "ax.plot (date, minT, color='blue', label = \"Min T\")\n",
    "ax.set_title (\"ALB Year %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Temperature ($^\\circ$F)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in precip data. This will be more challenging due to the presence of T(races).\n",
    "Let's remind ourselves what the `Dataframe` looks like, paying particular attention to the daily precip column (**PCP**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\"> <b>Exercise:</b> define a Pandas <code>DataSeries</code> called <code>precip</code> and populate it with the requisite column from our <code>Dataframe</code>. Then print out its values.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-success\"> <b>TIP:</b> After you have tried on your own, you can uncomment the first line of the cell below and re-run to <i>load</i> the solution.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load /spare11/atm350/common/feb23/02a.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The task now is to convert these values from strings to floating point values. Our task is more complicated due to the presence of strings that are clearly not numerical ... such as \"T\" for *trace*.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### As we did in the first Pandas notebook with max temperatures greater than or equal to 90, create a subset of our `Dataframe` that consists only of those days where precip was a trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "traceDays = df[precip=='T']\n",
    "traceDays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "traceDays.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "traceDays.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\"> <b>Exercise:</b> print out the total # of days where a trace of precip was measured. <b><i>Hint</i></b>: look back at how we calculated the total # of 90 degree days in our first Pandas notebook ... we used the <code>shape</code> attribute.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load /spare11/atm350/common/feb23/02b.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting back to our task of converting precip amounts from strings to floating point numbers, one thing we could do is to create a new array and populate it via a loop, where we'd use an `if-else` logical test to check for Trace values and set the precip value to 0.00 for each day accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### There is a more efficient way to do this, though! \n",
    "See https://stackoverflow.com/questions/49154068/fixing-a-typeerror-when-using-pandas-after-replacing-a-string-with-a-floating-po?rq=1 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We use the `loc` method of Pandas to find all elements of a DataSeries with a certain value, and then change that value to something else, all in the same line of code!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this case, let's set all values of 'T' to '0.00'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line below is what we want! Before we execute it, let's break it up into pieces.\n",
    "``` \n",
    "df.loc[df['PCP'] =='T', ['PCP']] = '0.00'\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### First, create a `Series` of booleans corresponding to the specified condition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PCP'] == 'T'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Next, build on that cell by using `loc` to display all rows that correspond to the condition being True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['PCP'] == 'T']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Further build this line of code by only returning the column of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['PCP'] =='T', ['PCP']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finally, we have arrived at the full line of code! Take the column of interest, in this case precip only on those days where a trace was measured, and set its value to 0.00."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[df['PCP'] =='T', ['PCP']] = '0.00'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['PCP']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This operation actually modifies the `Dataframe` *in place* . We can prove this by printing out a row from a date that we know had a trace amount."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### But first how do we simply print a specific row from a dataframe? Since we know that Jan. 3 had a trace of precip,  try this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jan03 = df['DATE'] == '2022-01-03'\n",
    "jan03"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### That produces a series of booleans; the one matching our condition is `True`. Now we can retrieve all the values for this date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[jan03]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We see that the precip has now been set to 0.00."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Having done this check, and thus re-set the values, let's now convert this series into floating point values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precip = df['PCP'].astype(\"float32\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot each day's precip total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, precip, color='blue', marker='+',label = \"Precip\")\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Precip (in.)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What if we just want to pick a certain time range? One simple way is to just pass in a subset of our x and y to the `plot` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot out just the trace for October. Corresponds to Julian days 214-245 ... thus, indices 213-244 (why?).\n",
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date[213:244], precip[213:244], color='blue', marker='+',label = \"Precip\")\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Precip (in.)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\"> <b>Exercise:</b> print out a table of days with precip amounts of at least <b>1.00</b> inches. In a separate cell, print out the total # of such days.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load '/spare11/atm350/common/feb23/02c.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas has a function to compute the cumulative sum of a series. We'll use it to compute and graph Albany's total precip over the year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precipTotal = precip.cumsum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precipTotal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can see that the final total is in the last element of the precipTotal array. How can we explicitly print out just this value?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One of the methods available to us in a Pandas DataSeries is `values`. Let's display it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "precipTotal.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\"> <b>Exercise:</b> It's an array! So, let's print out the last element of the array. What index # can we use?</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load '/spare11/atm350/common/feb23/02d.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the timeseries of the cumulative precip for Albany over the year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, precipTotal, color='blue', marker='.',label = \"Precip\")\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Precip (in.)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas has a plethora of statistical analysis methods to apply on tabular data. An excellent summary method is `describe`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxT.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minT.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precip.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-warning\"> <b>Exercise:</b> Why is the <i>mean</i> 0.10, but the <i>median</i> 0.00? Can you write a code cell that answers this question?\n",
    "    <b>Hint</b>: determine how many days had a trace or less of precip. \n",
    "    <ol><li>First, express the condition where precip is equal to 0.00.</li>\n",
    "        <li>Then, determine the # of rows of that resulting series.</li>\n",
    "    </ol>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load /spare11/atm350/common/feb23/02e.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We'll wrap up by calculating and then plotting *rolling means* over a period of days in the year, in order to smooth out the day-to-day variations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's replot the max and min temperature trace for the entire year, day-by-day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, maxT, color='red',label = \"Max T\")\n",
    "ax.plot (date, minT, color='blue', label = \"Min T\")\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Temperature ($^\\circ$F)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's calculate and plot the daily mean temperature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meanT = (maxT + minT) / 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, meanT, color='green',label = \"Mean T\")\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Temperature ($^\\circ$F)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's use Pandas' `rolling` method to calculate the mean over a specified number of days. We'll center the window at the midpoint of each period (thus, for a 30-day window, the first plotted point will be on Jan. 16 ... covering the Jan. 1 --> Jan. 30 timeframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meanTr5 = meanT.rolling(window=5, center=True)\n",
    "meanTr10 = meanT.rolling(window=10, center=True)\n",
    "meanTr15 = meanT.rolling(window=15, center=True)\n",
    "meanTr30 = meanT.rolling(window=30, center=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "meanTr30.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, meanT, color='green',label = \"Mean T\",alpha=0.2)\n",
    "ax.plot (date, meanTr5.mean(), color='blue',label = \"5 Day\", alpha=0.3)\n",
    "ax.plot (date, meanTr10.mean(), color='purple',label = \"10 Day\", alpha=0.3)\n",
    "ax.plot (date, meanTr15.mean(), color='brown',label = \"15 Day\", alpha=0.3)\n",
    "ax.plot (date, meanTr30.mean(), color='orange',label = \"30 Day\", alpha=1.0, linewidth=2)\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Temperature ($^\\circ$F)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Display just the daily and 30-day running mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15,10))\n",
    "ax.plot (date, meanT, color='green',label = \"Mean T\",alpha=0.2)\n",
    "ax.plot (date, meanTr30.mean(), color='orange',label = \"30 Day\", alpha=1.0, linewidth=2)\n",
    "ax.set_title (\"ALB Year  %d\" % year)\n",
    "ax.set_xlabel('Date')\n",
    "ax.set_ylabel('Temperature ($^\\circ$F)' )\n",
    "ax.xaxis.set_major_locator(MonthLocator(interval=1))\n",
    "dateFmt = DateFormatter('%b %d')\n",
    "ax.xaxis.set_major_formatter(dateFmt)\n",
    "ax.legend (loc=\"best\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 August 2022 Environment",
   "language": "python",
   "name": "aug22"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}