{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "# Xarray 1: Introduction to Xarray" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overview\n", "\n", "This notebook will introduce the basics of gridded, labeled data with Xarray. Since Xarray introduces additional abstractions on top of plain arrays of data, our goal is to show why these abstractions are useful and how they frequently lead to simpler, more robust code.\n", "\n", "We'll cover these topics:\n", "\n", "1. Create a `DataArray`, one of the core object types in Xarray\n", "1. Understand how to use named coordinates and metadata in a `DataArray`\n", "1. Combine individual `DataArrays` into a `Dataset`, the other core object type in Xarray\n", "1. Subset, slice, and interpolate the data using named coordinates\n", "1. Open netCDF data using XArray\n", "1. Basic subsetting and aggregation of a `Dataset`\n", "1. Brief introduction to plotting with Xarray" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prerequisites\n", "\n", "| Concepts | Importance | Notes |\n", "| --- | --- | --- |\n", "| [NumPy Basics](../numpy/numpy-basics) | Necessary | |\n", "| [Intermediate NumPy](../numpy/intermediate-numpy) | Helpful | Familiarity with indexing and slicing arrays |\n", "| [NumPy Broadcasting](../numpy/numpy-broadcasting) | Helpful | Familiar with array arithmetic and broadcasting |\n", "| [Introduction to Pandas](../pandas/pandas) | Helpful | Familiarity with labeled data |\n", "| [Datetime](../datetime/datetime) | Helpful | Familiarity with time formats and the `timedelta` object |\n", "| [Understanding of NetCDF](some-link-to-external-resource) | Helpful | Familiarity with metadata structure |\n", "\n", "- **Time to learn**: 40 minutes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Simmilar to `numpy`, `np`; `pandas`, `pd`; you may often encounter `xarray` imported within a shortened namespace as `xr`. `pythia_datasets` provides example data for us to work with." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from datetime import timedelta\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import xarray as xr\n", "from pythia_datasets import DATASETS" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introducing the `DataArray` and `Dataset`\n", "\n", "Xarray expands on the capabilities on NumPy arrays, providing a lot of streamlined data manipulation. It is similar in that respect to Pandas, but whereas Pandas excels at working with tabular data, Xarray is focused on N-dimensional arrays of data (i.e. grids). Its interface is based largely on the netCDF data model (variables, attributes, and dimensions), but it goes beyond the traditional netCDF interfaces to provide functionality similar to netCDF-java's [Common Data Model (CDM)](https://docs.unidata.ucar.edu/netcdf-java/current/userguide/common_data_model_overview.html). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creation of a `DataArray` object\n", "\n", "The `DataArray` is one of the basic building blocks of Xarray (see docs [here](http://xarray.pydata.org/en/stable/user-guide/data-structures.html#dataarray)). It provides a `numpy.ndarray`-like object that expands to provide two critical pieces of functionality:\n", "\n", "1. Coordinate names and values are stored with the data, making slicing and indexing much more powerful\n", "2. It has a built-in container for attributes\n", "\n", "Here we'll initialize a `DataArray` object by wrapping a plain NumPy array, and explore a few of its properties." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Generate a random numpy array\n", "\n", "For our first example, we'll just create a random array of \"temperature\" data in units of Kelvin:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = 283 + 5 * np.random.randn(5, 3, 4)\n", "data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Wrap the array: first attempt\n", "\n", "Now we create a basic `DataArray` just by passing our plain `data` as input:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp = xr.DataArray(data)\n", "temp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note two things:\n", "\n", "1. Xarray generates some basic dimension names for us (`dim_0`, `dim_1`, `dim_2`). We'll improve this with better names in the next example.\n", "2. Wrapping the numpy array in a `DataArray` gives us a rich display in the notebook! (Try clicking the array symbol to expand or collapse the view)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Assign dimension names\n", "\n", "Much of the power of Xarray comes from making use of named dimensions. So let's add some more useful names! We can do that by passing an ordered list of names using the keyword argument `dims`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp = xr.DataArray(data, dims=['time', 'lat', 'lon'])\n", "temp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is already improved upon from a NumPy array, because we have names for each of the dimensions (or axes in NumPy parlance). Even better, we can take arrays representing the values for the coordinates for each of these dimensions and associate them with the data when we create the `DataArray`. We'll see this in the next example." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create a `DataArray` with named Coordinates\n", "\n", "#### Make time and space coordinates\n", "\n", "Here we will use [Pandas](../pandas) to create an array of [datetime data](../datetime), which we will then use to create a `DataArray` with a named coordinate `time`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "times = pd.date_range('2018-01-01', periods=5)\n", "times" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll also create arrays to represent sample longitude and latitude:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lons = np.linspace(-120, -60, 4)\n", "lats = np.linspace(25, 55, 3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Initialize the `DataArray` with complete coordinate info\n", "\n", "When we create the `DataArray` instance, we pass in the arrays we just created:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp = xr.DataArray(data, coords=[times, lats, lons], dims=['time', 'lat', 'lon'])\n", "temp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Set useful attributes\n", "\n", "...and while we're at it, we can also set some attribute metadata:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp.attrs['units'] = 'kelvin'\n", "temp.attrs['standard_name'] = 'air_temperature'\n", "\n", "temp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Attributes are not preserved by default!\n", "\n", "Notice what happens if we perform a mathematical operaton with the `DataArray`: the coordinate values persist, but the attributes are lost. This is done because it is very challenging to know if the attribute metadata is still correct or appropriate after arbitrary arithmetic operations.\n", "\n", "To illustrate this, we'll do a simple unit conversion from Kelvin to Celsius:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp_in_celsius = temp - 273.15\n", "temp_in_celsius" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For an in-depth discussion of how Xarray handles metadata, start in the Xarray docs [here](http://xarray.pydata.org/en/stable/getting-started-guide/faq.html#approach-to-metadata)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The `Dataset`: a container for `DataArray`s with shared coordinates\n", "\n", "Along with `DataArray`, the other key object type in Xarray is the `Dataset`: a dictionary-like container that holds one or more `DataArray`s, which can also optionally share coordinates (see docs [here](http://xarray.pydata.org/en/stable/user-guide/data-structures.html#dataset)).\n", "\n", "The most common way to create a `Dataset` object is to load data from a file (see [below](#Opening-netCDF-data)). Here, instead, we will create another `DataArray` and combine it with our `temp` data.\n", "\n", "This will illustrate how the information about common coordinate axes is used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create a pressure `DataArray` using the same coordinates\n", "\n", "This code mirrors how we created the `temp` object above." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pressure_data = 1000.0 + 5 * np.random.randn(5, 3, 4)\n", "pressure = xr.DataArray(\n", " pressure_data, coords=[times, lats, lons], dims=['time', 'lat', 'lon']\n", ")\n", "pressure.attrs['units'] = 'hPa'\n", "pressure.attrs['standard_name'] = 'air_pressure'\n", "\n", "pressure" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Create a `Dataset` object\n", "\n", "Each `DataArray` in our `Dataset` needs a name! \n", "\n", "The most straightforward way to create a `Dataset` with our `temp` and `pressure` arrays is to pass a dictionary using the keyword argument `data_vars`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ds = xr.Dataset(data_vars={'Temperature': temp, 'Pressure': pressure})\n", "ds" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Notice that the `Dataset` object `ds` is aware that both data arrays sit on the same coordinate axes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Access Data variables and Coordinates in a `Dataset`\n", "\n", "We can pull out any of the individual `DataArray` objects in a few different ways.\n", "\n", "Using the \"dot\" notation:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ds.Pressure" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "... or using dictionary access like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ds['Pressure']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We'll return to the `Dataset` object when we start loading data from files." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Subsetting and selection by coordinate values\n", "\n", "Much of the power of labeled coordinates comes from the ability to select data based on coordinate names and values, rather than array indices. We'll explore this briefly here.\n", "\n", "### NumPy-like selection\n", "\n", "Suppose we want to extract all the spatial data for one single date: January 2, 2018. It's possible to achieve that with NumPy-like index selection:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "indexed_selection = temp[1, :, :] # Index 1 along axis 0 is the time slice we want...\n", "indexed_selection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "HOWEVER, notice that this requires us (the user / programmer) to have **detailed knowledge** of the order of the axes and the meaning of the indices along those axes!\n", "\n", "_**Named coordinates free us from this burden...**_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Selecting with `.sel()`\n", "\n", "We can instead select data based on coordinate values using the `.sel()` method, which takes one or more named coordinate(s) as keyword argument:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "named_selection = temp.sel(time='2018-01-02')\n", "named_selection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We got the same result, but \n", "- we didn't have to know anything about how the array was created or stored\n", "- our code is agnostic about how many dimensions we are dealing with\n", "- the intended meaning of our code is much clearer!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Approximate selection and interpolation\n", "\n", "With time and space data, we frequently want to sample \"near\" the coordinate points in our dataset. Here are a few simple ways to achieve that.\n", "\n", "#### Nearest-neighbor sampling\n", "\n", "Suppose we want to sample the nearest datapoint within 2 days of date `2018-01-07`. Since the last day on our `time` axis is `2018-01-05`, this is well-posed.\n", "\n", "`.sel` has the flexibility to perform nearest neighbor sampling, taking an optional tolerance:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp.sel(time='2018-01-07', method='nearest', tolerance=timedelta(days=2))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "where we see that `.sel` indeed pulled out the data for date `2018-01-05`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Interpolation\n", "\n", "Suppose we want to extract a timeseries for Boulder (40°N, 105°W). Since `lon=-105` is _not_ a point on our longitude axis, this requires interpolation between data points.\n", "\n", "The `.interp()` method (see the docs [here](http://xarray.pydata.org/en/stable/interpolation.html)) works similarly to `.sel()`. Using `.interp()`, we can interpolate to any latitude/longitude location:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "temp.interp(lon=-105, lat=40)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Info
\n", " Xarray's interpolation functionality requires the SciPy package!\n", "Info
\n", " The calling sequence forslice
always looks like slice(start, stop[, step])
, where step
is optional.\n",
"Info
\n", " Here we're getting the data from Project Pythia's custom library of example data, which we already imported above withfrom pythia_datasets import DATASETS
. The DATASETS.fetch()
method will automatically download and cache our example data file NARR_19930313_0000.nc
locally.\n",
"Info
\n", " Aggregation methods for Xarray objects operate over the named coordinate dimension(s) specified by keyword argumentdim
. Compare to NumPy, where aggregations operate over specified numbered axes
.\n",
"