Intro to Xarray

Outline¶

1. Introduction to Xarray
2. Anatomy of a DataArray

Imports¶

While Pandas is great for many purposes, extending its inherently 2-d data representation to a multidimensional dataset, such as 4-dimensional (time, vertical level, longitude (x) and latitude (y)) gridded NWP/Climate model output, is unwise.¶

Gridded data is typically written in a non-ASCII (i.e., not human-readable) format. The two most widely-used formats are GRIB and NetCDF. Another format, quickly gaining traction, is Zarr. These binary formats take up less storage space than text.¶

The Xarray package is ideally-suited for gridded data, in particular NetCDF and Zarr. It builds and extends on the multi-dimensional data structure in NumPy. We will see that some of the same methods we’ve used for Pandas have analogues in Xarray.¶

As a a general introduction to Xarray, below is part of a section from the Xarray Tutorial that was presented at the SciPy 2020 conference:¶

(Start of SciPy 2020 Xarray Tutorial snippet)

Multi-dimensional (a.k.a. N-dimensional, ND) arrays (sometimes called “tensors”) are an essential part of computational science. They are encountered in a wide range of fields, including physics, astronomy, geoscience, bioinformatics, engineering, finance, and deep learning. In Python, NumPy provides the fundamental data structure and API for working with raw ND arrays. However, real-world datasets are usually more than just raw numbers; they have labels which encode information about how the array values map to locations in space, time, etc.

Here is an example of how we might structure a dataset for a weather forecast:

You’ll notice multiple data variables (temperature, precipitation), coordinate variables (latitude, longitude), and dimensions (x, y, t). We’ll cover how these fit into Xarray’s data structures below.

Xarray doesn’t just keep track of labels on arrays – it uses them to provide a powerful and concise interface, with methods that look a lot like Pandas. For example:

• Apply operations over dimensions by name: x.sum('time').

• Select values by label (or logical location) instead of integer location: x.loc['2014-01-01'] or x.sel(time='2014-01-01').

• Mathematical operations (e.g., x - y) vectorize across multiple dimensions (array broadcasting) based on dimension names, not shape.

• Easily use the split-apply-combine paradigm with groupby: x.groupby('time.dayofyear').mean().

• Database-like alignment based on coordinate labels that smoothly handles missing values: x, y = xr.align(x, y, join='outer').

• Keep track of arbitrary metadata in the form of a Python dictionary: x.attrs.

The N-dimensional nature of xarray’s data structures makes it suitable for dealing with multi-dimensional scientific data, and its use of dimension names instead of axis labels (dim='time' instead of axis=0 or axis='columns') makes such arrays much more manageable than the raw numpy ndarray: with xarray, you don’t need to keep track of the order of an array’s dimensions or insert dummy dimensions of size 1 to align arrays (e.g., using np.newaxis).

The immediate payoff of using xarray is that you’ll write less code. The long-term payoff is that you’ll understand what you were thinking when you come back to look at it weeks or months later.

(End of SciPy 2020 Xarray Tutorial snippet)

Core XArray objects: the Dataset and the DataArray¶

As did Pandas with its Series and DataFrame core data structures, Xarray also has two “workhorses”: the DataArray and the Dataset. Just as a Pandas DataFrame consists of multiple Series, an Xarray Dataset is made up of one or more DataArray objects. Let’s first look at a Dataset, that is Xarray’s representation of an ERA5 gridded data file.¶

The ds Dataset has the following properties:¶

1. It has four named dimensions, in order: time, latitude, level, longitude, and level.
2. It has four coordinate variables, which (in this case, but not always) correspond to the dimensions.
3. It contains 50 data variables, such as sea_surface_temperature, geopotential and mean_sea_level_pressure. A data variable can be read in as a Data Array.
4. It has attributes which are the dataset’s metadata.
5. Its coordinate variables may have their own metadata as well.

DataArrays¶

When we analyze and display information in a gridded Xarray Dataset, we are actually working with one or more DataArrays within one or more Datasets. In the ERA5 dataset, there exist two gridded fields, one of whichis mean_sea_level_pressure. Let’s read it in as an Xarray DataArray object. You will see that we access it in a similar manner to how we accessed a column, aka Series, in a Pandas Dataframe.

Similar to (but not exactly the same as) the ds Dataset which contains it, this DataArray has the following properties:¶

1. It is a named data variable: mean_sea_level_pressure
2. It has three named dimensions, in order: time, latitude, longitude.
3. It has three coordinate variables, which (in this case, but not always) correspond to the dimensions.
4. It has attributes which are the data variable’s metadata.
5. Its coordinate variables may have their own metadata as well.

Let’s examine each of these five properties.¶

1. The data variable is represented by the DataArray object itself. We can query various properties of it, with methods similar to Pandas.¶

This invocation will return the lat, lon, and time of the minimum (or maximum) value in the DataArray (source: https://stackoverflow.com/questions/40179593/how-to-get-the-coordinates-of-the-maximum-in-xarray)

Similar to Pandas, we can use loc to select via dimension values, but note that the order of these values must correspond to the dimesnions’ order.

We can alternatively, (and preferably!) use Xarray’s sel indexing technique, where we specify the names of the dimension and the values we are selecting ... can be in any order ... and does not need to include all dimensions. In the cell below, we get the SLP values for this particular point for every time in the dataset.

Similar to Pandas, Xarray includes a plot method into Matplotlib which we can use to get a quick view of the data. Here, we will construct a time series at a specific gridpoint.¶

What if we passed in a set of coordinates that did not exactly match those in the dataset?¶

That failed, but fortunately Xarray has a handy nearest option when doing selections over coordinate dimensions!

2. Dimension names¶

In Xarray, dimensions can be thought of as extensions of Pandas` 2-d row/column indices (aka axes). We can assign names, or labels, to Pandas indexes; in Xarray, these labeled axes are a necessary (and excellent) feature.

3. Coordinates¶

Coordinate variables in Xarray are 1-dimensional arrays that correspond to the Data variable’s dimensions. In this case, slp has dimension coordinates of longitude, latitude, and time; each of these dimension coordinates consist of an array of values, plus metadata.

We can assign an object to each coordinate dimension.

4. The data variables will typically have attributes (metadata) attached to them.¶

5. The coordinate variables will likely (but not always) have metadata as well.¶

Create a quick plan-view data visualization at a particular time.¶

References¶

• Xarray Documentation
• Project Pythia’s Xarray Chapter
• ATM350 Xarray Content