Dask Arrays with Xarray

The scientific Python package known as Dask provides Dask Arrays: parallel, larger-than-memory, n-dimensional arrays that make use of blocked algorithms. They are analogous to Numpy arrays, but are distributed. These terms are defined below:

Parallel code uses many or all of the cores on the computer running the code.
Larger-than-memory refers to algorithms that break up data arrays into small pieces, operate on these pieces in an optimized fashion, and stream data from a storage device. This allows a user or programmer to work with datasets of a size larger than the available memory.
A blocked algorithm speeds up large computations by converting them into a series of smaller computations.

In this tutorial, we cover the use of Xarray to wrap Dask arrays. By using Dask arrays instead of Numpy arrays in Xarray data objects, it becomes possible to execute analysis code in parallel with much less code and effort.

Learning Objectives

Learn the distinction between eager and lazy execution, and performing both types of execution with Xarray
Understand key features of Dask Arrays
Learn to perform operations with Dask Arrays in similar ways to performing operations with NumPy arrays
Understand the use of Xarray DataArrays and Datasets as “Dask collections”, and the use of top-level Dask functions such as dask.visualize() on such collections
Understand the ability to use Dask transparently in all built-in Xarray operations

Prerequisites

Concepts	Importance	Notes
Introduction to NumPy	Necessary	Familiarity with Data Arrays
Introduction to Xarray	Necessary	Familiarity with Xarray Data Structures

Time to learn: 30-40 minutes

Imports

For this tutorial, as we are working with Dask, there are a number of Dask packages that must be imported. Also, this is technically an Xarray tutorial, so Xarray and NumPy must also be imported. Finally, the Pythia datasets package is imported, allowing access to the Project Pythia example data library.

import dask
import dask.array as da
import numpy as np
import xarray as xr
from dask.diagnostics import ProgressBar
from dask.utils import format_bytes
from pythia_datasets import DATASETS

Blocked algorithms

As described above, the definition of “blocked algorithm” is an algorithm that replaces a large operation with many small operations. In the case of datasets, this means that a blocked algorithm separates a dataset into chunks, and performs an operation on each.

As an example of how blocked algorithms work, consider a dataset containing a billion numbers, and assume that the sum of the numbers is needed. Using a non-blocked algorithm, all of the numbers are added in one operation, which is extremely inefficient. However, by using a blocked algorithm, the dataset is broken into chunks. (For the purposes of this example, assume that 1,000 chunks are created, with 1,000,000 numbers each.) The sum of the numbers in each chunk is taken, most likely in parallel, and then each of those sums are summed to obtain the final result.

By using blocked algorithms, we achieve the result, in this case one sum of one billion numbers, through the results of many smaller operations, in this case one thousand sums of one million numbers each. (Also note that each of the one thousand sums must then be summed, making the total number of sums 1,001.) This allows for a much greater degree of parallelism, potentially speeding up the code execution dramatically.

`dask.array` contains these algorithms

The main object type used in Dask is dask.array, which implements a subset of the ndarray (NumPy array) interface. However, unlike ndarray, dask.array uses blocked algorithms, which break up the array into smaller arrays, as described above. This allows for the execution of computations on arrays larger than memory, by using parallelism to divide the computation among multiple cores. Dask manages and coordinates blocked algorithms for any given computation by using Dask graphs, which lay out in detail the steps Dask takes to solve a problem. In addition, dask.array objects, known as Dask Arrays, are lazy; in other words, any computation performed on them is delayed until a specific method is called.

Create a `dask.array` object

As stated earlier, Dask Arrays are loosely based on NumPy arrays. In the next set of examples, we illustrate the main differences between Dask Arrays and NumPy arrays. In order to illustrate the differences, we must have both a Dask Array object and a NumPy array object. Therefore, this first example creates a 3-D NumPy array of random data:

shape = (600, 200, 200)
arr = np.random.random(shape)
arr

array([[[0.59882044, 0.89038597, 0.75264685, ..., 0.28575361,
         0.77133205, 0.49917285],
        [0.90349066, 0.11729641, 0.4646083 , ..., 0.31830512,
         0.4682105 , 0.09964511],
        [0.59601851, 0.1393428 , 0.1451719 , ..., 0.68692779,
         0.19844283, 0.23358143],
        ...,
        [0.58004553, 0.10514537, 0.94488101, ..., 0.02312172,
         0.61362743, 0.97977446],
        [0.52031164, 0.01479604, 0.64741797, ..., 0.86144959,
         0.3383711 , 0.05842095],
        [0.0522633 , 0.84912887, 0.14741265, ..., 0.840991  ,
         0.2148789 , 0.46491597]],

       [[0.8670142 , 0.4173522 , 0.29488315, ..., 0.62832042,
         0.31158513, 0.98188514],
        [0.54180686, 0.58585186, 0.6396745 , ..., 0.29762375,
         0.35190295, 0.27685184],
        [0.05604971, 0.55171283, 0.69862599, ..., 0.8649599 ,
         0.31128111, 0.78148618],
        ...,
        [0.98084697, 0.63897734, 0.23643916, ..., 0.85527493,
         0.27517906, 0.10073066],
        [0.13097734, 0.05187221, 0.62567868, ..., 0.34100082,
         0.24904174, 0.24744537],
        [0.83587677, 0.93942928, 0.87699588, ..., 0.26106564,
         0.48093972, 0.14538829]],

       [[0.81780786, 0.83584004, 0.95076377, ..., 0.93660647,
         0.96552286, 0.89119864],
        [0.14571403, 0.82362962, 0.79980835, ..., 0.19511307,
         0.46299543, 0.91160124],
        [0.94889911, 0.57237988, 0.93493412, ..., 0.62429476,
         0.86452692, 0.5443831 ],
        ...,
        [0.87222497, 0.65459307, 0.82116169, ..., 0.70790006,
         0.70011525, 0.10014658],
        [0.13599426, 0.61554204, 0.13890904, ..., 0.97463388,
         0.64439201, 0.89941897],
        [0.33958974, 0.94622266, 0.41707807, ..., 0.3110441 ,
         0.22638421, 0.63377419]],

       ...,

       [[0.02999301, 0.0263648 , 0.27584748, ..., 0.42909397,
         0.14590959, 0.24784125],
        [0.11041574, 0.80616766, 0.09298374, ..., 0.3421373 ,
         0.42563484, 0.64630186],
        [0.52209463, 0.41091542, 0.10486451, ..., 0.25035955,
         0.93942855, 0.66156348],
        ...,
        [0.64340089, 0.39739298, 0.8019468 , ..., 0.93968375,
         0.06318134, 0.36837465],
        [0.75963579, 0.49378495, 0.30624372, ..., 0.61438427,
         0.30653086, 0.14667532],
        [0.43440798, 0.63575079, 0.69847263, ..., 0.60040108,
         0.30731612, 0.81611883]],

       [[0.1283456 , 0.87703892, 0.23752744, ..., 0.07754181,
         0.93005294, 0.53346445],
        [0.43698266, 0.61582401, 0.02379739, ..., 0.65570026,
         0.97293592, 0.66378911],
        [0.73203301, 0.70717493, 0.36481202, ..., 0.86921607,
         0.43639917, 0.47504516],
        ...,
        [0.12499336, 0.66581081, 0.30877271, ..., 0.98772465,
         0.16656457, 0.77863969],
        [0.87453784, 0.4903757 , 0.68990367, ..., 0.29029562,
         0.94981556, 0.23970464],
        [0.17938946, 0.78932973, 0.51488989, ..., 0.63125446,
         0.70059609, 0.71615045]],

       [[0.09134313, 0.56968382, 0.25261009, ..., 0.47449169,
         0.5028163 , 0.18931504],
        [0.59257282, 0.79706897, 0.61743135, ..., 0.43450106,
         0.85023722, 0.37430368],
        [0.65896213, 0.00747606, 0.72997344, ..., 0.53380521,
         0.61987468, 0.81783034],
        ...,
        [0.20905158, 0.5007961 , 0.99068535, ..., 0.19769782,
         0.06665382, 0.84617634],
        [0.04103984, 0.70854959, 0.42010955, ..., 0.65786467,
         0.44327821, 0.4428599 ],
        [0.33840587, 0.79426061, 0.79334088, ..., 0.47697519,
         0.41063649, 0.38140662]]])

format_bytes(arr.nbytes)

'183.11 MiB'

As shown above, this NumPy array contains about 183 MB of data.

As stated above, we must also create a Dask Array. This next example creates a Dask Array with the same dimension sizes as the existing NumPy array:

darr = da.random.random(shape, chunks=(300, 100, 200))

By specifying values to the chunks keyword argument, we can specify the array pieces that Dask’s blocked algorithms break the array into; in this case, we specify (300, 100, 200).

Specifying Chunks

In this tutorial, we specify Dask Array chunks in a block shape. However, there are many additional ways to specify chunks; see this documentation for more details.

If you are viewing this page as a Jupyter Notebook, the next Jupyter cell will produce a rich information graphic giving in-depth details about the array and each individual chunk.

darr

	Array	Chunk
Bytes	183.11 MiB	45.78 MiB
Shape	(600, 200, 200)	(300, 100, 200)
Dask graph	4 chunks in 1 graph layer
Data type	float64 numpy.ndarray

The above graphic contains a symbolic representation of the array, including shape, dtype, and chunksize. (Your view may be different, depending on how you are accessing this page.) Notice that there is no data shown for this array; this is because Dask Arrays are lazy, as described above. Before we call a compute method for this array, we first illustrate the structure of a Dask graph. In this example, we show the Dask graph by calling .visualize() on the array:

darr.visualize()

../../_images/57f39e3d0829df5b12fc9298b4722d64bea6b8ef6a3e0b911e26309374691f26.png

As shown in the above Dask graph, our array has four chunks, each one created by a call to NumPy’s “random” method (np.random.random). These chunks are concatenated into a single array after the calculation is performed.

Manipulate a `dask.array` object as you would a numpy array

We can perform computations on the Dask Array created above in a similar fashion to NumPy arrays. These computations include arithmetic, slicing, and reductions, among others.

Although the code for performing these computations is similar between NumPy arrays and Dask Arrays, the process by which they are performed is quite different. For example, it is possible to call sum() on both a NumPy array and a Dask Array; however, these two sum() calls are definitely not the same, as shown below.

What’s the difference?

When sum() is called on a Dask Array, the computation is not performed; instead, an expression of the computation is built. The sum() computation, as well as any other computation methods called on the same Dask Array, are not performed until a specific method (known as a compute method) is called on the array. (This is known as lazy execution.) On the other hand, calling sum() on a NumPy array performs the calculation immediately; this is known as eager execution.

Why the difference?

As described earlier, a Dask Array is divided into chunks. Any computations run on the Dask Array run on each chunk individually. If the result of the computation is obtained before the computation runs through all of the chunks, Dask can stop the computation to save CPU time and memory resources.

This example illustrates calling sum() on a Dask Array; it also includes a demonstration of lazy execution, as well as another Dask graph display:

total = darr.sum()
total

	Array	Chunk
Bytes	8 B	8 B
Shape	()	()
Dask graph	1 chunks in 3 graph layers
Data type	float64 numpy.ndarray

total.visualize()

../../_images/caf93b42a7330563cb488018fff85a8cb1fef4526fde9351f6620316599f258c.png

Compute the result

As described above, Dask Array objects make use of lazy execution. Therefore, operations performed on a Dask Array wait to execute until a compute method is called. As more operations are queued in this way, the Dask Array’s Dask graph increases in complexity, reflecting the steps Dask will take to perform all of the queued operations.

In this example, we call a compute method, simply called .compute(), to run on the Dask Array all of the stored computations:

%%time
total.compute()

CPU times: user 315 ms, sys: 56.2 ms, total: 371 ms
Wall time: 216 ms

12001615.82024877

Exercise with `dask.arrays`

In this section of the page, the examples are hands-on exercises pertaining to Dask Arrays. If these exercises are not interesting to you, this section can be used strictly as examples regardless of how the page is viewed. However, if you wish to participate in the exercises, make sure that you are viewing this page as a Jupyter Notebook.

For the first exercise, modify the chunk size or shape of the Dask Array created earlier. Call .sum() on the modified Dask Array, and visualize the Dask graph to view the changes.

da.random.random(shape, chunks=(50, 200, 400)).sum().visualize()

../../_images/c72da04b1bfe3a506395391dafe24ac8521e9cf45f986769fdc135046116258b.png

As is obvious from the above exercise, Dask quickly and easily determines a strategy for performing the operations, in this case a sum. This illustrates the appeal of Dask: automatic algorithm generation that scales from simple arithmetic problems to highly complex scientific equations with large datasets and multiple operations.

In this next set of examples, we demonstrate that increasing the complexity of the operations performed also increases the complexity of the Dask graph.

In this example, we use randomly selected functions, arguments and Python slices to create a complex set of operations. We then visualize the Dask graph to illustrate the increased complexity:

z = darr.dot(darr.T).mean(axis=0)[::2, :].std(axis=1)
z

	Array	Chunk
Bytes	468.75 kiB	117.19 kiB
Shape	(100, 600)	(50, 300)
Dask graph	4 chunks in 12 graph layers
Data type	float64 numpy.ndarray

z.visualize()

../../_images/a75f3ba474e4d57105fe3c2cdb9111b3716c368c48feaa67dbaf0068330e50ca.png

Testing a bigger calculation

While the earlier examples in this tutorial described well the basics of Dask, the size of the data in those examples, about 180 MB, is far too small for an actual use of Dask.

In this example, we create a much larger array, more indicative of data actually used in Dask:

darr = da.random.random((4000, 100, 4000), chunks=(1000, 100, 500)).astype('float32')
darr

	Array	Chunk
Bytes	5.96 GiB	190.73 MiB
Shape	(4000, 100, 4000)	(1000, 100, 500)
Dask graph	32 chunks in 2 graph layers
Data type	float32 numpy.ndarray

The dataset created in the previous example is much larger, approximately 6 GB. Depending on how many programs are running on your computer, this may be greater than the amount of free RAM on your computer. However, as Dask is larger-than-memory, the amount of free RAM does not impede Dask’s ability to work on this dataset.

In this example, we again perform randomly selected operations, but this time on the much larger dataset. We also visualize the Dask graph, and then run the compute method. However, as computing complex functions on large datasets is inherently time-consuming, we show a progress bar to track the progress of the computation.

z = (darr + darr.T)[::2, :].mean(axis=2)

z.visualize()

../../_images/8481b13bcbc6932ef6afca8f5af6f92d436ef1aad49f5bf803ba1ec9074b8254.png

with ProgressBar():
    computed_ds = z.compute()

[                                        ] | 0% Completed | 317.64 us

[                                        ] | 0% Completed | 107.35 ms

[                                        ] | 0% Completed | 208.27 ms

[                                        ] | 0% Completed | 309.16 ms

[                                        ] | 0% Completed | 410.10 ms

[                                        ] | 0% Completed | 511.02 ms

[#                                       ] | 4% Completed | 612.28 ms

[#####                                   ] | 13% Completed | 717.50 ms

[#######                                 ] | 18% Completed | 820.11 ms

[##########                              ] | 26% Completed | 923.77 ms

[##############                          ] | 36% Completed | 1.03 s

[#####################                   ] | 52% Completed | 1.14 s

[#######################                 ] | 59% Completed | 1.25 s

[###########################             ] | 69% Completed | 1.36 s

[###############################         ] | 78% Completed | 1.46 s

[###################################     ] | 87% Completed | 1.58 s

[########################################] | 100% Completed | 1.68 s

Dask Arrays with Xarray

While directly interacting with Dask Arrays can be useful on occasion, more often than not Dask Arrays are interacted with through Xarray. Since Xarray wraps NumPy arrays, and Dask Arrays contain most of the functionality of NumPy arrays, Xarray can also wrap Dask Arrays, allowing anyone with knowledge of Xarray to easily start using the Dask interface.

Reading data with `Dask` and `Xarray`

As demonstrated in previous examples, a Dask Array consists of many smaller arrays, called chunks:

darr

	Array	Chunk
Bytes	5.96 GiB	190.73 MiB
Shape	(4000, 100, 4000)	(1000, 100, 500)
Dask graph	32 chunks in 2 graph layers
Data type	float32 numpy.ndarray

As shown in the following example, to read data into Xarray as Dask Arrays, simply specify the chunks keyword argument when calling the open_dataset() function:

ds = xr.open_dataset(DATASETS.fetch('CESM2_sst_data.nc'), chunks={})
ds.tos

/free1/ktyle/miniforge3/envs/foundations_book24/lib/python3.12/site-packages/xarray/conventions.py:440: SerializationWarning: variable 'tos' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.
  new_vars[k] = decode_cf_variable(

<xarray.DataArray 'tos' (time: 180, lat: 180, lon: 360)> Size: 47MB
dask.array<open_dataset-tos, shape=(180, 180, 360), dtype=float32, chunksize=(1, 180, 360), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) object 1kB 2000-01-15 12:00:00 ... 2014-12-15 12:00:00
  * lat      (lat) float64 1kB -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
  * lon      (lon) float64 3kB 0.5 1.5 2.5 3.5 4.5 ... 356.5 357.5 358.5 359.5
Attributes: (12/19)
    cell_measures:  area: areacello
    cell_methods:   area: mean where sea time: mean
    comment:        Model data on the 1x1 grid includes values in all cells f...
    description:    This may differ from "surface temperature" in regions of ...
    frequency:      mon
    id:             tos
    ...             ...
    time_label:     time-mean
    time_title:     Temporal mean
    title:          Sea Surface Temperature
    type:           real
    units:          degC
    variable_id:    tos

While it is a valid operation to pass an empty list to the chunks keyword argument, this technique does not specify how to chunk the data, and therefore the resulting Dask Array contains only one chunk.

Correct usage of the chunks keyword argument specifies how many values in each dimension are contained in a single chunk. In this example, specifying the chunks keyword argument as chunks={'time':90} indicates to Xarray and Dask that 90 time slices are allocated to each chunk on the temporal axis.

Since this dataset contains 180 total time slices, the data variable tos (holding the sea surface temperature data) is now split into two chunks in the temporal dimension.

ds = xr.open_dataset(
    DATASETS.fetch('CESM2_sst_data.nc'),
    engine="netcdf4",
    chunks={"time": 90, "lat": 180, "lon": 360},
)
ds.tos

/free1/ktyle/miniforge3/envs/foundations_book24/lib/python3.12/site-packages/xarray/conventions.py:440: SerializationWarning: variable 'tos' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.
  new_vars[k] = decode_cf_variable(

<xarray.DataArray 'tos' (time: 180, lat: 180, lon: 360)> Size: 47MB
dask.array<open_dataset-tos, shape=(180, 180, 360), dtype=float32, chunksize=(90, 180, 360), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) object 1kB 2000-01-15 12:00:00 ... 2014-12-15 12:00:00
  * lat      (lat) float64 1kB -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
  * lon      (lon) float64 3kB 0.5 1.5 2.5 3.5 4.5 ... 356.5 357.5 358.5 359.5
Attributes: (12/19)
    cell_measures:  area: areacello
    cell_methods:   area: mean where sea time: mean
    comment:        Model data on the 1x1 grid includes values in all cells f...
    description:    This may differ from "surface temperature" in regions of ...
    frequency:      mon
    id:             tos
    ...             ...
    time_label:     time-mean
    time_title:     Temporal mean
    title:          Sea Surface Temperature
    type:           real
    units:          degC
    variable_id:    tos

It is fairly straightforward to retrieve a list of the chunks and their sizes for each dimension; simply call the .chunks method on an Xarray DataArray. In this example, we show that the tos DataArray now contains two chunks on the time dimension, with each chunk containing 90 time slices.

ds.tos.chunks

((90, 90), (180,), (360,))

Xarray data structures are first-class dask collections

If an Xarray Dataset or DataArray object uses a Dask Array, rather than a NumPy array, it counts as a first-class Dask collection. This means that you can pass such an object to dask.visualize() and dask.compute(), in the same way as an individual Dask Array.

In this example, we call dask.visualize on our Xarray DataArray, displaying a Dask graph for the DataArray object:

dask.visualize(ds)

../../_images/14823a00387a43fce8fd7b4035c5b98e68204d306bb0ab6a33c23ad5d09dd244.png

Parallel and lazy computation using `dask.array` with Xarray

As described above, Xarray Datasets and DataArrays containing Dask Arrays are first-class Dask collections. Therefore, computations performed on such objects are deferred until a compute method is called. (This is the definition of lazy computation.)

z = ds.tos.mean(['lat', 'lon']).dot(ds.tos.T)
z

<xarray.DataArray 'tos' (lon: 360, lat: 180)> Size: 259kB
dask.array<sum-aggregate, shape=(360, 180), dtype=float32, chunksize=(360, 180), chunktype=numpy.ndarray>
Coordinates:
  * lat      (lat) float64 1kB -89.5 -88.5 -87.5 -86.5 ... 86.5 87.5 88.5 89.5
  * lon      (lon) float64 3kB 0.5 1.5 2.5 3.5 4.5 ... 356.5 357.5 358.5 359.5

As shown in the above example, the result of the applied operations is an Xarray DataArray that contains a Dask Array, an identical object type to the object that the operations were performed on. This is true for any operations that can be applied to Xarray DataArrays, including subsetting operations; this next example illustrates this:

z.isel(lat=0)

<xarray.DataArray 'tos' (lon: 360)> Size: 1kB
dask.array<getitem, shape=(360,), dtype=float32, chunksize=(360,), chunktype=numpy.ndarray>
Coordinates:
    lat      float64 8B -89.5
  * lon      (lon) float64 3kB 0.5 1.5 2.5 3.5 4.5 ... 356.5 357.5 358.5 359.5

Because the data subset created above is also a first-class Dask collection, we can view its Dask graph using the dask.visualize() function, as shown in this example:

dask.visualize(z)

../../_images/1ae1aba7705f9f4c003ebc48f3b0f687a2031ed7f9d6cb8c8393bf159c6b7188.png

Since this object is a first-class Dask collection, the computations performed on it have been deferred. To run these computations, we must call a compute method, in this case .compute(). This example also uses a progress bar to track the computation progress.

with ProgressBar():
    computed_ds = z.compute()

[                                        ] | 0% Completed | 181.35 us

[######################                  ] | 57% Completed | 101.09 ms

[########################################] | 100% Completed | 201.81 ms

Summary

This tutorial covered the use of Xarray to access Dask Arrays, and the use of the chunks keyword argument to open datasets with Dask data instead of NumPy data. Another important concept introduced in this tutorial is the usage of Xarray Datasets and DataArrays as Dask collections, allowing Xarray objects to be manipulated in a similar manner to Dask Arrays. Finally, the concepts of larger-than-memory datasets, lazy computation, and parallel computation, and how they relate to Xarray and Dask, were covered.

Dask Shortcomings

Although Dask Arrays and NumPy arrays are generally interchangeable, NumPy offers some functionality that is lacking in Dask Arrays. The usage of Dask Array comes with the following relevant issues:

Operations where the resulting shape depends on the array values can produce erratic behavior, or fail altogether, when used on a Dask Array. If the operation succeeds, the resulting Dask Array will have unknown chunk sizes, which can cause other sections of code to fail.
Operations that are by nature difficult to parallelize or less useful on very large datasets, such as sort, are not included in the Dask Array interface. Some of these operations have supported versions that are inherently more intuitive to parallelize, such as topk.
Development of new Dask functionality is only initiated when such functionality is required; therefore, some lesser-used NumPy functions, such as np.sometrue, are not yet implemented in Dask. However, many of these functions can be added as community contributions, or have already been added in this manner.

Learn More

For more in-depth information on Dask Arrays, visit the official documentation page. In addition, this screencast reinforces the concepts covered in this tutorial. (If you are viewing this page as a Jupyter Notebook, the screencast will appear below as an embedded YouTube video.)

from IPython.display import YouTubeVideo

YouTubeVideo(id="9h_61hXCDuI", width=600, height=300)

Resources and references

To find specific reference information about Dask and Xarray, see the official documentation pages listed below:
If you require assistance with a specific issue involving Xarray or Dask, the following resources may be of use:
- Dask tag on StackOverflow, for usage questions
- github discussions: dask for general, non-bug, discussion, and usage questions
- github issues: dask for bug reports and feature requests
- github discussions: xarray for general, non-bug, discussion, and usage questions
- github issues: xarray for bug reports and feature requests
Certain sections of this tutorial are adapted from the following existing tutorials:
- Dask Array Tutorial
- Parallel Computing with Xarray and Dask

Dask Arrays with Xarray

Learning Objectives

Prerequisites

Imports

Blocked algorithms

`dask.array` contains these algorithms

Create a `dask.array` object

Manipulate a `dask.array` object as you would a numpy array

What’s the difference?

Why the difference?

Compute the result

Exercise with `dask.arrays`

Testing a bigger calculation

Dask Arrays with Xarray

Reading data with `Dask` and `Xarray`

Xarray data structures are first-class dask collections

Parallel and lazy computation using `dask.array` with Xarray

Summary

Dask Shortcomings

Learn More

Resources and references

About

Give Feedback

Contribute

Dask Arrays with Xarray

Learning Objectives

Prerequisites

Imports

Blocked algorithms

dask.array contains these algorithms

Create a dask.array object

Manipulate a dask.array object as you would a numpy array

What’s the difference?

Why the difference?

Compute the result

Exercise with dask.arrays

Testing a bigger calculation

Dask Arrays with Xarray

Reading data with Dask and Xarray

Xarray data structures are first-class dask collections

Parallel and lazy computation using dask.array with Xarray

Summary

Dask Shortcomings

Learn More

Resources and references

`dask.array` contains these algorithms

Create a `dask.array` object

Manipulate a `dask.array` object as you would a numpy array

Exercise with `dask.arrays`

Reading data with `Dask` and `Xarray`

Parallel and lazy computation using `dask.array` with Xarray