ERA5 Data and Graphics Preparation for the Science-on-a-Sphere

ERA5 Data and Graphics Preparation for the Science-on-a-Sphere#

This notebook will create all the necessary files and directories for a visualization on NOAA’s Science-on-a-Sphere. For this example, we will produce a map of 850 hPa geopotential heights, temperature, and horizontal wind and sea-level pressure, using the ERA5, for the March 13-14 1993 winter storm, aka Superstorm ‘93.

Overview:#

Set up output directories for SoS
Specify the date/time range for your case
Use Cartopy’s add_cyclic_point method to avoid a blank seam at the dateline
Create small and large thumbnail graphics.
Create a standalone colorbar
Create a set of labels for each plot
Create 5 days worth of Science-on-a-Sphere-ready plots
Create an SoS playlist file

Prerequisites#

Concepts	Importance	Notes
Matplotlib	Necessary
Cartopy	Necessary
Xarray	Necessary
Metpy	Necessary
Linux command line / directory structure	Helpful

Time to learn: 30 minutes

Imports#

import xarray as xr
import numpy as np
import pandas as pd
import cartopy.feature as cfeature
import cartopy.crs as ccrs
import cartopy.util as cutil
from datetime import datetime as dt
import warnings
import metpy.calc as mpcalc
from metpy.units import units
import matplotlib.pyplot as plt

Do not output warning messages

warnings.simplefilter("ignore")

Set up output directories for SoS#

The software that serves output for the Science-on-a-Sphere expects a directory structure as follows:

Top level directory: choose a name that is consistent with your case, e.g.: SS93
2nd level directory: choose a name that is consistent with the graphic, e.g.: SLP_500Z
3rd level directories:
- 2048: Contains the graphics (resolution: 2048x1024) that this notebook generates
- labels: Contains one or two files:
  1. (required) A text file, labels.txt, which has as many lines as there are graphics in the 2048 file. Each line functions as the title of each graphic.
  2. (optional) A PNG file, colorbar.png, which colorbar which will get overlaid on your map graphic.
- media: Contains large and small thumbnails (thumbnail_small.jpg, thumbnail_large.jpg) that serve as icons on the SoS iPad and SoS computer apps
- playlist: A text file, playlist.sos, which tells the SoS how to display your product

As an example, here is how the directory structure on our SoS looks for the products generated by this notebook. Our SoS computer stores locally-produced content in the /shared/sos/media/site-custom directory (note: the SoS directories are not network-accessible, so you won’t be able to cd into them). The ERA5 visualizations go in the ERA5 subfolder. Your top-level directory sits within the ERA5 folder.

sos@sos1:/shared/sos/media/site-custom/ERA5/cle_superbomb/SLP_500Z$ ls -R
.:
2048  labels  media  playlist

./2048:
ERA5_1978012600-fs8.png  ERA5_1978012606-fs8.png  ERA5_1978012612-fs8.png  ERA5_1978012618-fs8.png
ERA5_1978012601-fs8.png  ERA5_1978012607-fs8.png  ERA5_1978012613-fs8.png  ERA5_1978012619-fs8.png
ERA5_1978012602-fs8.png  ERA5_1978012608-fs8.png  ERA5_1978012614-fs8.png  ERA5_1978012620-fs8.png
ERA5_1978012603-fs8.png  ERA5_1978012609-fs8.png  ERA5_1978012615-fs8.png  ERA5_1978012621-fs8.png
ERA5_1978012604-fs8.png  ERA5_1978012610-fs8.png  ERA5_1978012616-fs8.png  ERA5_1978012622-fs8.png
ERA5_1978012605-fs8.png  ERA5_1978012611-fs8.png  ERA5_1978012617-fs8.png  ERA5_1978012623-fs8.png

./labels:
colorbar.png  labels.txt

./media:
thumbnail_big.jpg  thumbnail_small.jpg

./playlist:
playlist.sos

Define the 1st and 2nd-level directories.

# You define these
caseDir = 'SS93'
prodDir = '850_T_Z_Wind'

The 3rd-level directories follow from the 1st and 2nd.

# These remain as is
graphicsDir = caseDir + '/' + prodDir + '/2048/'
labelsDir = caseDir + '/' + prodDir + '/labels/'
mediaDir = caseDir + '/' + prodDir + '/media/'
playlistDir = caseDir + '/' + prodDir + '/playlist/'

Create these directories via a Linux command

! mkdir -p {graphicsDir} {labelsDir} {mediaDir} {playlistDir}

Note: In a Jupyter notebook, the ! magic indicates that what follows is a Linux command.
The -p option for mkdir will create all subdirectories, and also will do nothing if the directories already exist.

# Remove all PNGs in the graphics directory - COMMENT OUT if you don't want to do this!
! rm -f  {graphicsDir}*.png

1) Specify a starting and ending date/time, and access ERA5 data#

Note: The sphere animation will be more effective with at least a minimum number of frames. Since the ERA5 repositories have a timestep of 6 hours (4x/day), let's set our time range to 5 days ... i.e. 20 frames.

# Date/Time specification
startYear = 1993
startMonth = 3
startDay = 11
startHour = 0
startMinute = 0
startDateTime = dt(startYear,startMonth,startDay, startHour, startMinute)
startDateTimeStr = dt.strftime(startDateTime, format="%Y%m%d")

endYear = 1993
endMonth = 3
endDay = 15
endHour = 18
endMinute = 0
endDateTime = dt(endYear,endMonth,endDay, endHour, endMinute)

# Vertical level specification
pLevel = 850
levStr = f'{pLevel}'

delta_time = endDateTime - startDateTime
time_range_max = 5*86400

if (delta_time.total_seconds() > time_range_max):
    raise RuntimeError("Your time range must not exceed 5 days. Go back and try again.")

if (delta_time.total_seconds() < 0):
    raise RuntimeError("Your end time must not be earlier than your start time! Go back and try again.")

Wb2EndDate = dt(2023,1,10)
if (endDateTime <= Wb2EndDate):
    ds = xr.open_dataset(
     'gs://weatherbench2/datasets/era5/1959-2023_01_10-wb13-6h-1440x721.zarr', 
     chunks={'time': 48},
     consolidated=True,
     engine='zarr'
)
else: 
    import glob, os
    input_directory = '/free/ktyle/era5'
    files = glob.glob(os.path.join(input_directory,'*.nc'))
    varDict = {'valid_time': 'time', 
               'pressure_level': 'level',
               'msl': 'mean_sea_level_pressure',
               'q': 'specific_humidity',
               't': 'temperature',
               'u': 'u_component_of_wind',
               'v': 'v_component_of_wind',
               'w': 'vertical_velocity',
               'z': 'geopotential'}
    dimDict = {'valid_time': 'time',
               'pressure_level': 'level'}
    ds = xr.open_mfdataset(files).rename_dims(dimDict).rename_vars(varDict)

Examine the Dataset

ds

<xarray.Dataset> Size: 47TB
Dimensions:                                           (time: 93544,
                                                       latitude: 721,
                                                       longitude: 1440,
                                                       level: 13)
Coordinates:
  * latitude                                          (latitude) float32 3kB ...
  * level                                             (level) int64 104B 50 ....
  * longitude                                         (longitude) float32 6kB ...
  * time                                              (time) datetime64[ns] 748kB ...
Data variables: (12/50)
    10m_u_component_of_wind                           (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    10m_v_component_of_wind                           (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    2m_dewpoint_temperature                           (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    2m_temperature                                    (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    angle_of_sub_gridscale_orography                  (latitude, longitude) float32 4MB dask.array<chunksize=(721, 1440), meta=np.ndarray>
    anisotropy_of_sub_gridscale_orography             (latitude, longitude) float32 4MB dask.array<chunksize=(721, 1440), meta=np.ndarray>
    ...                                                ...
    v_component_of_wind                               (time, level, latitude, longitude) float32 5TB dask.array<chunksize=(48, 13, 721, 1440), meta=np.ndarray>
    vertical_velocity                                 (time, level, latitude, longitude) float32 5TB dask.array<chunksize=(48, 13, 721, 1440), meta=np.ndarray>
    volumetric_soil_water_layer_1                     (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    volumetric_soil_water_layer_2                     (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    volumetric_soil_water_layer_3                     (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>
    volumetric_soil_water_layer_4                     (time, latitude, longitude) float32 388GB dask.array<chunksize=(48, 721, 1440), meta=np.ndarray>

ERA5 Data and Graphics Preparation for the Science-on-a-Sphere

Contents

ERA5 Data and Graphics Preparation for the Science-on-a-Sphere#

Overview:#

Prerequisites#

Imports#

Set up output directories for SoS#

1) Specify a starting and ending date/time, and access ERA5 data#

2) Specify a date/time range, and subset the desired `Dataset`s along their dimensions.#

Now create objects for our desired DataArrays based on the coordinates we have subsetted.#

Read in the desired variables; subset dimensions; and re-order longitudes.#

Examine one of the data arrays that you have read in#

Depending on what variables you have read in, set appropriate contour value ranges. The cells below are for temperature and geopotential height. Modify to fit the context of what you are visualizing!#

Create objects for the relevant coordinate arrays; in this case, longitude, latitude, and time.#

Modify so you are using a variable you have read in! The cell below assumes we have read in temperature as T.#

Create your plot.#

In this example, we will create contour lines of geopotential height, contour fills of temperature, and wind barbs.#

Subset the DataArrays for that single time (thus, eliminating the time dimension)#

If you’d like, go to https://globe-3d.vercel.app/ and view how your graphic might look on the sphere.#

Create small (128x128) and large (800x800) thumbnails. These will serve as icons in the SoS iPad and computer apps that go along with your particular product.#

Create a standalone colorbar#

Create a set of labels for each plot#

Define a function to create the plot for each hour.#

Create the graphics and titles#

Create a title string for the SoS labels file#

Create an SoS playlist file#

We’re done! The directory tree you have created can then be copied/synced to the correct directory on the SoS computer (Ross and Kevin will take care of this).#

Summary#

What’s Next?#

ERA5 Data and Graphics Preparation for the Science-on-a-Sphere

Contents

ERA5 Data and Graphics Preparation for the Science-on-a-Sphere#

Overview:#

Prerequisites#

Imports#

Set up output directories for SoS#

1) Specify a starting and ending date/time, and access ERA5 data#

2) Specify a date/time range, and subset the desired Datasets along their dimensions.#

Now create objects for our desired DataArrays based on the coordinates we have subsetted.#

Read in the desired variables; subset dimensions; and re-order longitudes.#

Examine one of the data arrays that you have read in#

Depending on what variables you have read in, set appropriate contour value ranges. The cells below are for temperature and geopotential height. Modify to fit the context of what you are visualizing!#

Create objects for the relevant coordinate arrays; in this case, longitude, latitude, and time.#

Modify so you are using a variable you have read in! The cell below assumes we have read in temperature as T.#

Create your plot.#

In this example, we will create contour lines of geopotential height, contour fills of temperature, and wind barbs.#

Subset the DataArrays for that single time (thus, eliminating the time dimension)#

If you’d like, go to https://globe-3d.vercel.app/ and view how your graphic might look on the sphere.#

Create small (128x128) and large (800x800) thumbnails. These will serve as icons in the SoS iPad and computer apps that go along with your particular product.#

Create a standalone colorbar#

Create a set of labels for each plot#

Define a function to create the plot for each hour.#

Create the graphics and titles#

Create a title string for the SoS labels file#

Create an SoS playlist file#

We’re done! The directory tree you have created can then be copied/synced to the correct directory on the SoS computer (Ross and Kevin will take care of this).#

Summary#

What’s Next?#

2) Specify a date/time range, and subset the desired `Dataset`s along their dimensions.#