MRG Interactive Developments

The Global Environmental Multiscale (GEM) model has been (and continues to be) under development at the Meterological Serivce of Canada's Numerical Research Laboratory (RPN). The GEM model is a state-of-the-art modelling system that is capable of running in both global and regional (limited area) modes, as well as on a stretched grid. The GEM documentation provided by the Canadian Meteorological Centre (CMC) describes the GEM model in detail. For information on how to download and install the latest release of the GEM model, visit the RPN COMM website. The files and documents available on this page are enhancements or updates to the official GEM model, not the full model itself. Please note that all programs and documents derived from the GEM model (for example, all source code on this page) is provided under the LGPL license, as required by the licensing of the GEM model itself.

The GEM model has been run operationally at the CMC on NEC SX-4, SX-5 and SX-6 platforms. However, the recent (c. 2001) changeover to a scalar IBM architecture has resulted in the slow degradation of vectorization in the model. In an effort to revectorize the model for use on new vector-based supercomputers such as the Earth Simulator, the Ouranos Consortium and the Canadian Regional Climate Modelling (CRCM) Network offered run time on one of their NEC SX-6 platforms. The vectorization/optimization project resulted in 98+% vectorization and significant performance enhancements (by approximately a factor of 3) on the SX-6. These links provide additional details concerning the Vectorization Project:

• Documentation of the Vectorization Project
• Vectorization Project source tarball

The GEM model uses a domain partitioning strategy for parallelization. The GEM Grid Calculator facilitates the design of valid domains, especially useful for integrations run over a large number of processors.

A solution to this bottleneck is to sacrifice a fraction of the interpolation accuracy in exchange for lower memory requirements. This interpolation truncation patch allows the user to access a truncated form of the 3D Lagrangian interpolation scheme as described in the ECMWF documentation on semi-cubic interpolation. Instead of using a 64 point cube around the origin (start of the back-trajectory), this method employs 32 points arranged in a 3D diamond and uses lower-order (linear) interpolation at points removed from the immediate viscinity of the point of interest. The compuatational cost is also lower, as the original 21 1D cubic interpolations are reduced to 7 1D cubic interpolations and 10 1D linear interpolations. It is likely, however, that performance gains seen with this truncated algorithm will focus on memory access rather than operation counts.

Installation of the interpolation truncation patch in version 3.2.0 (the latest COMM release as of June 2006 - if you are installing this patch on a different version of the GEM model, check the sections below for your model version before continuing here) is as simple as:

1. Open an experiment directory as described in the COMM model documentation.
2. Extract the required subprograms with omd_exp adw.cdk adw_interp.ftn adw_main_3_int.ftn mtn_cfg.ftn nml.cdk set_world_view1.ftn theonml.cdk.
3. Download the interpolation truncation patch (right click and save if necessary).
4. Apply the patch with patch -p 0 -b -z .orig < interp.patch
5. Rectify any reject patch segments (may occur in your Makefile - just check Makefile.rej and fix any errors).
6. Compile the code using make objloc.
7. Link using make gem to get a functional version of the model with the interpolation truncation patch.
8. Modify your gem_settings.nml settings file to remove any reference to Adw_lag3d_L and replace it with Adw_interp_type='truncated' (other valid options include 'cubic' (default) and 'lagrangian').

For reference, a copy of the interpolation subprogram is available here. The subprogram also contains additional documentation that describes the details of the simplified algorithm.

Interpolation Truncation in GEM Version 3.2.2

This patch also works on the latest internal release of the GEM model (version 3.2.2), with the exception of the first insertion for set_world_view1.ftn. This hunk contains the documentation string containing a short description of changes, and can be added manually from the set_world_view1.ftn.rej file. All modifications to the Makefile appear to fail in 3.2.2 as well, so the Makefile should be manually edited to add the modifications contained in Makefile.rej.

An additional modification must be made to adw_main_3_int.ftn at line 83: if ( Adw_lag3d_L ) then changes to if (Adw_interp_type(1:3).eq.'lag' .or. Adw_interp_type(1:3).eq.'tru') then

A vectorizing form of the interpolation truncation patch has been developed for GEM version 3.2.2, and is available for download here. This patch contains several additional files that provide vectorization for both the standard Lagrangian interpolation and the new truncated interpolation. The installation of the vectorizing version of the patch (which bit-matches with the scalar version and contains a preprocessor branch based on the NEC definition to select vector/scalar versions based on the target machine architecture) follows a similar set of steps to those listed above, with a few modifications:

1. Open an experiment directory as described in the COMM model documentation.
2. Extract the required subprograms with omd_exp adw.cdk adw_interp.ftn adw_interp2.ftn adw_main_3_int.ftn adw_main_3_intlag.ftn mtn_cfg.ftn nml.cdk set_world_view1.ftn theonml.cdk .
3. Download the interpolation truncation patch (right click and save if necessary).
4. Apply the patch with patch -p 0 -b -z .orig < interp_vector322.patch
5. Rectify any reject patch segments.
6. Add compilation options for adw_tricub_lag3d_vec.ftn, adw_tritrunc_lag3d.ftn and adw_tritrunc_lag3_vec.ftn to your Makefile.
7. Add specifications for adw_comp.cdk to your make_cdk file by inserting a target line that looks like: adw_comp.acdk: adw_comp.cdk adw_tricub_lag3d_vec.ftn adw_tritrunc_lag3d_vec.ftn and including adw_comp.cdk in the list of COMDECKS near the beginning of the generated file.
8. Compile the code using make objloc.
9. Link using make gem to get a functional version of the model with the interpolation truncation patch.
10. Modify your gem_settings.nml settings file to remove any reference to Adw_lag3d_L and replace it with Adw_interp_type='truncated' (other valid options include 'lagrangian', the default - note that the 'cubic' option is no longer available in GEM 3.2.2).

For reference, a copy of the vectorized version of the truncated interpolation subprogram is available here. The subprogram also contains additional documentation that describes the details of the simplified algorithm.

MRG Interactive is a supporter of open source initiatives and,

Questions or Comments? Contact our WebMaster.