MRG Interactive Developments

Mesoscale Research Group, McGill/UAlbany/UQAM

General GEMPAK Information and Products

  • GEMPAK Background
  • GEMPAK Installation Notes
  • The GEMPAK speedPatch
  • Grib Table Creation
  • Degribbing NARR Data
  • Horizontal Grid-to-Grid Interpolation Bit Packing
  • General GEMPAK Notes

  • General Information about GEMPAK

    The GEMPAK package is powerful diagnostic and display sofware developed over the past couple of decades by various folks in the United States government and academic institutions. Because its base language is Fortran77, it suffers from many of the limitations imposed by the age and inflexibility of that language. A few of the issues related to the installation and setup of the package are noted here; however, a search of the email archives will yield more information than that provided here.

    Installation and Setup of GEMPAK

    For the most part, GEMPAK does a pretty good job of installing itself once you have all of the required libraries (note that it doesn't always warn you if you don't until it's too late). However, there are a few setup issues that you should deal with before moving to the configuration and compilation steps. The primary issues involve the default parameters defined in the gempak/include subdirectory of your source distribution (note that you'll need the full source rather than a binary release to enact the changes described here). Modify the GEMPRM.PRM parameter file in the subdirectory to boost (at least) the following fixed parameters:

  • mmhdrs (max # of headers) from 30000 to 31700 (anything above 30000 is safe)
  • llmxtm (max # of different times) from 300 to 5000 (go as big as you can)
  • llmxgt (max # of grid times) from 1000 to 5000 (again - as big as you can)
  • This parameter file (comdec) contains a LOT of restrictions on the operation of the package, most of which are becoming unnecessary with increased computer power. Take a few minutes to read through the parameter file and increase the defaults as necessary for the work that you plan to do. Also, remember this file when you run into bizarre functionality while using the package ... especially if it inolves funny things happening to grids above or below a certain regular grid ID number.

    Before continuing with the installation, take a few seconds to read through the speedPatch section of this document. The GEMPAK speedPatch greatly reduces the run-time of GEMPAK applications while maintaining the stability of the system. Essentially, there's no reason not to apply the speedPatch to your installation.

    If you forget to modify these parameters before beginning the installation, I've found that it is preferrable to clean completely and begin from scratch. This is accomplished using the make distclean command from the top level of the distribution. Any other clean does not completely remove the objects that depend on the GEMPRM.PRM comdec file.

    Note to users with Portland Group compilers

    Portland Group compilers require some different configurations from standard compilers to get GEMPAK to build correctly. Fortunately, there is a Makeinc.linux.pgi file already provided in the basic distribution, so the modifications are fairly easy:

    1. Get to the configuration directory (cd ${NAWIPS}/config).
    2. Save the current linux configuration using mv Makeinc.linux Makeinc.linux.orig.
    3. Back up the linux PGI configuration using cp Makeinc.linux.pgi Makeinc.linux.pgi.orig.
    4. [VERSION 5.9.1 users] Run perl -p -e 's/^GEM((OLB)|(EXE))/#GEM$1/g' Makeinc.linux.pgi.orig > Makeinc.linux.pgi to correct an error in the macro settings of the configuration file.
    5. Link the configurations using ln -s Makeinc.linux.pgi Makeinc.linux.
    6. Go back to the package root directory (cd $NAWIPS) and continue with the build procedure as documented on the GEMPAK web pages.

    Another problem might occur because older PG C compilers do not recognize the '//' comment syntax. The easiest fix for this is to insert a -B into the COPT line of the $NAWIPS/config/Makeinc.linux configuration file. As well, it is possible that the bool data type will cause problems in the building of the C-based GEMPAK library. The easiest solution for this is to change the type of the bool declaration in $NAWIPS/os/linux/include/jasper/jas_image.h to char.

    Note to users with Intel compilers

    Portland Group compilers cost a ton to buy and support, so why not use the free academic license of the Intel compilers? Intel provides a full sweet of Fortran and C/C++ compilers for academic use. The disadvantage (there's always a catch, right?) is that they're unsupported unless you pay money, but that's not a really big deal if you can figure out how to make things work.

    In general, compiling GEMPAK with Intel compilers (Fortran: ifort; C/C++: icc) is fairly simple. Download the file and add it to your $NAWIPS/config directory (where $NAWIPS is the environment variable referencing the top level of your GEMPAK distribution - it should be set automatically when you source Gemenviron). Then move the exisiting Makeinc.linux settings to something like Makeinc.linux.orig and link your downloaded to Makeinc.linux. Note that I have installed my Motif libraries (actually OpenMotif) under /opt/motif. If you have installed your Motif libraries under a different path, you will have to change the to reflect that (just search for /opt/motif in the document you downloaded.

    Once you have the link made, go to $NAWIPS/gempak/include directory and move MCHPRM.Linux to something like MCHPRM.Linux.orig. Then link MCHPRM.X86 to MCHPRM.Linux. That will take care of syntax errors (according to the Intel compiler) in the include file.

    Once that link is done, make sure that you have updated your distribution as outlined in the installation section above, then go back to the $NAWIPS directory and continue with the installation using make Everything. Everything should go pretty well, except when Fortran objects are linked to C objects. When that happens, you'll get an error that looks like: /opt/intel/fce/9.1.039/lib/for_main.o(.text+0x2e): In function `main':
    : undefined reference to `MAIN__'
    Don't panic. Allow the build to continue and we'll fix this at the end.

    Once the build is finished, you need to edit your $NAWIPS/config/Makeinc.linux file to add the option -nofor_main to the list of Fortran compiler options. Search for the definition of FOPT and add this option to the line, so that it looks something like FOPT = $(GEMINC) $(GEMINC)/$(OPSYS) -O -nofor_main # -g. Then go back to the $NAWIPS directory and run make all (not Everything this time, otherwise you'll nuke all the work that you've already done). That should take care of the errors note above, and you should just have to run make install to get everything running smoothly. The two-step build is a little ugly, but it seems to work well.


    The ActiveGEM GUI for GEMPAK is an open source project of MRG Interactive that provides a free research-oriented Graphical User Interface (GUI) for a variety of GEMPAK applications. This software should be downloaded and installed separately from GEMPAK once the GEMPAK installation is complete. The ActiveGEM Homepage provides additional details, documentation and source code for the package.

    The GEMPAK SpeedPatch

    NOTE: The GEMPAK SpeedPatch is no longer required under GEMPAK 5.9.x and newer

    The functionality of the SpeedPatch has been integrated into GEMPAK starting sometime around version 5.9.1. Attempting to apply the SpeedPatch to newer versions of GEMPAK will just result in a long list of failed patches, so I don't suggest trying it since it won't do you any good in any case - the optimizations are already there it the base distribution. This information about the speed patch is kept for refrence, and for people who haven't upgraded their GEMPAK installations for a long time.

    The GEMPAK speedPatch is under development by MRG Interactive. Its application to the GEMPAK source code prior to building results in significantly enhanced performance, particularly on faster machines where the processing time is negligible compared to the communcation time. The installation of the patch is simple:

    1. Download the latest version of GEMPAK if you have not done so already. (To patch an existing GEMPAK build, run make distclean in the $NAWIPS directory before applying the patch and skip to step 5.)
    2. Decompress and untar GEMPAK using gunzip -c GEMPAKTARFILE | tar -xf - (where GEMPAKTARFILE is replaced by the name of your GEMPAK tarball.
    3. Get into the GEMPAK root directory (usually cd GEMPAK* will work).
    4. Modify the Gemenviron file as described in the GEMPAK installation guide.
    5. Download the speedPatch - last update 30 June 2005, version 2.1.2 (you may have to right-click to save the file to disk instead of opening it in your browser).
    6. Save (or move) speedPatch.tar.gz to the $NAWIPS (root) directory of your GEMPAK distribution (preferrably 5.8.1 or later).
    7. Run gunzip -c speedPatch.tar.gz | tar -xf - to unpack the speedPatch.
    8. Run ksh to apply the speedPatch.
    9. Check the speedPatch.log file to ensure that the patch was correctly applied (you should see patch reports and permission mask changes - possible reasons for errors include: lack of write permission [obtain permissions], old implementation of patch [update your patch utility]).
    10. Carry on with the standard GEMPAK build procedure as outlined in the GEMPAK documentation
    11. Update any software that depends on the GEMPAK libraries (the ActiveGEM package, for example)

    The speedPatch is still under development by MRG Interactive; we therefore ask that you assist us with the debugging of the speedPatch. Unknown issues may occur on operating systems and platforms to which we do not have access and may (in severe cases) lead to compilation errors (usually, but not always, referencing undefined symbols such as nanosleep or usleep). If you encounter a compilation error that you feel may result from the application of the speedPatch, please email us with detailed information including:

    If you encounter possible speedPatch-related errors at runtime, the same request holds. In this case, however, please try to reproduce the error and provide us with a list of instructions for generating the exception (e.g. stand on one leg and hop around with your mouse in your left hand and a stapler in your right while running gdplot2). This will give MRG Interactive a better chance at troubleshooting the reported problem.

    Updating speedPatch

    The latest version of speedPatch is available here. If you have already applied the speedPatch and wish to update to a more recent version because of one of the bug / issues listed below, the procedure is fairly simple. The first step is to remove the originally-applied patch, following by the standard procedure for installing the latest speedPatch.

    1. In the $NAWIPS directory from which you applied the original speedPatch, run patch -p 0 -R < speedPatch.diff to reverse the original patch application (if you didn't keep the original speedPatch, your best bet will be to download a new version of GEMPAK).
    2. Run make distclean to fully clean the GEMPAK distribution.
    3. Proceed to Step 5 in the installation list above.

    The Gory Details

    For those of you masochists who really want to know what the speedpatch does, here are the gory details (for more information concerning changes to particular files, refer to the ChangeLog installed during the patching procedure). The speedPatch implements two new features. The first is the use of semaphore locks on the message queues -- this allows us to make quick checks on the existence and status of the queues without necessarily going through the brute-force detection of whether or not a child connected across a queue is alive. The only time that a handshake is required is when a locked queue is detected -- this protects against those evil users who close their X-Windows improperly. The second new feature [the use of nanosleep() or usleep() - depending on the system -instead of sleep()] allows this handshake to happen very quickly. The only time that a user will end up waiting for a full second will be if they improperly terminated one of the children (either gplt or - more likely - device). The device event handler repeat interval is now also controlled using the sub-second system calls - almost all of the waiting during the handshake was required because of the slow repeats of the device event handler (ie you could actually wait a full second before hearing back from a fully-functional device child). The cleanup from gpend is quick and effective, removing all semaphores and message queues regardless of dead children etc. More information about specific changes to the functions and headers is contained in the $NAWIPS/ChangeLog file.

    Known Issues and Fixes

    30 June 2005: Chuck Alonge reported a compilation bug in the speedPatch. On machines without the semun union, an error in the speedPatch resulted in a compile-time error message csproc.c:42: error: storage size of `arg' isn't known. The error stemmed from the use of the semun union in csproc.c instead of the locally-defined (in gemsys.h) gemsemun union, designed specifically for systems without, or with a broken, native semun. If no compilation issues were experienced following speedPatch application, then there is no need to update the patch on your system. A new release of the speedPatch (v2.1.2) has been made to include a fix for this problem (released 30 June 2005).

    22 March 2005: Kevin Tyle has completed modifications to the GEMPAK cleanup utility. Issuing the cleanup command now removes all core files, message queues and semaphores assigned to the user. A new release of the speedPatch (v2.0.3) has been made to include this additional feature (released 22 March 2005).

    25 February 2005: An error has been detected in all versions of the speedPatch below version 2.0.2 (issued today). This error prevents the device driver event handler from properly calling the system sub-second sleep function and results in a looping condition. This 5A condition imposes an unnecessarily large load on the system and has been corrected by removing an unnecessary conditional wrapper around the system sleep call. To check if your version of the speedPatch incorrectly calls the event handler sleep interval, use the XW device and check your system statistics (ie using the top command) with the GEMPAK X-window active. If the xw process is consuming appreciable resources (given other loads on the system), then you will probably want to update your speedPatch to the latest version.

    18 February 2005: During testing of the original speedpatch, Kevin Tyle noted that the patched version of GEMPAK was non-functional for processes not connected to a terminal (for example, cron jobs). This is a result of the GEMPAK system library's key generation for setting the message queue and semaphore IDs. If the process is connected to a tty device, then the window name is used to obtain a unique key; however, non-tty devices use the parent process id number combined with an internal constant (obviously different for each of the gplt and device children). For non-tty processes, this key information is passed to and from the child process on temporary queues; as soon as the data is read, the queue is eliminated. However, the semaphores need to be able to independtly access this information after the initiation of the child in order to obtain a lock on the appropriate message queue. A forwarding system was implemented to ensure that adequate key information reaches the child process semaphore 'grabs'. In order to allow the destruction of the temporary queues immediatly upon locking of the main message queues, the semaphore ID number is now known to the child so that the semaphore 'free' process is reduced to a trivial operation. These modifications allow new (18 February 2005 and later) versions of the speedpatch to work for processes whether or not they are connected to a terminal.

    Grib Table Creation

    Grib formatted data stores field identifiers as ID numbers rather than as names. These IDs must be translated to GEMPAK names as the data is decoded from grib to GEMPAK format (usually using dcgrib2). The grib to GEMPAK conversion tables are contained in the $GEMTBL/grid subdirectory of the GEMPAK distribution. Each grib file contains information about which tables were used to create it. As dcgrib2 runs interactively, it alerts the user as to the tables loaded for ID translations. The utility may or may not complain about being able to find the listed grib tables. You should always check the logs/dcgrib2.log file to verify that the necessary tables were found. Any grids listed in the log file have not been decoded into GEMPAK format.

    The creation of new decoding tables can be tiresome and problematic. Spacing in the files is important, as is the distinction between /t and /s separators (tabs versus spaces). The tblconvert utility allows for the rapid creation of new GEMPAK grib tables from existing descriptions in incompatible formats. Download (left shift and click) and install tblconvert by copying it either to your local directory or to your $HOME/bin directory (don't forget to rehash your environment following the installation). Typing tblconvert -h will display a list of options available for running tblconvert. For decoding an NCL-formatted table (commonly distributed by NCEP) named ncl-table131.txt, simply type tblconvert -i ncl-table131.txt >ncepgrib131.tbl. Of course, the name following the output redirect can be anything you want based on the input file name. Note that tblconvert changes the file formatting, but does not modify the paramter names or grib IDs. This is, of course, done on purpose, but be sure to check that the fields in the new table file have the GEMPAK names (for example, TMPK for temperature in Kelvin).

    Degribbing NARR Data

    The North American Regional Reanalysis (NARR) is archived on a Northern Lambert Conformal grid as a grib dataset. Information and links concerning the NARR dataset are available on the MRG Interactive NARR Usage Notes page. The dcgrib2 utility provided with GEMPAK can be used to convert grib data to GEMPAK format; however, the grib parameter table used to create the NARR archive dataset (WMO Grib Table 2 Version 131) is not included with the standard GEMPAK distribution (at least not in 5.7.4 or before). It is not sufficient simply to copy the Version 130 table from $GEMTBL/grid/wmogrib130.tbl to your local working directory before running dcgrib2 since grib parameter ID's 34 and 35 will be incorrectly decoded as UREL and VREL instead of UWND and VWND, respectively. The result of incorrect Version 130 decoding is that all GEMPAK applications will rotate the grid-relative UREL and VREL to north-relative before plotting. The effects of this rotation are especially noticable at upper levels near the borders of the archive grid when winds are superposed on geopotential heights or streamfunctions.

    Mike Dickinson (Accurate Environmental Forecasting, Inc.) has pointed out that NCEP Grib Parameter Table 2 Version 131 does not exist either in the current (v5.7.4) GEMPAK distribution. This table is required for decoding grids with Grib ID's greater than 127 (the end of the WMO Grib Table 2). In particular, some surface and three dimensional microphysical fields are not decoded without NCEP Table 2 Version 131. A GEMPAK-ready version of the table is available here. This table was generated from the NCL-formatted text file by the tblconvert utility described earlier on this page.

    The decoding of NARR data is simple if you follow a few steps:

    1. Copy WMO Grib Table 2 Version 130 to your working directory and rename it as Version 131 using a command like cp $GEMTBL/grid/wmogrib130.tbl ./wmogrib131.tbl
    2. Edit the entries for grib ID's 034 and 035 (lines 41 and 42 of the new file) to replace UREL and VREL with UWND and VWND. The result should look something like: 033 u-component of wind m s**-1 UWND 0 -9999.00 0 0
      034 v-component of wind m s**-1 VWND 0 -9999.00 0 0
    3. Download a copy of NCEP Grib Table 2 Version 131 to your working directory (left shift and click)
    4. Run dcgrib2 using a command like dcgrib2 narrFileName.gem < narrFileName.grb

    Horizontal Grid-to-Grid Interpolation Bit Packing

    Download the minimum bit-packing patch (generated for GEMPAK v5.10.2) or read on for details.

    Bilinear interpolation from one grid to another is accomplished using the Unidata-supplied program gdbiint. A subrpogram used by this utility (called gdinterp), however, does its own determination of the bit packing to be used for output, rather than relying on the GPACK parameter. This can lead to problems of extreme "noise" in the output interpolated field, as shown in this example of an upper-level temperature field. This problem is, of course, even more evident in fields that involve derivatives of grids that lack the necessary precision (e.g. temperature advection).

    Click on image to zoom

    In this case, only 6 bits were used to save the temperatures, resulting in a precision of only 0.25 K, which is clearly insufficient to accurately represent the field. The solution to this problem is to apply a minimum value to the number of bits used to hold values (for example, 16). This ensures that all fields hold at least a minimal precision; however, it can increase the size of the resulting file by the order of 20%, depending on which fields, and how many, are present. The result of this modification to the gdbiint for the poorly-represented temperature field is shown here.

    Click on image to zoom

    In order to modify the gdbiint program, download the patch (created for GEMPAK v5.10.2 - left shift and click, if necessary) provided here, and save it into your $GEMPAK directory, then run the following commands (note: these commands assume that the GEMPAK distribution has already been installed on your machine and that you have sufficient permissions to install the executables): cd $GEMPAK
    patch -p 0 -b -z orig make install
    You can check that the new copy of gdbiint is accessible by running the command ls -lL `which gdbiint`. This command should return a file whose modification date/time matches (roughly) the current time. The minimum number of bits used to store the output from the interpolation will now be 16 any time this program is run.

    General GEMPAK Notes

    Profile Plotting

    The x:y axis ratio of SKEW-T plots cannot be changed using the ratio parameter of the PTYPE function in the gdprof program. The gplt process (sub $GEMPAK/source/gplt/graph/gsgraf.f) detects the SKEW-T reqest and send the plot control to the library sub $GEMPAK/source/gemlib/gg/ggskew.f. In this sub, the ratio is computed automatically based on the size of the available plot window. Any forced modification of the ratio in the ggskew.f sub will result in a log plot rather than a SKEW-T.

    MRG Interactive is a supporter of open source initiatives and, Logo

    Questions or Comments? Contact our WebMaster.