MRG Interactive Developments

Mesoscale Research Group, McGill/SUNY/UQAM


User Instructions for Running the GEMCLIM Model on Marvin



General Information

This section contains general information about the UQAM cluster computing and diagnostics environment that will likely be of use to you both during this tutorial and day-to-day use of the machines.

IP Addresses and Hosts

The "internal" IP addresses listed in the table below are accessible only from inside the Ouranos network (this building). For communication outside the building (including communication with Marvin, which physically resides in the UQAM computer facility), or from outside into this building, you will need to use the "external" IP addresses given here. Note that the firewall only allows selected machines to connect to network systems (Zaphod), so you will only be able to contact Zaphod at its external IP from Marvin.

Machine DescriptionIP AddressAddress TypeHost NameUnnecessary Image
Cluster132.208.132.89externalmarvin
Diagnostics/Storage192.168.0.30localzaphod
207.162.87.210externalzaphod

Data Storage on Marvin and Zaphod

All of the disks on Marvin are RAID-enabled. This reduces the chances of data loss in the event of a disk crash; however, it also makes disk space more valuable. The first level storage on the cluster is a set of FibreChannel disks mounted at /local/fiber1. This 1.7T RAID has the fastest I/O access on Marvin, and should be used only at runtime by the model. No final storage of data should be done on /local/fiber1. The second level storage on Marvin is a large (8T) RAID comprised of SATA disks mounted at /local/sata1. This is where the model output will appear following post-processing and the diagnostics, and it is from here that data should be transferred as soon as possible to longer-term storage at Ouranos.

Zaphod has a pair of non-RAID 500G disks for temporary storage of data during the postprocessing of model simulation results. There is no redundancy on these disks, so data there is not as secure as it is on the Ouranos backup systems.

MachineMount PointSize (G)Purpose
Marvin/local/fiber11700Runtime Storage (Level 1)
/local/sata18300Temporary Output Storage (Level 2)
/local/sata25500Temporary Output Storage (Level 2)
Zaphod/disk1440Temporary Diagnostics Storage (Level 2)
/disk2480Temporary Diagnostics Storage (Level 2)
CronosTape RobotUnlimitedFinal Storage (Level 3)

Changing Passwords on Marvin

The username and password information on Marvin (the cluster) uses a Network Information System to distribute information across nodes. This means that running passwd to change your password will not produce the desired result. To change your password run the command: yppasswd

Customizing the Environment

Your environment can be customized to your preferences (for example, choice of compilers and sets of aliases) using the ~/.profile_usr file that resides in your home directory. The standard Unix method for customizing the environment for Borne shells and derivatives is to modify the ~/.profile file; however, the RPN Library userps this file, and in turn "sources" the ~/.profile_usr file. Modify this file exactly as you would the ~/.profile file, and you can customize your environment as you see fit.

Visualization and Diagnostics

Some tools and utilties are very useful for processing the RPN "standard" files (fichiers standard). The "collaboration" site is password protected: both the user name and password are "science".

FunctionalityHelp and Support
List file contentsvoir
Display dataxrec
Pack/unpack archivecmcarc -help [very similar to Unix tar]
Modify file contentseditfst
Simple diagnosticsr.diag
FORTRAN/C interfaceFSTD Functions
Compiling coder.compile
Linking coder.build.

Additional Information

There is lots more information about the GEM Climate model online, most of it behind the password-protected RPN site. The login name for this site is "science" and the password is "science".

DescriptionLink
Complete description of GEM ClimateRPN GEM Climate Page
Description of GEM configurationConfiguration Notes

User Setup on Marvin

NOTE: Skip this section and go directly to the tutorial instructions if your account has already been created for the UQAM tutorial. In this section, we run through an example of creating a user account on Marvin, the UQAM cluster. The user name will be foo, and the user ID will be 9999. Both of these values should be replaced when creating an account on the cluster.

  1. [root] Check user ID on rhea.ouranos.ca using (replace the user name foo with the requested user name): ssh rhea.ouranos.ca grep foo /etc/passwd | awk 'BEGIN {FS=":"}{print $3}'
  2. [root] Add user account (replace the user name foo and id 9999 with the real user name and the result of the previous command, repectively): adduser -u 9999 -g 100 foo
  3. [root] Remove the default password from the account (replacing foo): perl -pi -e 's/^(foo:)x(:)/$1$2/o' /etc/passwd
  4. [root] Log in as the user (replacing foo): su -l foo
  5. [user] Have the user change his/her password: passwd
  6. [user] Log off user account: exit
  7. [root] Update NIS account and password information: cd /var/yp; make; cd -
  8. [root] Log in as the user (replacing foo): su -l foo
  9. [user] Set up password-free secure login for nodes (see the local SSH Page for more information): mkdir ~/.ssh; cd ~/.ssh; ssh-keygen -t rsa; cat id_rsa.pub >>authorized_keys; cd -
  10. [user] Check online at the SSM Installation Guide and follow the instructions under the User Accounts section.
  11. [user] Copy standard user setup into the user's home: cp /etc/.profile_usr .
  12. [user] (Optional) Open up user home to group: cd /home; chmod g+rx ${USER}; cd -
  13. [user] Log out of the user account: exit
  14. [user] Have the user log in an check his/her account on another window.
  15. [root] From your root window, create a space for the user on a free disk (for example, /local/sata1 for user foo): cd /local/sata1; mkdir foo; chown foo.users foo; cd -
  16. [user] Instruct the user to follow the COMM Model Running Instructions, with the $GEMSSM environment variable in step 1 set to (or replaced with) ~armnlib/ssm_gemclim_3.2.2, ignoring the permission error that will occur during step 1, and skipping the path modification in step 4 that has already been done. A detailed example of running the GEMCLIM benchmark is described below, but at least steps 1-4 of the COMM Model Running Instructions should be followed before continuing.

Tutorial Instructions

For the UQAM tutorial, your account setup has already been done. Only a few steps need to be taken before you can start running the model.

  1. Log in on Marvin: ssh -X ${USER}@132.208.132.89
  2. Change your password (the default password used during account creation will be given to you during the tutorial: yppasswd

Sample GEMCLIM Session

In this section, we will help you to set up and run a benchmark simulation with the GEM climate (GEMCLIM) model. If you need additional support for running the model, see the RPN COMM community modelling website or the SSM installation guide. This example assumes that you have already completed the basic account setup steps described above.

Data from this benchmark simulation will be stored in /local/sata1/${USER}/model_storage/final/bcmk. This data should be transferred to Zaphod for viewing, since only a rudimentray set of tools are available on the cluster.

  1. Make and enter a benchmark experiment directory: mkdir -p /local/sata1/${USER}/gemclim/v3.2.2/bcmk; cd /local/sata1/${USER}/gemclim/v3.2.2/bcmk
  2. Set up an active experiment: ouv_exp base -RCSPATH "/home/armnlib/ssm_gemclim_3.2.2/all/share/modeles/GEMCLIM/v_3.2.2/RCS_DYN /home/armnlib/ssm_gemclim_3.2.2/all/share/modeles/GEMCLIM/v_3.2.2/RCS_PHY /home/armnlib/ssm_gemclim_3.2.2/all/share/modeles/PHY/CLASS/v_3.0/RCS"
  3. Build a makefile: r.make_exp
  4. Create a build directory: mkdir malib${EC_ARCH}
  5. Build GEMCLIM: make gemclim
  6. If you wish to make modifications to the GEMCLIM source code, see the expanded description below.
  7. Build and enter a configuration directory: mkdir -p configs/lam; cd configs/lam
  8. Copy the benchmark configuration scripts into the current directory: cp /local/sata1/data/configs/gemclim_3.2.2-lam/* .
  9. [First time only]Create a model listing directory for runtime text output: mkdir -p $HOME/listings/headnode
  10. [First time only]Create a model runtime directory
  11. mkdir -p /local/fiber1/${USER}/runtime; cd ~; ln -s /local/fiber1/${USER}/runtime ./gemclim; cd -
  12. Replace the model configuration user name with your own user name: perl -pi -e 's/FOO/$ENV{USER}/g' configexp.dot.cfg
  13. Launch the model run: Um_launch

Expanded Description of Modifications

This section describes how to make a basic modification to the model by modifying the source code. It is assumed that you have completed both the account setup and benchmark run described above and that you are comfortable with both Linux and FORTRAN.

  1. Return to your experiment directory: cd /local/sata1/${USER}/gemclim/v3.2.2/bcmk
  2. Get an RCS repository listing for the dynamics segment of the model: ls $MODEL_PATH/RCS_DYN
  3. Get an RCS repository listing for the physics segment of the model: ls $MODEL_PATH/RCS_PHY
  4. Extract the radiation scheme souce from the physics RCS repository: omd_exp newrad4.ftn
  5. Use your favourite text editor to have a look at (and modify if you'd like) the source for newrad4.ftn.
  6. Compile the modified (or unmodified) subprogram: make newrad4.o
  7. (Optional) Directory cleanup: make clean
  8. Recompile the model to use your newly-compiled subprogram: make gemclim
  9. Return to your benchmark configuration directory: cd configs/lam
  10. Continue as described in the sample description above.


MRG Interactive is a supporter of open source initiatives and,

SourceForge.net Logo

Questions or Comments? Contact our WebMaster.