Shell Scripting

*ATM350*

*Spring 2016*

Note: all scripts referred to here can be found in /spare11/atm350/ktyle/week4

Introduction

User interaction with the UNIX environment takes place within a "shell"; in our case, we use a version of the "C-Shell" called "tcsh".

Previously, we have run a variety of commands (e.g., chmod, cp, weather), sequentially.

Now let's say you always wanted to run those commands, day in and day out, say to create a weather briefing for your assigned site. Obviously it would be quite a bit annoying having to type them over and over again.

So what you would do is create what's called a shell script that would contain each of those commands, one after another. Instead of having to type them in each time, you would have a much more efficient means of generating the files: you would just run the shell script on the command line as if it were a program (which, for practical purposes, it is!).

A shell script, at its simplest, consists of a single line at its top which tells the computer what UNIX shell the script is written for, followed by the individual commands you are running sequentially.

Let's take a look at a simple script that runs two instances of "weather" and displays the output to the screen.

Example 1:

wxscript5.csh

Let's create a file using a text editor of our choice, and name it "wxscript1.csh".

Note that we are using ".csh" as the file suffix, indicating that it is a type of "C shell script". This is totally optional; you could name the file whatever you want. Remember, UNIX does not "care" what you name a file. But it is helpful to have a naming convention that gives the user an idea of what a file might contain (e.g., .doc for a Word document, .txt for a plain-text file, etc.).

I

The contents of the file appear in between the "dashes":

wxscript1.csh

-------------------------------------

#!/bin/tcsh

weather -c flatmetar alb 12

weather -c foredis aly l

exit 0

-------------------------------------

Save the file to your "week4" atm350 directory (create and cd into that directory if it does not exist). Before we run it, let's take a look at each line. The middle two lines are the actual "meat and potatoes" of the script; i.e., they are actually running the programs to produce the output you are interested in!

The first line looks a bit strange. It starts with a "#!" string, followed by something that looks like a path to a file, "/bin/tcsh".

It turns out that in all UNIX shell scripts, and in a fair amount of other scripting languages, any line that begins with a "pound sign" ("#") is a "comment" line; i.e., a line that is included only for reference/documentation, and is not actually "run" by the computer.

... HOWEVER, IN THIS ONE SPECIAL CASE . . .

that top line MUST be present, in exactly the above form, to "tell" the computer that this file is to be run as a script, using the shell found in "/bin/tcsh".

ANY OTHER PLACE, the "#" character at the beginning of a line signifies a "comment" line!!

The next two lines are just two instances of the "weather" program.

The final line tells the script that it has reached the end and can now "exit". The "0" in UNIX signifies a "normal" exit.

!!!NOTE!!!

It turns out that this last "exit" line is not technically necessary, but it is good form to end your scripts that way, and that is what will be required in any scripts that you write in class!

Let's run the script!

First, do an "ls -l" in your current working directory to verify that the script is there.

You should see something like this appear in the output:

-rw-r--r-- 1 ktyle wheel 82 2009-01-28 22:55 wxscript1.csh

Now let's try to run it. Since it is a script, one might think all you need to do is just type the name of the file on the command line, viz:

% wxscript1.csh

What happens when you run it?

You will probably see a not-so-helpful message to the effect of:

wxscript1.csh: Permission denied.

What happened? Let's take a look again at the detailed file info from ls -l, paying attention to the first (left-most) field:

-rw-r--r-- 1 ktyle wheel 82 2009-01-28 22:55 wxscript1.cshwxscript5.csh

Remember, the first field in a long "ls" listing has to do with file permissions. In this case, we see that the file we have created has the default attributes of any file we create on the system: it is readable and writable by the owner (ktyle), and readable only by the group and world. What's missing?

Right!! It is not executable. Without the file being marked as "executable", we cannot run it by merely typing it on the command line, as we do for "ls", "weather", and other UNIX commands.

We need to use our old friend, chmod, to change the file permission rights.

Remember the "binary" method for assigning file permissions: here, it is like we are flipping these three "bits" on and off as we proceed from right to left:

BIT READ WRITE EXECUTE SETTING

0: no read, no write, no execute (---)

1: no read, no write, yes execute (--x)

2: no read, yes write, no execute (-w-)

3: no read, yes write, yes execute (-wx)wxscript5.csh

4: yes read, no write, no execute (r--)

5: yes read, no write, yes execute (r-x)

6: yes read, yes write, no execute (rw-)

7: yes read, yes write, yes execute (rwx)

So, right now, our script has a permission of 644. To change it to make it executable by everyone, we would therefore type:

chmod 755 wxscript1.csh

Another way to do it instead of using the numerical convention is to simply type the following:

chmod +x wxscript1.csh

This tells the system to add execute permission to everyone.

Not to get too bogged down with chmod, but let's say we wanted to make the file executable only by ourself (user) and the group we are in, and leave it as only readable by everyone else (others):

Method 1 (numerical): chmod 754 wxscript1.csh

Method 2 (alphabetical): chmod ug+x,o-x wxscript1.csh

Now that we've made our program executable, all we need to do now is just type it again and this time we should see first our twelve hours of METARs and the most recent FOREcastDIScussion.

Let's now redirect the output to a file.

wxscript1.csh > wxscript1.out

View the file with your favorite text editor or simply with more to verify that it looks okay.

Next Step: add some more to the script!

In my directory, /spare11/atm350/ktyle/week4, there is a file called "wxscript2.csh". Copy it to your ATM350 directory.

Note that the permissions show that it is an executable file! When you copy a file, the copy will preserve the original permissions of its source!

Let's look at the contents of the script. It contains some comment lines, and it also introduces a new shell command, "echo". Basically "echo" will print out verbatim what ever follows it on the line.

wxscript2.csh

---------------------------------------

#!/bin/tcsh

# The above line is mandatory to start the script off.

# Any line below that that starts with a # is a comment line.

# Now let's introduce some "echo statements" to the mix.

echo "About to get some METARS"

# Let's add a blank line to make the script more readable

echo " "

weather -c flatmetar alb 12

# Another blank line

echo " "

echo "About to get the latest forecast discussion"

weather -c foredis aly l

echo " "

echo "All done!"

exit 0

---------------------------------------

Run the script and note the output. The text following each "echo" line appears in the output of the script, while the comment lines are not. These two tricks are useful ways to "document" your file and also can help pinpoint where you are in the file when things don't work as expected.

Note that text delimiting characters, such as the double-quote character, ", need to occur in pairs; at the beginning of the text following "echo" and the end. If you don't do this, the script will abort.

Copy this file to wxscript3.csh, remove one of the double-quotes, run it, and watch what happens.

Next Script Example: use of "date" and "grep" commands

"date" is a useful tool that gives you the current date and time. Just type it to see its default output:

% date

Thu Jan 29 16:30:48 UTC 2009

The date command has a lot of formatting options that are almost worth an entire lecture to learn. Here is just one quick example that will output the date and time in a GEMPAK-like format.

% date +"%y%m%d/%H%M"

090129/1600

Note that all our systems are set to UTC time, aka "Coordinated Universal, Greenwich Mean, Zulu, Z" time. i.e. "Meteorological Standard Time".

There's a lot more documentation on the date command (type info coreutils 'date invocation' for details), but here's a way to get yesterday's date in GEMPAK format:

% date +"%y%m%d/%H%M" -d yesterday

090128/1600

Now let's look at the mighty grep command:

grep literally stands for "Generalized Regular Expression Parser". That's some serious geek-talk, but for our purposes, let's use it in its simplest and most common method: to search for strings (i.e., sets of characters or words) in a file.

In my week4 directory there is a file called foredis.txt. It contains an hour's worth of forecast discussions from various NWS offices. Copy this file to your directory.

Let's search for the word "pressure" in the file: you would think that a forecaster would certainly have at least some cause to use that word in a discussion once in a while!

Type:

% grep pressure foredis.txt

You will see that you only see a new terminal prompt. Did that mean that the search didn't work, or didn't turn up any instances of "pressure"?

Well, yes: remember, like most things in UNIX, the text search string in grep is case-sensitive. Recall that foercast discussions, like almost every text product that the NWS issues, is in ALL-CAPS!

So, let's retype our command line as:

% grep PRESSURE foredis.txt

And you should see all the lines containing PRESSURE appear as your output.

Sometimes it's a pain to have to switch between the lower and upper cases. So, let's use the "-i" argument to grep, which tells it to ignore case.

% grep -i pressure foredis.txt

This should look just like the previous example.

Now, let's give grep one more argument. Instead of printing out every line that contains our desired text string, let's just have it tell us the total count of matching lines, with the "-c" argument, and let's keep the "-i" argument as well.

% grep -i -c pressure foredis.txt

You should see the count (113) and only the count, appear.

And, a final little detail pertaining to typing UNIX command lines: you can save a precious keystroke or two by combining the arguments after the dash.

% grep -ic pressure foredis.txt

This is a general rule, but not universal, since it turns out that some UNIX commands need to have arguments isolated since the arguments themselves need their own parameters.

wxscript4.csh:

Let's put what we just learned into the next version of our script. Copy wxscript4.csh from /spare11/atm350/ktyle/week4 to your directory.

Notice that it has an instance of the date command, plus a few instances of grep. Here's one particular line:

weather -c foredis aly l | grep -i -c precip

Look at the middle of the line. You see a strange character, the vertical "bar" or | character.

What is this doing? Similar to the > character that causes command or script output to be redirected to a file, the | character tells the system to take the output of the first command and use it as input to the next one. It is called the "pipe" symbol since it functions like one: the output from the first command is figuratively stuffed into one end of the pipe, and then comes out the other end where it is acted on by the second command.

This is an extremely useful shell tool that makes scripting much more efficient.

Try it with a few different search strings and see what results you get.

Setting variables within a script

The goal in programming is to make your programs and/or scripts as flexible as possible. If we developed a script to run weather to get multiple products for one site, say alb, we would have instances of the string alb occurring in many places throughout the script. If we wanted to change our site, it would be inefficient to have to change every line that contained alb to our new site. Also, one could easily miss one of the lines that contain alb.

We can take care of that by using a shell variable. Although there are various ways to set a variable in a shell script, the simplest way is to use the set command, e.g.:

set stn = alb

In order to reference the shell variable that we just declared, you preface it at any point later in the script with a "$" (dollar sign). Although not always required, it is best practice to enclose the shell variable in braces (i.e., { }). Here is how we might do so using weather:

weather -c flatmetar ${stn} 24

weather -c zone ${stn} l

When the shell runs the script, it will automatically substitute in alb when it “sees” the ${stn} in the script.

You should be able to clearly see the benefit now of using a shell variable. All we would have to do if we edited the script to pick a different site is to edit the line that sets the value of stn. So, if you wanted to get data for Buffalo, instead, you would just change one line:

set stn = buf

Of course, we can set as many shell variables as we want in a script. The example below illustrates this:

wxscript5.csh:

#!/bin/tcsh

# This script illustrates the use of setting shell variables.

# Since the METAR site and the WFO site may have different id's, we will use two shell variables.

# ---- only these two lines below need to be edited if you change sites ----

set stn = alb

set wfo = aly

# ---- only these two lines above need to be edited if you change sites ----

echo "Processing site ${stn}"

# Now we use the shell variable to get a variety of products with the weather command.

weather -c flatmetar ${stn} 12

echo " "

weather -c zone ${stn} l

echo " "

weather -c nammos ${stn} l

echo " "

weather -c newavn ${stn} l

echo " "

weather -c gfsmos ${stn} l

echo " "

weather -c foredis ${wfo} l

echo " "

weather -c climo ${wfo} l

echo " "

weather -c pfm ${wfo} l

echo " "

weather -c cf6 ${wfo} l

echo " "

echo "All done!"

exit 0

Looping

Let's introduce a new concept to our shell script, namely, the ability to cycle, or loop through a series of variables. (If you've ever taken a programming course, this will be familiar to you, although the syntax will be different). Say we want to use weather to get data for more than one site. We'll define a new shell variable via a foreach . . . end loop that will first set this variable, and then reset it to a new value each time the loop cycles. The example here will run weather three times, for three sites:

wxscript6.csh:

#!/bin/tcsh

# This script illustrates the use of setting shell variables and the technique of looping via a foreach loop.

# ---- only this line below needs to be edited if you change sites ----

foreach stn ( alb buf lga )

# ---- only these line above needs to be edited if you change sites ----

# loop begins

# For readability, it's a good idea to indent for every distinct loop.

echo "Processing site ${stn}"

weather -c flatmetar ${stn} 12

echo " "

weather -c zone ${stn} l

echo " "

weather -c nammos ${stn} l

echo " "

weather -c newavn ${stn} l

echo " "

weather -c gfsmos ${stn} l

echo " "

# loop ends

end

echo "All done!"

exit 0

Nested Looping

One can have multiple loops within a script, one within another. This technique is called "nesting". Let's say we also want to loop over the product types we can request with weather, which will also make the script a bit easier to edit. Here we'll loop over three sites, and add an inner, nested loop that will loop over several products. We'll also use file output redirection so that each site gets its own separate file, each containing all requested products:

wxscript7.csh:

#!/bin/tcsh

# This script illustrates nested loops as well as file redirection.

foreach stn ( alb buf lga )

# outer loop begins

# For readability, it's a good idea to indent for every distinct loop.

# Let's remove any existing instances of the wxbriefing output text files before we start the next loop.

rm -f wxbriefing_${stn}.txt

echo "Processing site ${stn}"

foreach prod ( flatmetar zone nammos newavnmos gfsmos )

# outer loop begins

# For readability, it's a good idea to indent for every distinct loop.

# We write out to and append the weather program output as well as the blank lines from "echo" to a file.

weather -c ${prod} ${stn} l >> wxbriefing_${stn}.txt

echo " " >> wxbriefing_${stn}.txt

# inner loop ends

end

end

echo "All done!"

exit 0

It may not look much shorter, due to all the comment lines we included, but we have done a lot more work with significantly fewer lines of code!

While Loops and If—Endif blocks

Imagine you wanted to create surface maps for all hours of a particulcar day. You could just write a really long script that explicitly set the hour with the DATTIM pararmeter in GEMPAK, but it is far easier to use what's known as a "while loop". In this example we will set a variable "hour" to 0, then enter a "while loop" where we keep running sfmap, incrementing the hour by one, until we reach the last hour of the day (23). We will also introduce the use of the "if—endif" block since if the value of the hour is less than 10, we will want to add a leading zero (although we don't really need to in this case, sometimes we may need to deal with files that use a two-digit hour in their name).

sfmap_hourloop.csh:

#!/bin/tcsh

#

# sample GEMPAK script with shell variable to select date

# illustrates use of "@" technique to perform simple integer arithmetic

# illustrates use of "while' loop

# illustrates use of "if -- end if"

#

#----------------------------------------------------------

#

# get the current date

#

set curdat = `date +%y%m%d`

# echo it

echo "date today is $curdat"

#

# now get yesterday's date which we will then use for input to sffile

# (because we want to have all 24 hours available!)

#

set yesdat = `date +%y%m%d --d yesterday`

echo "date yesterday was $yesdat"

#

#set hour variable

#

set hour = 0

set MAXHR = 23

#

# Create a "while" loop to start at hour 0 and continue through hour 23

#

while ($hour <= 23)

#

# If hour is less than 10, we need to "pad" it with a leading zero.

# If not, use it as is.

# We will create a new variable, "hour2" for this.

#

if ( $hour < 10 ) then

set hour2 = 0${hour}

else

set hour2 = ${hour}

endif

#

#

# We will run sfmap and then tell the shell that all lines

# between the line containing << ENDSFMAP and ENDSFMAP should

# be considered as input to sfmap.

# GEMPAK "quirk": always left-justify lines that are GEMPAK

# commands.

# We'll restore the settings from example 1 of the user guide to

# serve as a base.

# Only those settings that I need to change will appear in the

# list of parameter changes to sfmap.

# I will center my map over Chicago's O'Hare Airport (KORD)

# I will output each map as a GIF image with the naming convention of

# sfmap_ord_$hour2.gif, where "hour2" is set earlier in the script.

#

sfmap << ENDSFMAP

restore $GEMRST/sfmap_ex1

sffile = $OBS/${yesdat}.sf

area = garea

garea = ord

dattim = $hour2

dev=gf|sfmap_ord_${hour2}.gif

run

exit

ENDSFMAP

#

# Now we are back to the shell

# We need to run gpend to finalize the graphic!

# Pay carefull attention to how we ordered things!

# first we "exit" sfmap

# then the "ENDSFMAP" line tells the shell that any subsequent lines

# are to be interpreted as UNIX shell commands, not inputs to sfmap

#

gpend

#

# Now we use "@" to increment the hour

# Pay careful attention to the syntax!!

#

@ hour = $hour + 1

#

# Now we reach the end of our "while loop". The script will

# go back to the first line (while $hour <= $MAXHR); once "hour"

# exceeds the value we set (23) for "MAXHR" it will exit the loop

# and continue with the line after the next line "end".

#

end

# exit the script

echo "All done"

exit 0