Shell Scripting
*ATM350*
*Spring 2013*
Note: all scripts referred to here can be found in /spare11/atm350/ktyle/class6
Introduction
User interaction with the UNIX environment takes place within a "shell"; in our case, we use a version of the "C-Shell" called "tcsh".
Previously, we have run a variety of commands (e.g., chmod, cp, weather), sequentially.
Now let's say you always wanted to run those commands, day in and day out, say to create a weather briefing for your assigned site. Obviously it would be quite a bit annoying having to type them over and over again.
So what you would do is create what's called a shell script that would contain each of those commands, one after another. Instead of having to type them in each time, you would have a much more efficient means of generating the files: you would just run the shell script on the command line as if it were a program (which, for practical purposes, it is!).
A shell script, at its simplest, consists of a single line at its top which tells the computer what UNIX shell the script is written for, followed by the individual commands you are running sequentially.
Let's take a look at a simple script that runs two instances of "weather" and displays the output to the screen.
Example 1:
Let's create a file using a text editor of our choice, and name it "wxscript1.csh".
Note that we are using ".csh" as the file suffix, indicating that it is a type of "C shell script". This is totally optional; you could name the file whatever you want. Remember, UNIX does not "care" what you name a file. But it is helpful to have a naming convention that gives the user an idea of what a file might contain (e.g., .doc for a Word document, .txt for a plain-text file, etc.).
I
The contents of the file appear in between the "dashes":
wxscript1.csh
-------------------------------------
#!/bin/tcsh
weather -c flatmetar alb 12
weather -c foredis aly l
exit 0
-------------------------------------
Save the file to your "class6" atm350 directory. Before we run it, let's take a look at each line. The middle two lines are the actual "meat and potatoes" of the script; i.e., they are actually running the programs to produce the output you are interested in!
The first line looks a bit strange. It starts with a "#!" string, followed by something that looks like a path to a file, "/bin/tcsh".
It turns out that in all UNIX shell scripts, and in a fair amount of other scripting languages, any line that begins with a "pound sign" ("#") is a "comment" line; i.e., a line that is included only for reference/documentation, and is not actually "run" by the computer.
... HOWEVER, IN THIS ONE SPECIAL CASE . . .
that top line MUST be present, in exactly the above form, to "tell" the computer that this file is to be run as a script, using the shell found in "/bin/tcsh".
ANY OTHER PLACE, the "#" character at the beginning of a line signifies a "comment" line!!
The next two lines are just two instances of the "weather" program.
The final line tells the script that it has reached the end and can now "exit". The "0" in UNIX signifies a "normal" exit.
!!!NOTE!!!
It turns out that this last "exit" line is not technically necessary, but it is good form to end your scripts that way, and that is what will be required in any scripts that you write in class!
Let's run the script!
First, do an "ls -l" in your current working directory to verify that the script is there.
You should see something like this appear in the output:
-rw-r--r-- 1 ktyle wheel 82 2009-01-28 22:55 wxscript1.csh
Now let's try to run it. Since it is a script, one might think all you need to do is just type the name of the file on the command line, viz:
% wxscript1.csh
What happens when you run it?
You will probably see a not-so-helpful message to the effect of:
wxscript1.csh: Permission denied.
What happened? Let's take a look again at the detailed file info from ls -l, paying attention to the first (left-most) field:
-rw-r--r-- 1 ktyle wheel 82 2009-01-28 22:55 wxscript1.csh
Remember, the first field in a long "ls" listing has to do with file permissions. In this case, we see that the file we have created has the default attributes of any file we create on the system: it is readable and writable by the owner (ktyle), and readable only by the group and world. What's missing?
Right!! It is not executable. Without the file being marked as "executable", we cannot run it by merely typing it on the command line, as we do for "ls", "weather", and other UNIX commands.
We need to use our old friend, chmod, to change the file permission rights.
Remember the "binary" method for assigning file permissions: here, it is like we are flipping these three "bits" on and off as we proceed from right to left:
BIT READ WRITE EXECUTE SETTING
0: no read, no write, no execute (---)
1: no read, no write, yes execute (--x)
2: no read, yes write, no execute (-w-)
3: no read, yes write, yes execute (-wx)
4: yes read, no write, no execute (r--)
5: yes read, no write, yes execute (r-x)
6: yes read, yes write, no execute (rw-)
7: yes read, yes write, yes execute (rwx)
So, right now, our script has a permission of 644. To change it to make it executable by everyone, we would therefore type:
chmod 755 wxscript1.csh
Another way to do it instead of using the numerical convention is to simply type the following:
chmod +x wxscript1.csh
This tells the system to add execute permission to everyone.
Not to get too bogged down with chmod, but let's say we wanted to make the file executable only by ourself (user) and the group we are in, and leave it as only readable by everyone else (others):
Method 1 (numerical): chmod 754 wxscript1.csh
Method 2 (alphabetical): chmod ug+x,o-x wxscript1.csh
Now that we've made our program executable, all we need to do now is just type it again and this time we should see first our twelve hours of METARs and the most recent FOREcastDIScussion.
Let's now redirect the output to a file.
wxscript1.csh > wxscript1.out
View the file with your favorite text editor or simply with more to verify that it looks okay.
Next Step: add some more to the script!
In my directory, /spare11/atm350/ktyle/class6, there is a file called "wxscript2.csh". Copy it to your ATM350 directory.
Note that the permissions show that it is an executable file! When you copy a file, the copy will preserve the original permissions of its source!
Let's look at the contents of the script. It contains some comment lines, and it also introduces a new shell command, "echo". Basically "echo" will print out verbatim what ever follows it on the line.
wxscript2.csh
---------------------------------------
#!/bin/tcsh
# The above line is mandatory to start the script off.
# Any line below that that starts with a # is a comment line.
# Now let's introduce some "echo statements" to the mix.
echo "About to get some METARS"
# Let's add a blank line to make the script more readable
echo " "
weather -c flatmetar alb 12
# Another blank line
echo " "
echo "About to get the latest forecast discussion"
weather -c foredis aly l
echo " "
echo "All done!"
exit 0
---------------------------------------
Run the script and note the output. The text following each "echo" line appears in the output of the script, while the comment lines are not. These two tricks are useful ways to "document" your file and also can help pinpoint where you are in the file when things don't work as expected.
Note that text delimiting characters, such as the double-quote character, ", need to occur in pairs; at the beginning of the text following "echo" and the end. If you don't do this, the script will abort.
Copy this file to wxscript3.csh, remove one of the double-quotes, run it, and watch what happens.
Next Script Example: use of "date" and "grep" commands
"date" is a useful tool that gives you the current date and time. Just type it to see its default output:
% date
Thu Jan 29 16:30:48 UTC 2009
The date command has a lot of formatting options that are almost worth an entire lecture to learn. Here is just one quick example that will output the date and time in a GEMPAK-like format.
% date +"%y%m%d/%H%M"
090129/1600
Note that all our systems are set to UTC time, aka "Coordinated Universal, Greenwich Mean, Zulu, Z" time. i.e. "Meteorological Standard Time".
There's a lot more documentation on the date command (type info coreutils 'date invocation' for details), but here's a way to get yesterday's date in GEMPAK format:
% date +"%y%m%d/%H%M" -d yesterday
090128/1600
Now let's look at the mighty grep command:
grep literally stands for "Generalized Regular Expression Parser". That's some serious geek-talk, but for our purposes, let's use it in its simplest and most common method: to search for strings (i.e., sets of characters or words) in a file.
In my class6 directory there is a file called foredis.txt. It contains an hour's worth of forecast discussions from various NWS offices. Copy this file to your directory.
Let's search for the word "pressure" in the file: you would think that a forecaster would certainly have at least some cause to use that word in a discussion once in a while!
Type:
% grep pressure foredis.txt
You will see that you only see a new terminal prompt. Did that mean that the search didn't work, or didn't turn up any instances of "pressure"?
Well, yes: remember, like most things in UNIX, the text search string in grep is case-sensitive. Recall that foercast discussions, like almost every text product that the NWS issues, is in ALL-CAPS!
So, let's retype our command line as:
% grep PRESSURE foredis.txt
And you should see all the lines containing PRESSURE appear as your output.
Sometimes it's a pain to have to switch between the lower and upper cases. So, let's use the "-i" argument to grep, which tells it to ignore case.
% grep -i pressure foredis.txt
This should look just like the previous example.
Now, let's give grep one more argument. Instead of printing out every line that contains our desired text string, let's just have it tell us the total count of matching lines, with the "-c" argument, and let's keep the "-i" argument as well.
% grep -i -c pressure foredis.txt
You should see the count (113) and only the count, appear.
And, a final little detail pertaining to typing UNIX command lines: you can save a precious keystroke or two by combining the arguments after the dash.
% grep -ic pressure foredis.txt
This is a general rule, but not universal, since it turns out that some UNIX commands need to have arguments isolated since the arguments themselves need their own parameters.
wxscript4.csh:
Let's put what we just learned into the next version of our script. Copy wxscript4.csh from /spare11/atm350/ktyle/class6 to your directory.
Notice that it has an instance of the date command, plus a few instances of grep. Here's one particular line:
weather -c foredis aly l | grep -i -c precip
Look at the middle of the line. You see a strange character, the vertical "bar" or | character.
What is this doing? Similar to the > character that causes command or script output to be redirected to a file, the | character tells the system to take the output of the first command and use it as input to the next one. It is called the "pipe" symbol since it functions like one: the output from the first command is figuratively stuffed into one end of the pipe, and then comes out the other end where it is acted on by the second command.
This is an extremely useful shell tool that makes scripting much more efficient.
Try it with a few different search strings and see what results you get.