if [[ -s $OUTFILE ]] then echo “\nFull Filesystem(s) on $THISHOST\n” cat $OUTFILE print fi Listing 5.10 fs_mon_AIX_MB_FREE_excep.ksh shell script. (continued) The script in Listing 5.10 is good, and we have covered all of the bases, right? If you want to stop here, you will be left with an incomplete picture of what we can accom- plish. There are several more things to consider, and, of course, there are many more ways to do any of these tasks, and no one is correct. Let’s consider mixing the filesys- tem percentage used and the MB of free filesystem space techniques. With a mecha- nism to auto-detect the way we select the usage, the filesystem monitoring script could be a much more robust tool—and a must-have tool where you have a mix of regular and large filesystems to monitor. Percentage Used—MB Free and Large Filesystems Now we’re talking! Even if most of your filesystems are large file enabled or are just huge in size, the small ones will still kill you in the end. For a combination of small and large filesystems, we need a mix of both the percent used and MB of free space tech- niques. For this combination to work, we need a way to auto-detect the correct usage, which we still need to define. There are different combinations of these auto-detect techniques that can make the monitoring work differently. For the large filesystems we want to use the MB of free space, and for regular filesystems we use the percentage method. We need to define a trigger that allows for this free space versus percentage moni- toring transformation. The trigger value will vary by environment, but this example uses 1GB as the transition point from percentage used to MB of free space. Of course, the value should be more like 4–6GB, but we need an example. We also need to con- sider how the $EXCEPTIONS file is going to look. Options for the exceptions file are a combined file or two separate files, one for percentage used and one for MB free. The obvious choice is one combined file. What are combined entries to look like? How are we going to handle the wrong entry type? The entries need to conform to the specific test type the script is looking for. The best way to handle this is to require that either a % or MB be added as a suffix to each new entry in the exceptions file. With the MB or 118 Chapter 5 % suffix we could override not only the triggering level, but also the testing method! If an entry has only a number without the suffix, then this exceptions file entry will be ignored and the shell script’s default values will be used. This suffix method is the most flexible, but it, too, is prone to mistakes in the exceptions file. For the mistakes, we need to test the entries in the exceptions to see that they conform to the standard that we have decided on. The easiest way to create this new, more robust script is to take large portions of the previous scripts and convert them into functions. We can simply insert the word function followed by a function name and enclose the code within curly braces—for example, function test_function { function_code }. Or if you prefer the C-type func- tion method, we can use this example, test_function () { function_code }. The only difference between the two function methods is one uses the word function to define the function while the other just adds a set of parentheses after the function’s name. When we use functions, it is easy to set up a logical framework from which to call the functions. It is always easiest to set up the framework first and then fill in the middle. The logic code for this script will look like Listing 5.11. load_File_System_data > $WORKFILE if EXCEPTIONS_FILE exists and is > 0 size then load_EXCEPTIONS_FILE_data fi while read $WORKFILE, which has the filesystem data do if EXCEPTIONS data was loaded then check_exceptions_file RC=Get Return code back from function case $RC in 1) Found exceeded by % method 2) Found out-of-limit by MB Free method 3) Found OK in exceptions file by a testing method 4) Not found in exceptions file esac else # No exceptions file Use script defaults to compare fi done if we have anything out of limits then display_output fi Listing 5.11 Logic code for a large and small filesystem freespace script. File System Monitoring 119 This is very straightforward and easy to do with functions. From this logical description we already have the main body of the script written. Now we just need to modify the check_exceptions function to handle both types of data and create the load_FS_data, load_EXCEPTIONS_data, and display_output functions. For this script we are also going to do things a little differently because this is a learning process. As we all know, there are many ways to accomplish the same task in Unix; shell scripting is a prime example. To make our scripts a little easier to read at a glance, we are going to change how we do numeric test comparisons. We currently use the standard bracketed test functions with the numeric operators, -lt, -le, -eq, -ne, -ge, and -gt: if [ $VAR1 -gt $VAR2 ] We are now going to use the bracketed tests for character strings only and do all of our numerical comparisons with the double parentheses method: if (( VAR1 > VAR2 )) The operators for this method are <, <=, ==, !=, >=, >. When we make this small change, it makes the script much easier to follow because we know immediately that we are dealing with either numeric data or a character string without knowing much at all about the data being tested. Notice that we did not reference the variables with a $ (dol- lar sign) for the numeric tests. The $ omission is not the only difference, but it is the most obvious. The $ is omitted because it is implied that anything that is not numeric is a variable. Other things to look for in this script are compound tests, math and math within tests, the use of curly braces with variables, ${VAR1}MB, a no-op using a : (colon), data validation, error checking, and error notification. These variables are a lot to look for, but you can learn much from studying the script shown in Listing 5.12. Just remember that all functions must be defined before they can be used! Failure to define functions is the most common mistake when working with them. The second most common mistake has to do with scope. Scope deals with where a variable and its value are known to other scripts and functions. Top level down is the best way to describe where scope lies. The basic rules say that all of a shell script’s variables are known to the internal, lower-level, functions, but none of the function’s variables are known to any higher-calling script or function, thus the top level down definition. We will cover a method called a co-process of dealing with scope in a later chapter. So, in this script the check_exceptions function will use the global script’s vari- ables, which are known to all of the functions, and the function will, in turn, reply with a return code, as we defined in the logic flow of Listing 5.11. Scope is a very important concept, as is the placement of the function in the script. The comments in this script are extensive, so please study the code and pay particular attention to the boldface text. NOTE Remember: You have to define a function before you can use it. 120 Chapter 5 #!/usr/bin/ksh # # SCRIPT: fs_mon_AIX_PC_MBFREE_excep.ksh # AUTHOR: Randy Michael # DATE: 08-22-2001 # REV: 4.3.P # PURPOSE: This script is used to monitor for full filesystems, # which is defined as “exceeding” the MAX_PERCENT value. # A message is displayed for all “full” filesystems. # # PLATFORM: AIX # # REV LIST: # Randy Michael - 08-27-2001 # Changed the code to use MB of free space instead of # the %Used method. # # Randy Michael - 08-27-2001 # Added code to allow you to override the set script default # for MIN_MB_FREE of FS Space # # Randy Michael - 08-28-2001 # Changed the code to handle both %Used and MB of Free Space. # It does an “auto-detection” but has override capability # of both the trigger level and the monitoring method using # the exceptions file pointed to by the $EXCEPTIONS variable # # set -n # Uncomment to check syntax without any execution # set -x # Uncomment to debug this script # ##### DEFINE FILES AND VARIABLES HERE #### MIN_MB_FREE=”100MB” # Min. MB of Free FS Space MAX_PERCENT=”85%” # Max. FS percentage value FSTRIGGER=”1000MB” # Trigger to switch from % Used to MB Free WORKFILE=”/tmp/df.work” # Holds filesystem data >$WORKFILE # Initialize to empty OUTFILE=”/tmp/df.outfile” # Output display file >$OUTFILE # Initialize to empty EXCEPTIONS=”/usr/local/bin/exceptions” # Override data file DATA_EXCEPTIONS=”/tmp/dfdata.out” # Exceptions file w/o # rows EXCEPT_FILE=”N” # Assume no $EXCEPTIONS FILE Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continues) File System Monitoring 121 THISHOST=`hostname` # Hostname of this machine ###### FORMAT VARIABLES HERE ###### # Both of these variables need to be multiplied by 1024 blocks (( MIN_MB_FREE = $(echo $MIN_MB_FREE | sed s/MB//g) * 1024 )) (( FSTRIGGER = $(echo $FSTRIGGER | sed s/MB//g) * 1024 )) ####### DEFINE FUNCTIONS HERE ######## function check_exceptions { # set -x # Uncomment to debug this function while read FSNAME FSLIMIT do IN_FILE=”N” # If found in file, which test type to use? # Do an NFS sanity check and get rid of any “:”. # If this is found it is actually an error entry # but we will try to resolve it. It will # work only if it is an NFS cross mount to the same # mount point on both machines. echo $FSNAME | grep ‘:’ >/dev/null \ && FSNAME=$(echo $FSNAME | cut -d ‘:’ -f2) # Check for empty and null variable if [[ ! -z “$FSLIMIT” && “$FSLIMIT” != ‘’ ]] then if [[ $FSNAME = $FSMOUNT ]] # Found it! then # Check for “MB” Characters Set IN_FILE=MB echo $FSLIMIT | grep MB >/dev/null && IN_FILE=”MB” \ && (( FSLIMIT = $(echo $FSLIMIT \ | sed s/MB//g) * 1024 )) # check for “%” Character Set IN_FILE=PC, for % echo $FSLIMIT | grep “%” >/dev/null && IN_FILE=”PC” \ && FSLIMIT=$(echo $FSLIMIT | sed s/\%//g) case $IN_FILE in MB) # Use Megabytes of free space to test # Up-case the characters, if they exist FSLIMIT=$(echo $FSLIMIT | tr ‘[a-z]’ ‘[A-Z]’) # Get rid of the “MB” if it exists FSLIMIT=$(echo $FSLIMIT | sed s/MB//g) # Test for blank and null values if [[ ! -z $FSLIMIT && $FSLIMIT != ‘’ ]] Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continued) 122 Chapter 5 then # Test for a valid filesystem “MB” limit if (( FSLIMIT >= 0 && FSLIMIT < FSSIZE )) then # Check the limit if (( FSMB_FREE < FSLIMIT )) then return 1 # Found out of limit # using MB Free method else return 3 # Found OK fi else echo “\nERROR: Invalid filesystem MAX for\ $FSMOUNT - $FSLIMIT” echo “ Exceptions file value must be\ less than or” echo “ equal to the size of the filesystem\ measured” echo “ in 1024 bytes\n” fi else echo “\nERROR: Null value specified in exceptions\ file” echo “ for the $FSMOUNT mount point.\n” fi ;; PC) # Use the Percent used method to test # Strip out the % sign if it exists PC_USED=$(echo $PC_USED | sed s/\%//g) # Test for blank and null values if [[ ! -z $FSLIMIT && $FSLIMIT != ‘’ ]] then # Test for a valid percentage, i.e. 0-100 if (( FSLIMIT >= 0 && FSLIMIT <= 100 )) then if (( PC_USED > FSLIMIT )) then return 2 # Found exceeded by % Used method else return 3 # Found OK fi else echo “\nERROR: Invalid percentage for\ $FSMOUNT - $FSLIMIT” echo “ Exceptions file values must be” echo “ between 0 and 100%\n” fi Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continues) File System Monitoring 123 else echo “\nERROR: Null value specified in exceptions” echo “ file for the $FSMOUNT mount point.\n” fi ;; N) # Test type not specified in exception file, use default # Inform the user of the exceptions file error echo “\nERROR: Missing testing type in exceptions file” echo “ for the $FSMOUNT mount point. A \”%\” or” echo “ \”MB\” must be a suffix to the numerical” echo “ entry. Using script default values \n” # Method Not Specified - Use Script Defaults if (( FSSIZE >= FSTRIGGER )) then # This is a “large” filesystem if (( FSMB_FREE < MIN_MB_FREE )) then return 1 # Found out of limit using MB Free else return 3 # Found OK fi else # This is a standard filesystem PC_USED=$(echo $PC_USED | sed s/\%//g) #Remove the % FSLIMIT=$(echo $FSLIMIT | sed s/\%//g) #Remove the % if (( PC_USED > FSLIMIT )) then return 2 # Found exceeded by % Used method else return 3 # Found OK fi fi ;; esac fi fi done < $DATA_EXCEPTIONS # Feed the loop from the bottom!!! return 4 # Not found in $EXCEPTIONS file } #################################### function display_output { if [[ -s $OUTFILE ]] then Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continued) 124 Chapter 5 echo “\nFull Filesystem(s) on $THISHOST\n” cat $OUTFILE print fi } #################################### function load_EXCEPTIONS_data { # Ignore any line that begins with a pound sign, # # and omit all blank lines cat $EXCEPTIONS | grep -v “^#” | sed /^$/d > $DATA_EXCEPTIONS } #################################### function load_FS_data { df -k | tail +2 | egrep -v ‘/dev/cd[0-9]|/proc’ \ | awk ‘{print $1, $2, $3, $4, $7}’ > $WORKFILE } #################################### ######### START OF MAIN ############ #################################### load_FS_data # Do we have a nonzero size $EXCEPTIONS file? if [[ -s $EXCEPTIONS ]] then # Found a nonempty $EXCEPTIONS file load_EXCEPTIONS_data EXCEP_FILE=”Y” fi while read FSDEVICE FSSIZE FSMB_FREE PC_USED FSMOUNT do if [[ $EXCEP_FILE = “Y” ]] then check_exceptions CE_RC=”$?” # Check Exceptions Return Code (CE_RC) case $CE_RC in Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continues) File System Monitoring 125 1) # Found exceeded in exceptions file by MB Method (( FS_FREE_OUT = FSMB_FREE / 1000 )) echo “$FSDEVICE mounted on $FSMOUNT has ${FS_FREE_OUT}MB\ Free” \ >> $OUTFILE ;; 2) # Found exceeded in exceptions file by %Used method echo “$FSDEVICE mount on $FSMOUNT is ${PC_USED}%” \ >> $OUTFILE ;; 3) # Found OK in exceptions file : # NO-OP Do Nothing ;; 4) # Not found in exceptions file - Use Script Default Triggers if (( FSSIZE >= FSTRIGGER )) then # This is a “large” filesystem # Remove the “MB”, if it exists FSMB_FREE=$(echo $FSMB_FREE | sed s/MB//g) typeset -i FSMB_FREE if (( FSMB_FREE < MIN_MB_FREE )) then (( FS_FREE_OUT = FSMB_FREE / 1000 )) echo “$FSDEVICE mounted on $FSMOUNT has\ ${FS_FREE_OUT}MB Free” >> $OUTFILE fi else # This is a standard filesystem PC_USED=$(echo $PC_USED | sed s/\%//g) MAX_PERCENT=$(echo $MAX_PERCENT | sed s/\%//g) if (( PC_USED > MAX_PERCENT )) then echo “$FSDEVICE mount on $FSMOUNT is ${PC_USED}%” \ >> $OUTFILE fi fi ;; esac else # NO $EXECPTIONS FILE USE DEFAULT TRIGGER VALUES if (( FSSIZE >= FSTRIGGER )) then # This is a “large” filesystem - Use MB Free Method FSMB_FREE=$(echo $FSMB_FREE | sed s/MB//g) # Remove the “MB” if (( FSMB_FREE < MIN_MB_FREE )) then (( FS_FREE_OUT = FSMB_FREE / 1000 )) echo “$FSDEVICE mounted on $FSMOUNT has\ Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continued) 126 Chapter 5 ${FS_FREE_OUT}MB Free” >> $OUTFILE fi else # This is a standard filesystem - Use % Used Method PC_USED=$(echo $PC_USED | sed s/\%//g) MAX_PERCENT=$(echo $MAX_PERCENT | sed s/\%//g) if (( PC_USED > MAX_PERCENT )) then echo “$FSDEVICE mount on $FSMOUNT is ${PC_USED}%” \ >> $OUTFILE fi fi fi done < $WORKFILE # Feed the while loop from the bottom!!! display_output # End of Script Listing 5.12 fs_mon_AIX_PC_MBFREE_excep.ksh shell script. (continued) In the script shown in Listing 5.12, we made tests to confirm the data’s integrity and for mistakes in the exceptions file (of course, we can go only so far with mistakes!). The reason is that we made the exceptions file more complicated to use. Two of my testers consistently had reverse logic on the MB free override option of the script by thinking greater than instead of less than. From this confusion, a new exceptions file was created that explained what the script is looking for and gave example entries. Of course, all of these lines begin with a pound sign, #, so they are ignored when data is loaded into the $DATA_EXCEPTIONS file. Listing 5.13 shows the exceptions file that worked best with the testers. # FILE: “exceptions” # # This file is used to override both the default # trigger value in the filesystem monitoring script # fs_mon_excep.ksh, but also allows overriding the # monitoring technique used, i.e. Max %Used and # minimum MB of filesystem space. The syntax to # override is a /mount-point and a trigger value. # # EXAMPLES: # # /usr 96% # Flag anything ABOVE 96% # OR Listing 5.13 Example exceptions file. (continues) File System Monitoring 127 [...]... %IUSED MOUNTED ON /dev/hd4 32 768 1 637 6 51% 16 63 11% / /dev/hd2 1212416 57592 96% 36 386 13% /usr /dev/hd9var 532 48 30 824 43% 540 5% /var /dev/hd3 106496 99 932 7% 135 1% /tmp /dev/hd1 4096 39 16 5% 25 % /home /proc /proc /dev/hd10opt 638 976 24456 97% 15457 10% /opt /dev/scripts_lv 102400 95264 7% 435 2% /scripts /dev/cd0 656756 0 100% 32 837 8 100% /cdrom File System Monitoring Table 5 .3 AIX df Output Columns... /dev/vg00/lvol3 151552 89500 58669 60% / /dev/vg00/lvol1 47829 24109 18 937 56% /stand /dev/vg00/lvol9 131 0720 860829 422 636 67% /var /dev/vg00/lvol8 972800 55 439 2 39 235 8 59% /usr File System Monitoring Table 5.8 (Continued) FILESYSTEM KBYTES USED AVAIL %USED MOUNTED ON /dev/vg 13/ lvol1 4190208 1155095 2850597 29% /u2 /dev/vg00/lvol7 102400 4284 92256 4% /tmp /dev/vg00/lvol 13 2 039 808 16640 73 352294 83% /test2... 131 132 Chapter 5 Table 5.6 SUN/Solaris df -k Command Output FILESYSTEM KBYTES USED AVAIL MOUNTED CAPACITY ON /dev/dsk/c0d0s0 1924 23 18206 154975 11% /dev/dsk/c0d0s6 1015542 488678 465 932 52% /usr /proc 0 0 0 0% /proc fd 0 0 0 0% /dev/fd mnttab 0 0 0 0% /etc/mnttab /dev/dsk/c0d0s3 96455 5 931 80879 7% /var swap 554 132 0 554 13 0% /var/run /dev/dsk/c0d0s5 47975 1221 41957 3% /opt swap 554428 296 554 132 ... Wed Jun 5 21:48:16 EDT 2002 Total MB of Paging Space: Total MB of Paging Space Used: Total MB of Paging Space Free: 33 6MB 33 MB 30 3MB Percent of Paging Space Used: 10% Percent of Paging Space Free: 90% Listing 6.1 Required paging and swap space report Before we get started creating the shell scripts, we need the command syntax for each operating system Each of the commands produces a different result,... required output for the AIX shell script Take a look at the entire shell script shown in Listing 6 .3, and pay particular attention to the boldface type #!/usr/bin/ksh # # SCRIPT: AIX_paging_mon.ksh # # AUTHOR: Randy Michael # DATE: 5 /31 /2002 # REV: 1.1.P # Listing 6 .3 AIX_paging_mon.ksh shell script listing (continues) 151 152 Chapter 6 # PLATFORM: AIX Only # # PURPOSE: This shell script is used to produce... handle as we will see in the shell scripting section for Solaris later in this chapter Creating the Shell Scripts Now that we have the basic syntax of the commands to get paging and swap space statistics, we can start our scripting of the solutions In each case you should notice which pieces of data are missing from our required output, as shown in Listing 6.1 All of these shell scripts are different... interchangeably Each of our four Unix flavors, AIX, HP-UX, Linux, and Solaris, use different commands to list the swap space usage; the output for each command and OS varies also The goal of this chapter is to create five shell scripts: one script of each of the four operating systems and an all-in-one shell script that will run on any of our four Unix flavors Each of the shell scripts must produce the... 1024-blocks Column 3 The kilobytes of free filesystem space, Free Column 4 The percentage of used capacity, %Used Column 7 The mount point of the filesystem, Mounted on Table 5.4 Linux df -k Command Output FILESYSTEM 1K-BLOCKS USED AVAILABLE USE% MOUNTED ON /dev/hda16 101089 32 949 62921 34 % / /dev/hda5 1011928 104 960420 0% /backup /dev/hda1 54416 2647 48960 5% /boot /dev/hda8 202220 13 191767 0% /download... /dev/hda8 202220 13 191767 0% /download /dev/hda9 202220 1619 190161 1% /home /dev/hda12 124427 19 117984 0% /tmp /dev/hda6 1011928 907580 52944 94% /usr — /dev/hda10 155545 36 147479 0% /usr/local /dev/hda11 124427 29670 8 833 3 25%/var Table 5.5 Linux df Output Columns of Interest DF OUTPUT COLUMNS COLUMN CONTENTS Column 1 The filesystem device name, Filesystem Column 2 The size of the filesystem in... specifies to produce a total line for a summary of all virtual memory This command output is shown here [root@dino]/> swapinfo -tm Mb Mb Mb TYPE AVAIL USED FREE dev 96 21 73 reserve 46 -46 memory 15 5 10 total 111 72 37 PCT USED 22% 33 % 65% START/ Mb LIMIT RESERVE 928768 - - 0 PRI 1 - NAME /dev/dsk/c0t6d0 147 148 Chapter 6 Notice in this output that HP-UX splits up virtual memory into three categories: . %IUSED ON /dev/hd4 32 768 1 637 6 51% 16 63 11% / /dev/hd2 1212416 57592 96% 36 386 13% /usr /dev/hd9var 532 48 30 824 43% 540 5% /var /dev/hd3 106496 99 932 7% 135 1% /tmp /dev/hd1 4096 39 16 5% 25 % /home /proc. ON /dev/vg00/lvol3 151552 89500 58669 60% / /dev/vg00/lvol1 47829 24109 18 937 56% /stand /dev/vg00/lvol9 131 0720 860829 422 636 67% /var /dev/vg00/lvol8 972800 55 439 2 39 235 8 59% /usr 132 Chapter 5 Table. ON /dev/vg 13/ lvol1 4190208 1155095 2850597 29% /u2 /dev/vg00/lvol7 102400 4284 92256 4% /tmp /dev/vg00/lvol 13 2 039 808 16640 73 352294 83% /test2 /dev/vg00/lvol6 720896 531 295 1779 53 75% /opt /dev/vg00/lvol5