PINGSTAT2=$(ping_host $HOSTPINGING | grep transmitted \ | awk ‘{print $4}’) if (( PINGSTAT2 == 0 )) then # It REALLY IS unreachable Notify!! echo “Unreachable” echo “Unable to ping $HOSTPINGING from $THISHOST” \ | tee -a $PING_OUTFILE else echo “OK” fi else echo “OK” fi done fi } ###################################################### function send_notification { if [ -s $PING_OUTFILE -a “$MAILOUT” = “TRUE” ]; then case $UNAME in AIX|HP-UX|Linux) SENDMAIL=”/usr/sbin/sendmail” ;; SunOS) SENDMAIL=”/usr/lib/sendmail” ;; esac echo “\nSending e-mail notification” $SENDMAIL -f randy@$THISHOST $MAILLIST < $PING_OUTFILE fi } ################################################## ############ START of MAIN ####################### ################################################## ping_nodes send_notification # End of script Listing 12.1 pingnodes.ksh shell script listing. (continued) 328 Chapter 12 Now we get to the fun stuff! Let’s start out with the three functions because they do all of the work anyway. The first function is ping_host. The idea here is to set up a case statement, and based on the response from the uname command, which was assigned to the UNAME variable in the definitions section, we execute the specific ping command for the particular Unix flavor. If an unlisted Unix flavor is given, an ERROR message is given to the user, and this shell script exits with a return code 1. We must do this because we have no idea what the correct syntax for a ping command should be for an unknown operating system. The ping_host function is called from the ping_nodes function on every loop iteration. Inside the ping_nodes function we first ensure that the $PINGNODES vari- able is set to TRUE; otherwise, the pinging of nodes is disabled. We use the $PINGFILE file to load a variable, PINGLIST, with a list of nodes that we want to ping. This extra step is done to give the user the ability to comment out spe- cific node(s) in the $PINGFILE. Without this ability you would leave the user in a state of annoyance for all of the notifications because of a single node being down for a period of time. The command to strip out the commented lines and leave the remain- ing nodes in the list is shown here. PINGLIST=$(cat $PINGFILE | grep -v ‘^#’) Notice how this command substitution works. We cat the $PINGFILE and pipe the output to a grep command. In the grep part of the statement we use the -v switch. The -v switch tells grep to list everything except for the following pattern, which is “^#” in this case. Now let’s look at the ^# part. When you put a carat character (^) in front of a pattern in this grep statement, we are ignoring any line that begins with a pound sign (#). The carat (^) means begins with. A for loop is started using the $PINGLIST variable as a list, which contains each node in the /usr/local/bin/ping.list file that is not commented out. For each node in the listing we echo to the screen the target node name and call the ping_host function inside of a command substitution statement on each loop iteration, which is shown here. echo “Pinging > ${HOSTPINGING} \c” PINGSTAT=$(ping_host $HOSTPINGING | grep transmitted | awk ‘{print $4}’) For each node in the $PINGLIST the echo statement and the command substitution statement are executed. There are three possible results for the command substitution statement, and we test for two; the last one is assumed. (1) The PINGSTAT value is 0, zero. If the packets received are 0, zero, then we sleep for $INTERVAL seconds and try to reach the node again, this time assigning the packets received to the PINGSTAT2 variable. (2) The PINGSTAT value is NULL. This results when you try to ping a node that is unknown to the system. In this case we echo to the screen Unknown host and continue to the next node in the list. (3) The PINGSTAT value is nonzero and non- NULL, which means that the ping was successful. Please study each of these tests in the ping_nodes function. Automated Hosts Pinging with Notification 329 Notice the tests used in the if statements. Each of these is a mathematical test so we use the double parentheses method of testing, as shown here. if (( PINGSTAT == 0 )) There are two things to notice in this if statement. The first is that there is no dollar sign ($) in front of the PINGSTAT variable. The dollar sign is not needed in a mathe- matical test when using the double parentheses method because the shell assumes that any nonnumeric string is a variable for this type of mathematical test. I have had cases where I added the dollar sign ($) in front of the variable, and it took me four days to figure out why the script was failing. In other cases I have seen the dollar sign used and the script worked without error. I always remove the dollar sign, just in case. This problem is extremely hard to find should an error occur. The second point I want to make in the previous if statement is the use of the dou- ble equal signs (==). Using this type of mathematical test, a single equal sign is an assignment, not an equality test. This sounds a little strange, but you can actually assign a value to a variable in a test. To test for equality, always use double equal signs (==) with this test method. The last function in this shell script is the send_notification function. This function is used to send an email notification to each address listed in the /usr/local/bin/mail.list file, which is pointed to by the MAILFILE variable. Before attempting any notification the function tests to see if the $PING_OUTFILE file has anything in it or if its size is greater than zero bytes. The second test is to ensure that the MAILOUT variable is set to TRUE. If the $PING_OUTFILE has some data and the MAILOUT variable is set to TRUE, then the function will attempt to notify each email address in the $MAILFILE. In the send_notification function notice that I am using the sendmail com- mand, as opposed to the mail or mailx commands. I use the sendmail command because I worked at a shop where I had a lot of trouble getting mail through the fire- wall because I was sending the mail as root. I found a solution by using the sendmail command because I can specify a valid nonroot user as the person who sent the email. The command I use is shown here. sendmail -f randy@$THISHOST $MAILLIST < $PING_OUTFILE In this statement the -f <user@host> specifies who is sending the e-mail. The $MAILLIST is the list of persons who should receive the email, and the < $PING_OUTFILE input redirection is the body text of the email, which is stored in a file. I still have one little problem, though. The sendmail command is not always located in the same directory, and sometimes it is not in the $PATH. On AIX, HP-UX, and Linux the sendmail command is located in /usr/sbin. On Solaris the sendmail command is located in the /usr/lib directory. To get around this little problem we need a little case statement that utilizes the $UNAME variable that we used in the ping_host function. With a little modification we have the function shown in List- ing 12.2. 330 Chapter 12 function send_notification { if [ -s $PING_OUTFILE -a “$MAILOUT” = “TRUE” ]; then case $UNAME in AIX|HP-UX|Linux) SENDMAIL=”/usr/sbin/sendmail” ;; SunOS) SENDMAIL=”/usr/lib/sendmail” ;; esac echo “\nSending e-mail notification” $SENDMAIL -f randy@$THISHOST $MAILLIST < $PING_OUTFILE fi } Listing 12.2 send_notification function listing. Notice that we used a single line for AIX, HP-UX, and Linux in the case statement. At the end of the function we use the $SENDMAIL variable to point to the correct full path of the sendmail command for the specific operating system. Let’s not forget to look at the pingnodes.ksh shell script in action! In the follow- ing output, shown in Listing 12.3, the node dino is unknown to the system, and the mrranger node is powered down so there is no response from the ping to the system. # ./pinghostfile.ksh.new Pinging > yogi OK Pinging > bambam OK Pinging > booboo OK Pinging > dino Unknown host Pinging > wilma OK Pinging > mrranger Unreachable Trying one more time Unreachable Unable to ping mrranger from yogi Sending e-mail notification Listing 12.3 pingnodes.ksh shell script in action. From the output in Listing 12.3, notice the result of pinging the node dino. I com- mented out the hostname dino in the /etc/hosts file. By doing so I made the node Automated Hosts Pinging with Notification 331 unknown to the system because DNS is not configured on this system. The mrranger node is powered down so it is known but not reachable. Notice the difference in the outputs for these two similar, but very different, situations. Please study the code related to both of these tests in the ping_nodes function. Other Options to Consider As always, we can improve on any shell script, and this one is no exception. I have listed some options that you may want to consider. $PINGLIST Variable Length Limit Problem In this scripting solution we gave the user the capability to comment out specific nodes in the $PINGFILE. We assigned the list of nodes, which is a list without the comments, to a variable. This is fine for a relatively short list of nodes, but a problem arises when the maximum variable length, which is usually 2048 characters, is exceeded. If you have a long list of nodes that you want to ping and you notice that the script never gets to the end of the ping list, you have a problem. Or if you see a funny-looking node name, which is probably a hostname that has been cut off by the variable limit and associated with a system error message, then you have a problem. To resolve this issue, define a new file to point to the PINGLIST variable, and then we will use the file to store the ping list data instead of a variable. To use PINGLIST as a file, add/ change the following lines: ADD THIS LINE: PINGLIST=/tmp/pinglist.out CHANGE THIS LINE: PINGLIST=$(cat $PINGFILE | grep -v ‘^#’) TO THIS LINE: cat $PINGFILE | grep -v ‘^#’ > $PINGLIST CHANGE THIS LINE: for HOSTPINGING in $(echo $PINGLIST) TO THIS LINE: for HOSTPINGING in $(cat $PINGLIST) 332 Chapter 12 Using the file to store the ping list data changes the limit to the maximum file size that the system supports or when the filesystem fills up, which should be plenty of space for anyone. This modified shell script is located on this book’s companion Web site. The script name is pingnodes_using_a_file.ksh. Ping the /etc/hosts File Instead of a List File This may be overkill for any large shop, but it is easy to modify the shell script to accomplish this task. You want to make the following change to the shell script after completing the tasks in the previous section “$PINGLIST Variable Length Limit Prob- lem” to the shell script shown in Listing 12.1. CHANGE THESE LINES: if [ -s $PINGFILE ] then PINGLIST=$(cat $PINGFILE | grep -v ‘^#’) TO THESE LINES: if [ -s /etc/hosts ] then # Ping all nodes in the /etc/hosts file cat /etc/hosts | sed /^#/d | sed /^$/d | grep -v 127.0.0.1 \ | awk ‘{print $2}’ > $PINGLIST In this changed code we cat the /etc/hosts file and pipe the output to a sed statement, sed /^#/d. This sed statement removes every line in the /etc/hosts file that begins with a pound sign (#). The output of this sed statement is then piped to another sed statement, sed /^$/d, which removes all of the blank lines in the /etc/hosts file (the blank lines are specified by the ^$). This sed output is sent to a grep command that removes the loopback address from the list. Finally, the remaining output is piped to an awk statement that extracts the hostname out of the second field. The resulting output is redirected to the $PINGLIST file. This modified shell script to ping the /etc/hosts file is included on the Web site that accompanies the book. The filename is pinghostsfile.ksh. Logging I have not added any logging capability to this shell script. Adding a log file, in addi- tion to user notification, can help you find trends of when nodes are unreachable. Adding a log file is not too difficult to do. The first step is to define a unique log file- name in the definitions section and assign the filename to a variable, maybe LOGFILE. In the script test for the existence of the file, using a test similar to the following state- ment will work. Automated Hosts Pinging with Notification 333 ADD THESE LINES: LOGPATH=/usr/local/log LOGFILE=${LOGPATH}/pingnodes.log if [ ! -s $LOGFILE ] then if [ ! -d $LOGPATH ] then echo “\nCreating directory ==> $LOGPATH\c” mkdir /usr/local/log if (( $? != 0 )) then echo “\nUnable to create the $LOGPATH directory EXITING \n” exit 1 fi chown $USER /usr/local/log chmod 755 $LOGPATH echo fi echo “\nCreating Logfile ==> $LOGFILE\c” cp /dev/null > $LOGFILE chown $USER $LOGFILE echo fi After adding these lines of code, use the tee -a $LOGFILE command in a pipe to both display the text on the screen and log the data in the $LOGFILE. Notification of “Unknown Host” You may want to add notification, and maybe logging too, for nodes that are not known to the system. This usually occurs when the machine cannot resolve the node name into an IP address. This can be caused by the node not being listed in the /etc/hosts file or failure of the DNS lookup. Check both conditions when you get the Unknown host message. Currently, this shell script only echoes this information to the screen. You may want to add this message to the notification. Notification Method In this shell script we use email notification. I like email notification, but if you have a network failure this is not going to help you. To get around the network down problem with email, you may want to set up a modem, for dial-out only, to dial your alpha- numeric pager number and leave you a message. At least you will always get the message. I have had times, though, when I received the message two hours later due to a message overflow to the modem. 334 Chapter 12 You may just want to change the notification to another method, such as SNMP traps. If you execute this shell script from an enterprise management tool, then the response required back to the program is usually an SNMP trap. Refer to the docu- mentation of the program you are using for details. Automated Execution Using a Cron Table Entry I know you do not want to execute this shell script from the command line every 15 minutes yourself! I use a root cron table entry to execute this shell script every 15 min- utes, 24 hours a day, Monday through Saturday, and 8:00 A.M. to midnight on Sunday; of course, this requires two cron table entries. Because weekly backups and reboots happen early Sunday morning, I do not want to be awakened every Sunday morning when a machine reboots, so I have a special cron entry for Sunday. Both root cron table entries shown execute this script every 15 minutes. 5,20,35,50 * * * 1-6 /usr/local/bin/pingnodes.ksh >/dev/null 2>&1 5,20,35,50 8-23 * * 0 /usr/local/bin/pingnodes.ksh </dev/null 2>&1 The first entry executes the pingnodes.ksh shell script at 5, 20, 35, and 50 minutes of every hour from Monday through Saturday. The second entry executes the ping-nodes.ksh shell script at 5, 20, 35, and 50 minutes from 8:00 A.M. until 11:59 P.M., with the last ping test running at 11:50 P.M. Sunday night. Summary In this chapter we took a different approach than that of some other shell scripts in this book. Instead of creating a different function for each operating system, we created a single shell script and then used a separate function to execute the correct command syntax for the specific operating system. The uname command is a very useful tool for shell scripting solutions for various Unix flavors in a single shell script. I hope you enjoyed this chapter. I think we covered some unique ways to solve the scripting problems that arise when programming for multiple Unix flavors in the same script. In the next chapter we will dive into the task of taking a system snapshot. The idea is to get a point-in-time system configuration for later comparison if a system problem has you puzzled. See you in the next chapter! Automated Hosts Pinging with Notification 335 337 Have you ever rebooted a system and it came up in an unusual state? Any time you reboot a system you run a risk that the system will not come back up properly. When problems arise it is nice to have before and after pictures of the state of the machine. In this chapter we are going to look at some options for shell scripts that execute a series of commands to take a snapshot of the state of the machine. Some of the things to con- sider for this system snapshot include filesystems that are mounted, NFS mounts, processes that are running, network statistics and configuration, and a list of defined system resources, just to name a few. This is different from gathering a snapshot of performance statistics, which is gathered over a period of time. All we are looking for is system configuration data and the system’s state at a point in time, specifically before the system is rebooted or when it is running in a normal state with all of the applications running properly. With this information captured before a system reboot, you have a better chance of fixing a reboot problem quickly and reducing down time. I like to store snapshot infor- mation in a directory called /usr/local/reboot with the command names used for filenames. For this shell script all of the system information is stored in a single file with a section header added for each command output. Overall, this is not a difficult shell script to write, but gathering the list of commands that you want to run can some- times be a challenge. For example, if you want to gather an application’s configuration you need to find the commands that will produce the desired output. I always prefer having too much information, rather than not enough information, to troubleshoot a problem. Taking a System Snapshot CHAPTER 13 [...]... 19, 19, 19, 19, 10, 10, 13, 10, 11, 13, 13, 7, 7, 7, 7, 5, 5, 1, 18, 18, 26, 26, 2, 0 1 2 3 4 5 6 7 0 10 14 10 0 15 30 0 1 3 2 0 1 0 0 1 0 1 3 Jun Mar Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jun Jul Jul Jul 23 29 26 26 26 26 26 26 26 29 26 29 26 26 26 26 26 26 26 26 26 24 24 23 26 26 26 15:23 13:49 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001... UGc 0 10.10/ 16 10.10.10.1 U 37 127/8 127.0.0.1 U 5 Use 0 135807 264 Route Tree for Protocol Family 24 (Internet v6): Listing 13.3 AIXsysconfig.ksh shell script in action (continues) If PMTU Exp en0 en0 lo0 - - 355 3 56 Chapter 13 ::1 ::1 UH 0 0 lo0 168 96 - ################################################# Network Interface Statistics Name en0 en0 lo0 lo0 lo0 Mtu 1500 1500 168 96 168 96 168 96 Network link#2... 10924 67 % 61 680 96% 10 568 81% 70184 35% 3892 5% 164 60 98% 252 96 76% 3504 56 15% node mounted mounted over options -/dev/hd4 / jfs /dev/hd2 /usr jfs /dev/hd9var /var jfs /dev/hd3 /tmp jfs /dev/hd1 /home jfs /proc /proc procfs /dev/hd10opt /opt jfs /dev/scripts_lv /scripts jfs /dev/lv_temp /tmpfs jfs Iused %Iused Mounted on 1854 12% / 40941 12% /usr 67 3 6% /var 223 1% /tmp 55 6% /home... Ipkts Ierrs 2 .60 .8c.2d.75.b1 112330 yogi 112330 28302 loopback 28302 28302 0 0 0 0 0 Opkts Oerrs 10 869 7 10 869 7 28304 28304 28304 Coll 0 0 0 0 0 0 0 0 0 0 ################################################# Filesystem Statistics Filesystem 1024-blocks /dev/hd4 32 768 /dev/hd2 1449984 /dev/hd9var 53248 /dev/hd3 1 064 96 /dev/hd1 40 96 /proc /dev/hd10opt 65 5 360 /dev/scripts_lv 102400 /dev/lv_temp 40 960 0 Free %Used... root 267 2 1 0 Jul 23 root 3140 1 0 Jul 23 root 364 2 464 4 0 17:11:20 root 3882 1950 0 Jul 23 connections root 4 168 1950 0 Jul 23 root 4388 1950 0 Jul 23 root 464 4 1950 0 Jul 23 nobody 49 06 5418 0 Jul 23 daemon 8798 1950 0 Jul 23 root 9034 1950 0 Jul 23 root 92 96 1950 0 Jul 23 root 9554 1950 0 Jul 23 root 9814 1950 0 Jul 23 root 103 36 1 0 Jul 23 root 10588 1950 0 Jul 23 root 10842 1 0 Jul 23 root 11 360 ... Space Information Page Space hd6 Physical Volume hdisk0 Total Paging Space 336MB Volume Group Size rootvg 336MB %Used Active Auto 10 yes yes Percent Used 10% ################################################# Hard Disks Defined hdisk0 hdisk1 0000 367 7cf 068 b62 00012 560 8a48c132 ################################################# Volume Group Hard Drives Listing 13.3 AIXsysconfig.ksh shell script in action (continues)... 40941 12% /usr 67 3 6% /var 223 1% /tmp 55 6% /home - /proc 162 60 10% /opt 887 4% /scripts 26 1% /tmpfs vfs date -Jul 23 18: 56 Jul 23 18: 56 Jul 23 18: 56 Jul 23 18: 56 Jul 23 18:57 Jul 23 18:57 Jul 23 18:57 Jul 23 18:57 Jul 23 18:57 ################################################# Defined Volume Groups rootvg Listing 13.3 AIXsysconfig.ksh shell script in action (continued) rw,log=/dev/hd8 rw,log=/dev/hd8... Drive Manufacturer IBMRISC Machine Type and Model 066 4M1H Part Number .86F0101 ROS Level and ID 5 5A Serial Number .00221833 EC Level 8951 86 FRU Number 86F0118 Device Specific.(Z0) 000002029F00001E Device Specific.(Z1) 75G 364 4 Device Specific.(Z2) 0983 Device Specific.(Z3) 95123 Device Specific.(Z4) 0002 Device Specific.(Z5) 22 Device Specific.(Z6) 895172 rmt0 00-00-0S-5,0 5.0 GB 8mm Tape Drive... /usr/sbin/portmap /usr/sbin/inetd /usr/sbin/tftpd -n /usr/sbin/rpc.statd /usr/sbin/biod 6 /usr/sbin/nfsd 3891 /usr/sbin/rpc.mountd /usr/sbin/rpc.lockd /usr/sbin/uprintfd qdaemon /usr/sbin/cron /usr/sbin/writesrv pts/0 0:03 dtfile Listing 13.3 AIXsysconfig.ksh shell script in action (continues) 359 360 Chapter 13 root 17540 165 38 0 21: 16: 59 pts/3 0:00 /usr/bin/ksh #################################################... ################################################# Logical Volume Information by Volume Group rootvg: LV NAME POINT hd5 hd6 hd8 hd4 hd2 hd9var hd3 hd1 hd10opt scripts_lv lv_temp TYPE LPs PPs PVs LV STATE MOUNT boot paging jfslog jfs jfs jfs jfs jfs jfs jfs jfs 2 84 1 8 354 13 26 1 160 25 100 2 84 1 8 354 13 26 1 160 25 100 1 1 1 1 2 2 1 1 2 1 1 closed/syncd open/syncd open/syncd open/syncd open/syncd open/syncd open/syncd . useful tool for shell scripting solutions for various Unix flavors in a single shell script. I hope you enjoyed this chapter. I think we covered some unique ways to solve the scripting problems. execute the specific ping command for the particular Unix flavor. If an unlisted Unix flavor is given, an ERROR message is given to the user, and this shell script exits with a return code 1. We must. overkill for any large shop, but it is easy to modify the shell script to accomplish this task. You want to make the following change to the shell script after completing the tasks in the previous