43 ■ ■ ■ CHAPTER 7 Indirect Reference Variables I t is possible to set a normal (direct) variable in three ways: • Directly, by assigning it a value • By storing the output of a command • By storing the results of some type of calculation In all of these cases, although you might not know a variable’s value in advance of any given point, you do know the name of the variable that is to receive the value. For example, AGE=36 is a direct variable assignment. The value of 36 might change at a later stage and it may also have some calculation applied to it, but the variable name AGE will not change. In some cases, however, you may need the ability to generate variable names on the fly. You may not know the number or names of variables you are going to need at the time of execution. These could be referred to as variable variables, or variable variable names. This chapter shows how to create and use such indirect variables. Log File Monitoring with Indirect Variables The following example monitors log files and notifies the user when specified string val- ues show up in the file. The script is designed to run continuously while keeping track of where it last left off in the file; thus, it knows where to start the next time it looks. The configuration value at the beginning of the script points to the log files it needs to watch and the string values to be tracked. This configuration value can consist of many entries, each specifying strings the script needs to watch for in all the tracked files. There may be several entries specifying different strings for the same log file. Our example script is configured in this way. #!/bin/sh #set -x debug=0 DELAY=120 44 CHAPTER 7 ■ INDIRECT REFERENCE VARIABLES LOGCHKS="/var/log/messages:authentication%20failure:\ rbpeters:warn /var/log/messages:recv%20failure::error" This LOGCHKS variable configures which log files will be monitored. You could create a separate configuration file to hold this information, but to keep things straightforward, I’ve included the configuration variables as part of the script. Each entry consists of four fields separated by colons (:). Here are the meanings of the four fields: First: The full path to the log file being watched; nothing special here. Second: The string or strings to watch for. Multiple strings can be specified in this field by separating each with a pipe character, |. Because the entries in the configu- ration string are themselves separated by spaces, you can’t have a space within the watch string. If you want to watch for a phrase that includes spaces, the spaces need to be replaced with %20, as shown in the LOGCHKS variable assignment. Third: Exception strings that are to be ignored. In our example, the script will watch for any authentication-failure messages in the /var/log/messages file, with the exception of those containing the string rbpeters. The subfields of this field, like those of the sec- ond, are pipe-separated, and any spaces should be replaced with %20. Fourth: Notification level. The two values here are warn and error. If the notification is not an error, the script defaults to warn. The notification strings are left undefined because they will be determined by the user implementing them. The Main Monitor Loop To begin, start the infinite loop in which the script will be running. Then you can look at each configuration entry. while : do entry_count=0 for LOGCHK in `echo $LOGCHKS` do Now all values of the configured entry have to be parsed and then assigned to a direct variable. The script also replaces %20 characters with real spaces. logfile=`echo $LOGCHK | cut -d: -f1` strings="`echo $LOGCHK | cut -d: -f2` strings="`echo $strings | sed -e \"s/%20/ /g\"`" exceptions=`echo $LOGCHK | cut -d: -f3` exceptions="`echo $exceptions | sed -e \"s/%20/ /g\"`" notify=`echo $LOGCHK | cut -d: -f4` CHAPTER 7 ■ INDIRECT REFERENCE VARIABLES 45 entry_count represents the number assigned to the specific entry in the configuration string. If two log files are configured to be watched, their entry_count values will be 1 and 2, respectively. This variable will be used later to create a new variable on the fly. entry_count=`expr $entry_count + 1` The suffix is nothing more than the name of the log file. Slashes (/) and dots (.) are replaced with underscores (_). You’ll also use the suffix value later to build indirect vari- able names. The combination of this suffix and the entry_count allows us to create unique variable names specific to the log files that the script is working with. suffix=`echo $logfile | sed -e s/\\\//_/g` suffix=`echo $suffix | sed -e s/\\\./_/g` Next comes the first reference to an indirect variable. The shell evaluates a normal line of code within a script so that any variables are replaced with their values before any com- parisons or calculations are performed. The eval command is used when you want the shell to perform an additional evalua- tion prior to the normal evaluation. This allows you to construct names for new variables using the values of existing variables. In this case, the line in the script is as follows: if [ "`eval echo '$COUNT'${suffix}_$entry_count`" = "" ] After the first explicit evaluation (in the embedded eval), to the shell, the line would look like this: if [ "$COUNT_var_log_messages_1" = "" ] Then, when the shell evaluates the line normally, it sees a direct variable, although we know that the direct variable has been conjured up by a prior evaluation. Now back to the code logic: the code then checks whether the log file’s base line count is null. The only time it will be null is the first time the line count is tested. If this is the case, the base line count is set to the current file length (in number of lines). if [ "`eval echo '$COUNT'${suffix}_$entry_count`" = "" ] then eval BASE${suffix}_$entry_count=`wc -l $logfile | awk '{ print $1 }'` fi Resetting the value of the line count the first time the loop is executed is a safe way of not having the monitor find any previous string entries being watched for in the log file. We don’t want to see strings that were there prior to the monitor ever running. Now the line count of the log file is set. The line count is different from the base count. Let us assume the script starts up to find 10 lines in the log file. The log’s base is then set to 10. The script sets the line count of the file to 10, sees there is no difference between the line count and the base count, and completes running this script segment. eval COUNT${suffix}_$entry_count=`wc -l $logfile | awk '{ print $1 }'` 46 CHAPTER 7 ■ INDIRECT REFERENCE VARIABLES The script then sleeps for the number of seconds specified at the beginning of the script and wakes up again. Suppose that it now finds 13 lines in the file. The difference between the file’s line count and the base count is used to detect new log entries. The following code checks to see whether the log file has grown. If it has, we use the tail command to check the newly added lines inside the log file for the desired strings. The script then resets the file’s base count to whatever the current line count happens to be so you don’t look at lines that have already been checked. if [ `eval echo '$COUNT'${suffix}_$entry_count` -gt `eval echo '$BASE'${suffix}_$entry_count` ] then LINES=`eval expr '$COUNT'${suffix}_$entry_count - '$BASE'${suffix}_$entry_count` eval \ BASE${suffix}_$entry_count='$COUNT'${suffix}_$entry_count if [ "$exceptions" != "" ] then MSGS=`tail -$LINES $logfile | egrep -i "\"$strings\"" | egrep -iv "$exceptions"` test $debug -gt 0 && echo "MSGS is $MSGS" else MSGS=`tail -$LINES $logfile | egrep -i "$strings"` test $debug -gt 0 && echo "MSGS is $MSGS" fi If any messages found in the log file match what you’re looking for, the script should send a notification. As mentioned earlier, there are two possible forms of notification: a warning and an error. In the environments I’ve worked with, both notification methods would normally result in an e-mail with a warning status message, or in an alphanumeric page if it were a more critical message. You must decide how to configure the notifica- tions, so the code here simply echoes a message depending on the notification type. if [ ! -z "$MSGS" ] then if [ "$notify" != "error" ] then echo Send a warning notification . else echo Send an error notification . fi fi If the file’s line count is less than the base value (the value from the previous loop through the code), you need to reset the base value. elif [ `eval echo '$COUNT'${suffix}_$entry_count` -lt `eval echo '$BASE'${suffix}_$entry_count` ] then # This resets the tracked size of the # log if the log size gets smaller eval BASE${suffix}_$entry_count='$COUNT'${suffix}_$entry_count CHAPTER 7 ■ INDIRECT REFERENCE VARIABLES 47 if [ "$exceptions" != "" ] then MSGS=`cat $logfile | egrep -i "\"$strings\"" | egrep -iv "$exceptions"` test $debug -gt 0 && echo "MSGS is $MSGS" else MSGS=`cat $logfile | egrep -i "$strings"` test $debug -gt 0 && echo "MSGS is $MSGS" fi if [ ! -z "$MSGS" ] then if [ "$notify" != "error" ] then echo Send a warning notification . else echo Send an error notification . fi fi A likely scenario for this occurrence is when the log file is trimmed to a preset size to save disk space, such as with the logrotate utility. If this is the case, we want to check the whole file for the strings we’re looking for. If we don’t, we might miss something. If there is no change in the file size, nothing needs to be done. We just complete the loop, and go back and repeat the same operations for all the other log files in the configu- ration string. Finally, the script should sleep for the specified amount of time before starting over again. else test $debug -gt 0 && echo "No change in size of $logfile" fi done sleep $DELAY done . names. This chapter shows how to create and use such indirect variables. Log File Monitoring with Indirect Variables The following example monitors log files. 43 ■ ■ ■ CHAPTER 7 Indirect Reference Variables I t is possible to set a normal (direct) variable in three