252 C HAPTER ฀ D EP L OYING YOU R FIR S T A P P L IC A TION security, as well as differences in how much time they’re willing to spend investigating new file-distribution methods System administrators end up becoming very adept at copying files from many different sources to many different destinations If you’re at or near the beginning of your SA career, study the options and tools in this chapter on your own and learn them well The time will be well spent At this stage in the construction of our example infrastructure, it might seem like we’re almost finished After all, we had the required infrastructure to deploy our first application, right? Well, yes, we have a functional infrastructure, but it’s still in its early stages We’re lacking in several key areas, and we’ll address some of our basic reporting needs in the next chapter CHAPT ER Generating Reports and Analyzing Logs Y ou need to know about errors on the systems in your environment before they turn into major problems You also need the ability to see the actions your automation system is performing This means you’ll need two types of reporting: ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ You want to know right away if a system has serious hardware issues or major application issues We’re going to run a reporting system that looks for unwanted words or phrases Here is one such unwanted message: This particular message means that you’re running a version of BIND that works ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ We’ll use real-time alerting to pick up on this condition as well as others Reporting on cfengine Status You have two main ways of tracking the actions and changes that cfengine makes across ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀you have the output of ฀ ฀ ฀ ฀ ฀ ฀ ฀ included in the syslog entries When runs ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀output of any commands run by ฀ ฀ ฀ ฀ ฀ ฀ e-mails the output of to the e-mail address as defined in ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ 253 254 C HAPTER ฀ GE NER A TING R EP OR TS A ND A NA L YZIN G LO G S We are very interested in the contents of ฀ ฀ ฀ ฀ ฀ gate them centrally We can later run interactive checks such as simple ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ we can write scripts to flag and e-mail particular output-file contents to the administrators This sort of scheme is useful if you’d rather use custom reporting instead of the e-mail functionality of The first step is to get the directory contents from all hosts aggregated to ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ You don’t want to keep an explicit list of all your systems and have one system try to pull the directory contents from each You’d rather have each system be responsible for pushing its directory contents to a central location We can take advantage of the rsync daemon that we placed on our cfengine master to accomplish this This approach brings with it some important security considerations: ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ access or destroy parts of the filesystem on the cfengine master ฀ ฀ ฀ ฀ ฀ ฀ tally or maliciously ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ - ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ We can case because we run our rsync daemon as a non ฀ ฀ ฀ ฀don’t want to start running it as root because software bugs such as buffer overflows would result in remote attackers’ ability to execute code as root on our system We’d rather run the daemon as a non user and protect ourselves another way Rsync allows a script ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ security checks Caution You’ll see the code-continuation character ( ) in some of this chapter’s code sections This character signifies that the line in which it appears is actually a single line, but could not be represented as such because of print-publishing restrictions It is important that you incorporate the line in question as a single line in your environment You can download all the code for this book from the Downloads section of the Apress web site at ฀ ฀add this section to the goldmaster repository at ฀ : ฀ ฀ ฀ ฀ G E N E R A T I N G R E P O R T S A N D A N A LY Z I N G LO G S C HA P TER We define the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ setting Note The option allows you to specify a space-separated list of rsync command-line options that will be refused by your rsync daemon We utilize it to keep clients from deleting files or from leaving partially copied files On goldmaster we create the directory into it named ฀ ฀ ฀these contents: ฀ ฀ ฀ ฀ 255 256 C HAPTER ฀ GE NER A TING R EP OR TS A ND A NA L YZIN G LO G S ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ that the client is copying to matches what we expect We allow a client to copy only to a directory matching either its short or fully qualified name Note that this scheme relies on the security and integrity of the DNS You could easily modify this technique to use IP ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ control your DNS servers ฀we’ll enhance our current rsync server task at so it looks like this: C HA P TER ฀ G E N E R A T I N G R E P O R T S A N D A N A LY Z I N G LO G S In ฀ ฀ ฀ ฀ ฀ script as well as create the directory where we’ll upload the files from clients We also include a action to remove files older than 60 days from this new directory The directory will grow without bounds if we don’t ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ We don’t currently have the action defined in our ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ : 257 258 C HAPTER ฀ GE NER A TING R EP OR TS A ND A NA L YZIN G LO G S ฀ To upload the Add this task to the every run ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ with these contents: ฀ ฀ ฀ ฀ ฀ directory on C HA P TER ฀ G E N E R A T I N G R E P O R T S A N D A N A LY Z I N G LO G S USING RED HAT AS THE AGGREGATOR HOST If you run Red Hat Linux on the host where you’d like to aggregate your to be aware of two things: ฀ ฀ ฀ ฀ ฀ a daemon ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ will need to be changed to be owned by the nobody user instead of , so directories, you need will need to be modified to start rsync as ฀ ฀ We don’t cover automation of Red Hat systems for this role, but wanted to point out the obvious modifications in case you want to try it on your own ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ Solaris installation process We’ll want to install it from the Blastwave repository as part ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ Modify the JumpStart script so that rsync is installed by ฀ ฀ ฀ this line on hemingway฀ ฀ ฀ ฀ ฀ ฀ ฀ : We simply want to append rsync to the list: Now that the central host goldmaster is getting populated with the directory ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀฀ ฀ ฀ from all runs ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ probably find it difficult to keep up with and make sense of the e-mails as a whole A simple hourly or daily script to summarize and e-mail the aggregated directory contents would make more sense at that point Create a simple script for this purpose at with these contents: 259 260 C HAPTER ฀ GE NER A TING R EP OR TS A ND A NA L YZIN G LO G S This shell script simply looks for any new files created in the centralized ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ command using the command ฀ ฀ ฀ ฀ ฀ Note The pipe to the mail command is outside the for loop, so we don’t get a separate mail for each directory under If you’re not sure you understand why this is necessary, try moving the pipe to the mail command to the same line as the find command Experimenting with shell scripts is one of the best ways to increase your shell-scripting knowledge The contents of are already synchronized to all hosts at the location ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ be copied to the goldmaster฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ make sure that it attempts to run only when invoked on the correct host ฀ G E N E R A T I N G R E P O R T S A N D A N A LY Z I N G LO G S C HA P TER ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ The ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ directory is stored on the host serving the role of —because ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀this line to the file : The e-mail output of the script is very basic: This is a good way to report on output from When you have new ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ offers this useful feature: it doesn’t send a new e-mail when the output of the current run matches that of the previous run Our example script doesn’t implement this functionality; this is left as an exercise for the reader ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ 261 CHAPTER 10 ฀ MONITORING ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ build time because of the relative lack of compile-time dependencies Now that we have built Nagios, we’ll need to copy it to our cfengine master for later deployment: Step 3: Building the Nagios Plug-ins Now, we have Nagios compiled, but it won’t be useful without plug-ins To compile the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ Then, we copied the programs over to the cfengine master: 281 282 ฀ M ONITOR ING C HAPTER 10 ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ plug-ins compiled for all platforms at our site for use with NRPE, covered later in this ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ the rhmaster system, where we have a C development environment: Then, from the cfengine master, we copied the programs: ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ Again, we copied the programs over to the cfengine master: Step 4: Copying the Nagios Start-up Script on the cfengine Master Next, we created a directory named copied the script from on the cfengine master and on the system where we built Nagios for ฀ CHAPTER 10 ฀ ฀ ฀ ฀ ฀ ฀ ฀ ries later in this chapter ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ MONITORING command that we ran when ฀ ฀ ฀ ฀ directo- Step 5: Separating the Nagios Configuration Directory from the Program Directory ฀place the Nagios daemon configuration files at by moving the directory up one directory level and into The directory layout inside ฀ ฀ ฀ ฀ ฀ ฀ , as set up by the ฀ ฀ ฀ ฀ ฀ : ฀ ฀ ฀ com฀ Notice the directory named : it is where the example configuration places all ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ this directory for the objects that we define ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ line from this: to this: The ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ directory outside of the main Nagios directory in step three, so we need to have Nagios look for them in the new directory 283 284 C HAPTER 10 ฀ M ONITOR ING Create the Nagios Web Interface Configuration Files In this section, we’ll cover steps six through eight, which are the creation of Apache con฀ ฀ ฀ ฀ ฀ ฀ ฀ Step 6: Generating an SSL Certificate for the Nagios Web Interface ฀ ฀ ฀ ฀ ฀ root user on the host etchlamp): ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ automate the distribution of this file using cfengine, as usual ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ Step 7: Creating the Apache VirtualHost Configuration for the Nagios Web Interface ฀need to configure Apache with the required directives to serve our Nagios web interface using the Nagios CGI programs In addition, we need to make sure that authentication is used, since the Nagios web interface contains information we only want ฀ ฀ ฀ ฀ ฀ Modification operations are those that stop alerts for some or all systems, send man฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ of our Nagios framework by controlling access Here is our Apache configuration file, which we’ve placed on the cfengine master at the location : ฀ CHAPTER 10 ฀ MONITORING ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ we continue the trend here Just be aware that you shouldn’t remove the authentication requirements if you have trouble making user accounts work Take the time to it right ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀important Step 8: Create the Nagios Web Interface Authentication File ฀ ฀ ied it to the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ etchlamp), and copdirectory on the cfengine master: 285 286 ฀ M ONITOR ING C HAPTER 10 The user is special, in that it will have all the required access to the Nagios web interface that you will require Always create this user account Once you have Nagios up and running properly, read the online Nagios authentica฀ ฀ ฀ ฀ ฀ to learn to configure additional users Step 9: Copying the Nagios Daemon and Configuration Files with cfengine In steps one through eight, we put together all the building blocks to set up a working Nagios instance The bare minimum setup is in place: ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ have everything that we need to automate the copy and setup of Nagios and the Nagios ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ on the cfengine master ฀ ฀ ฀ ฀ ฀ and put a task named ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ section): ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ all the places that the version-specific directory name is needed will make it much easier ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ cfengine master and update the variable in this task: CHAPTER 10 ฀ MONITORING Here, we set up a class based on the existence of the current Nagios directory to be used in the next section Here, we copy the Nagios programs when the directory meant to hold the current ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ) class negation opera฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ so we also make use of the and classes to make sure that we only copy the binaries to the correct platform: 287 288 C HAPTER 10 ฀ M ONITOR ING Next, we copy the entire ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ end up deploying several versions of Nagios at once, but we’d like the path to the configuration files should always remain constant This is easy to ensure when the configuration files are maintained separately from the programs themselves: ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ate the proper symlinks in the directories Notice that two different classes ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ), and the other is meant to ensure that the start-up script symlinks are properly created ): CHAPTER 10 ฀ MONITORING The three copies in the preceding code are used to place web interface files in place: the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ for nagios.campin.net: The preceding restarts are triggered when configuration file or program file updates are made in earlier ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ into immediate use, and these take care of that for us: ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ utility which is used to create links in the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ symlinks to create in the cfengine configuration, but frankly, this is easier The rest of the task follows: 289 290 C HAPTER 10 ฀ M ONITOR ING Here, we create a directory used by Nagios to store state information It is critical that the ownership of the directory and permissions allow the user running the Nagios ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ These are simple process monitors that cause Apache and Nagios to be started up if they’re not running on the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ step ten This is the end of the cfengine task Notice that we’re careful to copy ฀ ฀only to appropriate hosts by specifying the class in the copy It obvi฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ cfengine tasks and allow copies to happen only when conditions exactly match what we are expecting In the action in the preceding task, we copy the Nagios binaries only when the ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ larly sync the files If you’re worried about something outside of cfengine changing those files, you could remove the portion from the action and always enforce the proper directory contents ฀ CHAPTER 10 ฀ MONITORING Step 10: Configuring a Nagios Monitoring Host Role in cfengine ฀making the host etchlamp the nine, and to set it, we added this line to machine mentioned in the task in step : Step 11: Creating a Hostgroup File for the Monitoring Host Role in cfengine To complete our Nagios role configuration in cfengine, we added this line to : Then, we created a file on the cfengine master at the location with these contents: Step 12: Copying the Nagios Plug-ins with cfengine ฀will handle the distribution of the Nagios plug-ins in a second task, which we will now ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ As we did in , we use a variable to contain the version-specific directory name, which makes it extremely easy to deploy updates later on but still keep a copy of the previous build The rest of this task simply copies the proper plug-in binaries to each platform at our site and has special single file copies that enforce the setuid bit on binaries that require it ฀ ฀ ฀ ฀ ฀ 291 292 C HAPTER 10 ฀ M ONITOR ING CHAPTER 10 ฀ MONITORING 293 294 C HAPTER 10 ฀ M ONITOR ING CHAPTER 10 ฀ MONITORING ฀ ฀ ฀ variable in this task to create version-specific directories on our hosts and have cfengine create a symlink so that we always have a single filesystem path to ฀ ฀ ฀ ฀ ฀ ฀ ฀ will always work) ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ with this line: Step 13: Creating a DNS Entry for the Monitoring Host ฀then added an alias to our DNS so that we can use the hostname nagios.campin.net when accessing the Nagios server Using the alias will allow us to easily migrate Nagios to ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀https://nagios campin.net and never refer to the system’s real hostname To create the DNS alias, we added this line to ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ and ran ): ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ), we visited ฀ ฀https://nagios.campin.net฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ account we created, and we were presented with the Nagios web interface If you click Service Detail in the left-hand frame, you’ll see details for the system ฀ ฀ ฀right-hand frame It should look like this screenshot: 295 ... software has bugs; and all manner of issues crop up that cause hosts and applications to fail System and service monitoring is automation’s companion tool, used to notify us when systems and applications... grows ฀have chosen Nagios for our system and service monitoring system Nagios is flexible and mature, and help is widely available on Internet mailing lists and newsgroups ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀ ฀... monitoring systems Applications and hosts are added; applications change and need to be monitored differently; hosts fail permanently; and critical thresholds change You need to know your monitoring systems