Contents Overview 1 Cluster Maintenance 2 Troubleshooting Cluster Service 11 Lab A: Cluster Maintenance 24 Review 30 Module 7: Server Cluster Maintenance and Troubleshooting Information in this document is subject to change without notice. The names of companies, products, people, characters, and/or data mentioned herein are fictitious and are in no way intended to represent any real individual, company, product, or event, unless otherwise noted. Complying with all applicable copyright laws is the responsibility of the user. No part of this document may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without the express written permission of Microsoft Corporation. If, however, your only means of access is electronic, permission to print one copy is hereby granted. Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.  2000 Microsoft Corporation. All rights reserved. Microsoft, Active Directory, BackOffice, Jscript, PowerPoint, Visual Basic, Visual Studio, Win32, Windows, Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the U.S.A. and/or other countries. Other product and company names mentioned herein may be the trademarks of their respective owners. Program Manager: Don Thompson Product Manager: Greg Bulette Instructional Designers: April Andrien, Priscilla Johnston, Diana Jahrling Subject Matter Experts: Jack Creasey, Jeff Johnson Technical Contributor: James Cochran Classroom Automation: Lorrin Smith-Bates Graphic Designer: Andrea Heuston (Artitudes Layout & Design) Editing Manager: Lynette Skinner Editor: Elizabeth Reese Copy Editor: Bill Jones (S&T Consulting) Production Manager: Miracle Davis Build Manager: Julie Challenger Print Production: Irene Barnett (S&T Consulting) CD Production: Eric Wagoner Test Manager: Eric R. Myers Test Lead: Robertson Lee (Volt Technical) Creative Director: David Mahlmann Media Consultation: Scott Serna Illustration: Andrea Heuston (Artitudes Layout & Design) Localization Manager: Rick Terek Operations Coordinator: John Williams Manufacturing Support: Laura King; Kathy Hershey Lead Product Manager, Release Management: Bo Galford Lead Technology Manager: Sid Benavente Lead Product Manager, Content Development: Ken Rosen Group Manager, Courseware Infrastructure: David Bramble Group Product Manager, Content Development: Julie Truax Director, Training & Certification Courseware Development: Dean Murray General Manager: Robert Stewart Module 7: Server Cluster Maintenance and Troubleshooting iii Instructor Notes This module is intended to prepare the students to successfully back up and restore a server cluster. Students need to know how to use the troubleshooting tools available for troubleshooting server cluster problems. The module covers common Cluster service problems and possible resolutions. After completing this module, you will be able to:  Perform the steps to successfully back up a server cluster.  Perform the steps to successfully restore a server cluster.  Evict a node from a server cluster.  Identify the tools that are necessary to troubleshoot a cluster failure.  Interpret the entries on the cluster log.  Identify and troubleshoot common server cluster failures: network communications, small computer system interface (SCSI) configuration problems, group, resource, and quorum failures. Materials and Preparation This section provides the materials and preparation tasks that you need to teach this module. Required Materials To teach this module, you need the Microsoft® PowerPoint® file 2087A_02.ppt Preparation Tasks To prepare for this module, you should:  Read the materials for this module and anticipate questions students may ask.  Read Q224075, Q257892, Q248998, Q172951, Q266274, Q234767, Q193890, Q245762 and “Interpreting MSCS Cluster Log, on the Student compact disk.  Be familiar with the Resource Kit Utilities.  Practice the labs.  Study the review questions and prepare alternative answers for discussion. Presentation: 45 Minutes Lab: 15 Minutes iv Module 7: Server Cluster Maintenance and Troubleshooting Module Strategy Use the following strategy to present this module: Because backing up the cluster is a key maintenance task, the first section begins with information on how to backup the cluster configuration files. The following pages cover the complete procedure for restoring an entire cluster in case of catastrophic failure. You can also use each of the topics as a separate procedure for performing a specific task. The troubleshooting section lists the tools that are available for troubleshooting Cluster service and gives common problems and suggested resolutions.  Cluster Maintenance Cluster service is self-tuning and requires no maintenance other than daily backups. • Backup: Backing up the system state backs up the cluster configuration files; however, you also need to back up each node’s data and operating system and the cluster disks. • Restoring the First Node: The overall procedure for restoring a cluster is outlined on this page. The first step, restoring the operating system on the first node, is also covered. The remaining steps are covered in detail on the following pages. • Restoring Cluster Disks: Cluster service uses the disk signature file to identify the cluster disk. To replace this disk, you must write the disk signature file of the old disk onto the new disk. • Restoring the Second Node: Restoring the remaining nodes of the cluster is similar to restoring the first node, except that after it is restored, you need to test the failover capabilities of the cluster before putting the cluster back into the production environment. • Evicting a Node: Evicting a node is a manual process through Cluster Administrator. As always, it is important to have a good backup of the server prior to the eviction process. Module 7: Server Cluster Maintenance and Troubleshooting v  Troubleshooting Cluster Service The key point of this section is to give the students the tools and techniques that are useful in reducing the time it takes to find a root cause for common Cluster service problems. • Troubleshooting Tools: The tools that are used to help troubleshoot a problem with Cluster service are the same tools that are used to help troubleshoot a server running Microsoft Windows ® 2000. • Examining the Cluster Log: Cluster service logs every change configuration and problem to the cluster log. It is important for the students to become familiar with the syntax of the log. • Troubleshooting Network Communications: Students need to know that there are different troubleshooting paths to follow depending on whether the network problem is a node-to-node or a client-to-node problem. • SCSI Configuration Problems: SCSI is less reliable than Fibre. There can be problems with the SCSI controller, SCSI termination, and SCSI cabling. • Group and Resource Failures: Remind students to keep dependency trees vertical so that if a resource fails, it is easier to find a root cause as to which resource is causing the failure of the group. • Quorum Log Corruption: If Cluster service cannot write information to the quorum log, it will not start. You can attempt to reset the quorum log, or you can delete the quorum log and let Cluster service create a new log. vi Module 7: Server Cluster Maintenance and Troubleshooting Instructor Setup for a Lab Lab Strategy This lab is designed to prepare the students to use Backup and Clusrest.exe to perform the proper backup and restore procedures. Students will uninstall Cluster service in preparation for the Network Load Balancing (NLB) portion of the course. NLB and Cluster service cannot run on the same computer. Lab A: Cluster Maintenance To conduct this lab:  Read though the lab carefully, paying close attention to the instructions and details.  Students will need the Clusrest utility from c:\moc\2087\labfiles\mscs  Students work in teams of two, grouped together by their shared bus.  Help the students determine whether they are Node A or Node B. In these exercises each node performs a specific task in the backup and restoration procedures. Both nodes will uninstall Cluster service. Module 7: Server Cluster Maintenance and Troubleshooting 1 Overview  Cluster Maintenance  Troubleshooting Cluster Service ***************************** ILLEGAL FOR NON-TRAINER USE****************************** Server cluster maintenance and troubleshooting are considered two separate disciplines. Maintenance is continuous, whereas troubleshooting has a beginning when the problem is discovered, and an end when the problem is resolved. The two disciplines are complimentary, however. When every troubleshooting procedure that you follow fails, you will need to rebuild the cluster from a backup tape that was generated during a maintenance procedure. After completing this module, you will be able to:  Perform the steps to successfully back up a server cluster.  Perform the steps to successfully restore a server cluster.  Evict a node from a server cluster.  Identify the tools that are necessary to troubleshoot a cluster failure.  Interpret the entries on the cluster log.  Identify and troubleshoot common server cluster failures: network communications, small computer system interface (SCSI) configuration problems, group, resource, and quorum failures. Topic Objective To provide an overview of the module topics and objectives. Lead-in In this module, we will cover Cluster maintenance in the form of backing up and restoring a cluster, and troubleshooting Cluster service. 2 Module 7: Server Cluster Maintenance and Troubleshooting    Cluster Maintenance  Backup  Restoring the First Node  Restoring Cluster Disks  Restoring the Second Node  Evicting a Node ***************************** ILLEGAL FOR NON-TRAINER USE****************************** Cluster service uses the self-tuning features of Microsoft ® Windows® 2000 and requires very little maintenance. The only day-to-day maintenance operation that you need to perform is to back up the cluster. Under special circumstances, a node in the cluster may need to be replaced, for example, when your organization decides to perform a hardware upgrade. In this situation, you need to evict a node from the cluster and add the upgraded node to the cluster. Topic Objective To introduce the fundamental tasks for maintaining a server cluster. Lead-in The only maintenance performed on a cluster is backing up and restoring Cluster service. Module 7: Server Cluster Maintenance and Troubleshooting 3 Backup  Backing Up the System State  Backing Up the Local Disk  Backing Up the Cluster Disk ***************************** ILLEGAL FOR NON-TRAINER USE****************************** Backing up the cluster is no different from backing up Microsoft Windows 2000 Advanced Server. It is recommended that you perform regular backups by using the Windows 2000 Backup program (NTBackup), or other compatible backup programs. Additional backup agents are still necessary to back up applications running on the cluster, such as Microsoft SQL Server ™ and Microsoft Exchange. A cluster-aware backup program will be able to perform the same backup operations as NTBackup, especially with regard to backing up the System State and the cluster configuration database. Backing Up the System State The configuration information for the cluster is located on the registry on each node (HKEY_LOCAL_MACHINE\Cluster). The Backup tool that is included with Windows 2000 backs up the cluster database when you back up each node’s system state. NTBackup backs up the system state on each node. The system state includes:  The quorum log.  The local registry.  The Cluster registry hive. Topic Objective To describe how to back up the system state, node, and cluster disks. Lead-in A backup of the cluster includes the system state, the node, and the cluster disk. Note 4 Module 7: Server Cluster Maintenance and Troubleshooting Backing Up the Local Disk Follow standard computer backup procedures to back up the operating system and the data on the local drives. You must also back up key cluster files on the local disks.  On each node, back up the cluster database files: %systemroot%\cluster\CLUSDB %systemroot%\cluster\CLSUDB.LOG  On each node, back up the clustering service: %systemroot%\cluster\*.* Backup is essential, but regular testing to make sure that backups and restores actually work as expected is also necessary. A good practice is to schedule test backup and restore operations frequently. Backing Up the Cluster Disks It is critical to back up cluster files on the quorum disk and data on the cluster disks, because Cluster service will write information to files in the \mscsdirectory on the quorum disk and cluster-aware applications will likely be placing data on the cluster disk. Because either node of the cluster could own the cluster disk resource at any time, it is possible for each node to back up the data on the drive. However, having each node back up data would require you to install backup hardware and software on each cluster node, which is not the best solution. One possibility is to identify a nonclustered server running Windows 2000 Server and schedule it to back up data remotely through a network connection to the Cluster disk’s administrative share or a hidden share that you create. For example, you might create FBackup$, GBackup$, HBackup$, and WBackup$ file share resources on the virtual server for the root of drives F, G, H, and W. F, G, and H would be cluster disks with data, and W would be the drive letter for the quorum disk. Hidden shares would not appear in a browse list and you could configure them to allow access only to members of the Backup Operators group. Note [...]... the recommendation of the manufacturer 20 Module 7: Server Cluster Maintenance and Troubleshooting Group and Resource Failures Topic Objective To describe how to troubleshoot group and resource failures Cluster Administrator – [MYCLUSTER (MYCLUSTER)] File View Window Help MYCLUSTER Groups Cluster Group Mygroup SQL Group Resources Cluster Configuration SERVER1 SERVER2 Lead-in Group failures are most likely... restart the cluster You use the same tools to identify problems on the cluster as you would use to identify problems on a physical server The best resource for troubleshooting is the cluster log because Cluster service records the activity of each node in the cluster log This log can help you identify problems on the node or in the cluster 12 Module 7: Server Cluster Maintenance and Troubleshooting Troubleshooting... remaining node 4 Stop Cluster service on the node that is to be removed 5 Evict the node 6 Unplug the server from the shared bus (if the shared bus is a SCSI bus, be careful about termination) Note If a new server is to join the cluster later, run the Cluster Installation Wizard and select Join a Cluster Module 7: Server Cluster Maintenance and Troubleshooting 11 Troubleshooting Cluster Service Topic... the Run dialog box, type cmd and then click OK 9 In the command prompt, type cd\program files\resource kit and then press ENTER 10 In the command prompt, type clusrest and press ENTER 28 Module 7: Server Cluster Maintenance and Troubleshooting 11 In the command prompt, type y to continue 12 Wait for clusrest before proceeding 13 Open Cluster Administrator 14 Expand Groups and notice that the Test Group... 24 Module 7: Server Cluster Maintenance and Troubleshooting Lab A: Cluster Maintenance Topic Objective To introduce the lab Lead-in In this lab, you will back up and restore the cluster configuration files, evict a node from the cluster, and uninstall Cluster service *****************************ILLEGAL FOR NON-TRAINER USE****************************** Delivery Tip The students are uninstalling Cluster. .. component Performing a complete restore of a server cluster is a straightforward process 1 Restore a node of the cluster 2 Restore the cluster disks of the restored first node 3 Restore the remaining node of the cluster 4 Perform node testing 6 Module 7: Server Cluster Maintenance and Troubleshooting Restoring a Node of the Cluster To restore a node in a server cluster, you follow the same procedure that... Components 5 Clear the Cluster Service check box, and then click Next 6 Click Finish 7 Click Yes to restart the computer 29 30 Module 7: Server Cluster Maintenance and Troubleshooting Review Topic Objective To reinforce module objectives by reviewing key points Lead-in The review questions cover some of the key concepts taught in the module Cluster Maintenance Troubleshooting Cluster Service *****************************ILLEGAL... NTBackup does not restore the cluster files to the cluster disk NTBackup places the cluster files on the local node Module 7: Server Cluster Maintenance and Troubleshooting 27 To examine the cluster files that are restored by NTBackup 1 Click Start, and then click Run 2 In the Run dialog box, type %systemroot% \cluster and then click OK 3 Double-click the cluster_ backup folder to view the files that are.. .Module 7: Server Cluster Maintenance and Troubleshooting 5 Restoring the First Node Topic Objective To list the steps for restoring a server cluster and describe how to restore the first node Lead-in In the event of a complete cluster failure, you first restore a node Steps For Restoring a Server Cluster: 1 Restore the first node 2 Restore the cluster disks 3 Restore the... and the time of the restoration to the new computer may affect the computer account on the domain controller You may have to join a workgroup and then rejoin the domain Module 7: Server Cluster Maintenance and Troubleshooting 7 Restoring Cluster Disks Topic Objective To describe how to restore the cluster disk by restoring signature files, data and cluster configuration files Lead-in Restoring a cluster . 1 Cluster Maintenance 2 Troubleshooting Cluster Service 11 Lab A: Cluster Maintenance 24 Review 30 Module 7: Server Cluster Maintenance and Troubleshooting. Cluster maintenance in the form of backing up and restoring a cluster, and troubleshooting Cluster service. 2 Module 7: Server Cluster Maintenance and Troubleshooting