High Availability and Clustering • Chapter 6 225 Figure 6.39, we can see that member fw2 is active. If we right-click fw2 and select Stop Member, we will force fw1 to switch to active.This assumes that fw1—or another cluster member—is available. Be sure to check this status before stopping the current active member! Take note of the Running Mode field, which states whether the member is active or not. N OTE Note that if a member has been disabled using “Stop member,” the ClusterXL Details pane might still show the member as active. This is because we have lost contact with the ClusterXL module on that member, and the GUI is still displaying the last known status. It is worth checking the last updated time for the ClusterXL status and forcing an update (right-click ClusterXL and select Update). A stopped member can be revived by right-clicking the member name and selecting Start Member. Note that it will stay in Standby mode irrespective of its pri- ority if Maintain Current Active gateway is set in the cluster object. Test 3: FTP Session Through the Cluster When an Interface Fails As with all cluster solutions, the best tests are those simulating real-world failure. Physically damaging cluster members is probably the most challenging test but probably not a popular option, either.A more acceptable test is disconnecting a network cable from the current master member during a file download through the cluster. www.syngress.com Figure 6.39 SmartView Status GUI Showing ClusterXL HA New Mode with Member fw2 Active 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 225 226 Chapter 6 • High Availability and Clustering In our example, we initiate a command-line FTP session from the internal host on 192.l68.1.200 to 192.168.12.133 (refer to Figure 6.12).The default gateway of host 192.168.1.200 will be the cluster VIP address for that subnet (192.168.1.130).The default gateway for 192.168.12.133 will be VIP 192.168.12.130. We will use the ftp hash command in order to display the blocks downloaded so we can see the download’s progress. A large file should be chosen that will take at least a minute to download; that gives us time to test failover. If you pull out the external interface of the active member (for example, if member fw1 were active, removing the Ethernet cable from qfe0 would cause a fail condition), you should see member fw2 become active and the FTP session should continue, prob- ably after a pause of a few seconds.This particular test is useful because it tests the fol- lowing things: ■ The hosts communicating have the correct default gateway. ■ The hubs and switches are working correctly in an HA environment. ■ The firewall members are failing over correctly. ■ The hosts on the local subnet respond to the failover gratuitous arp. ■ The firewall members’ state tables are fully synchronized. Command-Line Diagnostics on ClusterXL Let’s take a look at some useful command-line tools that can be used to monitor ClusterXL. fw hastat The fw hastat command can be used to check the basic status of each cluster member locally or remotely.The fw hastat command has the following syntax: fw hastat <hostname / or IP address> A typical response if this command is run on a local firewall cluster member module is: HOST NUMBER HIGH AVAILABILITY STATE MACHINE STATUS localhost1 1 active OK cphaprob The cphaprob command is probably the most versatile command that can be used to monitor and manipulate ClusterXL. Here we cover just a few of the common syntaxes of this command, but it can do a lot more than merely show information about the www.syngress.com 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 226 High Availability and Clustering • Chapter 6 227 cluster.This command can be used in order to integrated tailored status checking— maybe checking hardware health of a member. The command can be used on either of the cluster members (not on the firewall management module). Running cphaprob stat on either of the firewall cluster members should tell you the status of each of the cluster members from the point of view of the cluster member you are running the command on. Here is an example output: Working mode: Active up (unique IPs) Number Unique Address State 1 192.168.11.132 active 2 (local) none* standby NOTE If you see none in the unique address for one of the cluster members, you need to reboot the module, then run the cphaprob state command again. It can also mean that the member is not correctly configured in the SmartDashboard GUI and that no secured interface exists on the member. You can also use this command with different arguments to provide details of inter- faces.The syntax for examining the interfaces on the local member is cphaprob -a if.The command will tell you the status of each interface and the virtual cluster IP addresses. In this example, the local cluster member is in Standby mode: Required interfaces: 3 Required secured interfaces: 1 hme0 UP (secured, unique) qfe0 DOWN (2505.8 secs) (non secured, unique) qfe2 UP (non secured, unique) qfe3 UP (non secured, unique) Virtual cluster interfaces: 3 qfe0 195.166.16.130 qfe2 192.168.12.130 qfe3 192.168.1.130 www.syngress.com 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 227 228 Chapter 6 • High Availability and Clustering In this example, we can see that the interface qfe0 is down—probably a cable or interface problem. Looking at the information further down, we see that qfe0 is associ- ated with the VIP address of 195.166.16.130, the external interface, so that is where we should start looking for network problems. Until this problem is resolved, we expect this member to stay in Standby mode; hopefully another member in the cluster will be active. cpstat ha The cpstat ha command gives detailed status details from the local member—similar information to that displayed by the SmartView Status GUI. Run without arguments, the output to this command is something like: Product name: High Availability Version: NG Feature Pack 3 Status: OK HA installed: 1 Working mode: High availability HA started: yes More usefully, you can use the syntax cpstat –f all ha to get this: Product name: High Availability Major version: 5 Minor version: 0 Service pack: 3 Version string: NG Feature Pack 3 Status code: 0 Status short: OK Status long: OK HA installed: 1 Working mode: High availability HA protocol version: 2 HA started: yes HA state: active HA identifier: 1 Interface table www.syngress.com 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 228 High Availability and Clustering • Chapter 6 229 |Name|IP |Status|Verified|Trusted|Shared| |hme0|192.168.11.131|Up | 0| 1| 0| |qfe0|195.166.16.131|Up | 500| 0| 0| |qfe2|192.168.12.131|Up | 0| 0| 0| |qfe3| 192.168.1.131|Up | 0| 0| 0| Problem Notification table |Name |Status|Priority|Verified|Descr| |Synchronization|OK | 0| 198| | |Filter |OK | 0| 188| | |cphad |OK | 0| 0| | |fwd |OK | 0| 0| | How Does ClusterXL HA New Mode Work? In HA New mode, on each member of the cluster, each interface that will share a VIP address will keep its existing MAC address. No additional shared MAC addresses are used. When a client that is on the nonsecured network ARPs for the virtual IP (which will be the client’s default gateway IP address), the cluster member that is active will reply with its MAC address and so will receive the through routed traffic. Note that a client should still be able to connect to any of the valid IP addresses of the cluster on the same local subnet, regardless of which member is active (assuming that the interface is not down, the OS hasn’t crashed, or the local firewall policy does not prevent it). Because all members are “live” but only one handles traffic, HA New mode could be seen as load sharing but with 100 percent of the traffic going through one member only and all other members on standby having 0 percent of the traffic.This is opposed to tradi- tional HA solutions in which the standby members are “offline” and unreachable. If we consider the diagram in Figure 6.40, we can see that member fw1 is active and fw2 is in Standby mode. www.syngress.com 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 229 230 Chapter 6 • High Availability and Clustering All network traffic should be routed through firewall member fw1—but only if its default gateway is set to the VIP address of 192.168.1.130. If we take an example in which host 192.168.1.200 initiates a connection to a host out on the Internet and we are using Hide NAT behind the external cluster IP of 195.166.16.130, it will first ARP for the default gateway IP address. Host fw1 should respond because it is the active member in the cluster, with its internal interface MAC address of 08:00:20:ca:64:fb.This will be put in the ARP cache table of host 192.168.1.200, and a TCP connection—source IP 192.168.1.200, destination IP = 216.238.8.44, destination MAC 08:00:20:ca:64:fb—will originate from the host. It is normal to use Hide NAT when internal hosts access the Internet, and this also makes it easy for replies to get back to your site. When the packet from host 192.168.1.200 leaves host fw1, the source IP will be address translated to 195.166.16.130 www.syngress.com Figure 6.40 Active Traffic Routing Through the Active Cluster Member fw1 fw2 Hub Hub Hub ISP Router PDC 192.168.11.131 hme0 MAC=08:00:20:94:20:67 192.168.11.132 hme0 MAC=08:00:20:a1:32:f3 195.166.16.131 qfe0 MAC= 08:00:20:ca:64:f8 195.166.16.132 qfe0 MAC= 08:00:20:a4:99:ec 192.168.1.131 qfe3 MAC=08:00:20:ca:64:fb 192.168.1.132 qfe3 MAC= 08:00:20:a4:99:ef 192.168.1.200 Default route = 192.168.1.130 Out to the Internet 195.166.16.129 Secured Network 192.168.11.0 /24 No VIP External Network 195.166.16.0/24 VIP = 195.166.16.130 Internal Network 192.168.1.0/24 VIP = 192.168.1.130 State table sync on Secured network ACTIVE STANDBY 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 230 High Availability and Clustering • Chapter 6 231 (and the source port will also change).The packet will then be routed out toward the ISP router (based on the default gateway of member fw1). The reply packet will come back through the ISP router, which will ARP for a MAC address for 195.166.16.130.The fw-1 member is active and will respond with its external interface MAC of 08:00:20:ca:64:f8.The ISP router adds this into its ARP cache and sends the reply packet for the session back to 195.166.16.130, MAC address 08:00:20:ca:64:f8. Member fw-1 then uses its stateful inspection to address translate the existing Hide NAT session so that the destination IP is changed from 195.166.16.130 to 192.168.1.200.The reply is then sent from interface qfe3 on fw1, source MAC address 08:00:20:ca:64:fb, to the host on 192.168.1.200. ClusterXL HA New Mode Failover On failover from the active cluster member to the standby member, adjacent routers and hosts still maintain the MAC address for the failed member in their ARP caches. Packets sent at this point arrive at the failed host and probably go no further.The cluster member that has just come active resolves this problem by issuing a “gratuitous ARP.”The ARP is broadcast on the local subnet of all interfaces that have a VIP and will have the MAC address for the local interface of the new active member in the cluster.This should mean that adjacent routers will learn the new MAC addresses for the VIP addresses. NOTE Under some circumstances in NG FP3, there is a problem where the cluster member that comes online does not always issue a gratuitous ARP. This should be resolved in hotfix releases. It is a good idea to obtain and apply the latest released hotfix. Let’s now look in detail at what happens if the active member fails. If we consider the diagram in Figure 6.40, we can see that traffic is routing through the active member, and Hide NAT is being done to hide the internal host of 192.168.1.200 behind the cluster IP address of 195.166.16.130. Should an interface fail (as shown in Figure 6.41, for example), all traffic from 192.168.1.200 will not be able to get through to the qfe3 interface on member fw1, and traffic that is coming back will not get back to 192.168.1.200 because the interface is down. At this point, fw2 will notice that fw1 is not responding on the qfe3 interface and will take note of this situation. If the interface stays down for a period of time, fw2 will start running its pre-online tests.These pre-online tests allow fw2 to determine if it is www.syngress.com 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 231 232 Chapter 6 • High Availability and Clustering healthy enough to take over from host fw1. (We discuss these tests in more detail in the “Nokia Failover Conditions” section of this chapter.) Once fw2 has determined that it is able to take over from fw1, it will issue a gratuitous ARP on all its interfaces that have a VIP address (see Figure 6.42). This will be cached by all devices on the local subnet of the interfaces of the fire- wall cluster, and they will update their ARP tables appropriately.This means that host 192.168.1.200 will now have a MAC address of 08:00:20:a4:99:ef in its ARP cache for IP address 192.168.1.130, so current through connections—perhaps an FTP session— should be able to continue. On the external interface of the cluster, the ISP router would also have received a gratuitous ARP, updating its ARP table for 195.166.16.130 with MAC address 08:00:20:a4:99:ec—the external MAC address of qfe0 of fw2. At this point, fw1 will www.syngress.com Figure 6.41 Interface Failure on Active Member fw1 fw2 Hub Hub Hub ISP Router PDC 192.168.11.131 hme0 MAC=08:00:20:94:20:67 192.168.11.132 hme0 MAC=08:00:20:a1:32:f3 195.166.16.131 qfe0 MAC= 08:00:20:ca:64:f8 195.166.16.132 qfe0 MAC= 08:00:20:a4:99:ec 192.168.1.131 qfe3 MAC=08:00:20:ca:64:fb 192.168.1.132 qfe3 MAC= 08:00:20:a4:99:ef 192.168.1.200 Default route = 192.168.1.130 MAC address used = 08:00:20:ca:64:fb Out to the Internet 195.166.16.129 Secured Network 192.168.11.0 /24 No VIP External Network 195.166.16.0/24 VIP = 195.166.16.130 Internal Network 192.168.1.0/24 VIP = 192.168.1.130 State table sync on Secured network ACTIVE STANDBY interface fail 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 232 High Availability and Clustering • Chapter 6 233 have considered itself offline in the SmartView Status GUI, stating that interface qfe3 is down. Once the FTP session recovers (which will only be the case if the gratuitous ARP is issued by fw2 and if state table sync is enabled between member fw1 and fw2), all traffic will continue to go through member fw2, as shown in Figure 6.43. Should member fw1 recover, the cluster can be configured to either fail back to fw1 (which will have the highest priority) or continue working through fw2, which could have a lower priority.This can be configured in the Cluster Gateway Object | Cluster XL Screen (see Figure 6.22). www.syngress.com Figure 6.42 Gratuitous ARP by fw2 to Take Over from fw1 on Failure fw1 fw2 Hub Hub Hub ISP Router PDC 192.168.11.131 hme0 MAC=08:00:20:94:20:67 192.168.11.132 hme0 MAC=08:00:20:a1:32:f3 195.166.16.131 qfe0 MAC= 08:00:20:ca:64:f8 195.166.16.132 qfe0 MAC= 08:00:20:a4:99:ec 192.168.1.131 qfe3 MAC=08:00:20:ca:64:fb 192.168.1.132 qfe3 MAC= 08:00:20:a4:99:ef 192.168.1.200 Default route = 192.168.1.130 MAC address updated from 08:00:20:ca:64:fb to 08:00:20:a4:99:ef for IP 192.168.1.130 Out to the Internet 195.166.16.129 Secured Network 192.168.11.0 /24 No VIP External Network 195.166.16.0/24 VIP = 195.166.16.130 Internal Network 192.168.1.0/24 VIP = 192.168.1.130 State table sync on Secured network FAIL ! ACTIVE interface fail Gratuitous arp IP=195.166.16.130 source MAC = 08:00:20:a4:99:ec Destination MAC = FF:FF:FF:FF:FF:FF Gratuitous arp IP=192.168.1.130 source MAC=08:00:20:a4:99:ef Destination MAC=FF:FF:FF:FF:FF:FF arp table updated for 195.166.16.130 with MAC address 08:00:20:a4:99:ec 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 233 234 Chapter 6 • High Availability and Clustering ClusterXL Failover Conditions There are a number of conditions in which failover from one member to another will occur.These are: ■ An interface or cable fails. ■ Security policy is uninstalled. ■ The machine crashes. ■ Any process or device that specified with the cphaprob command (such as the fwd process) fails. These conditions can be listed using the command cphaprob list. www.syngress.com Figure 6.43 ClusterXL in HA New Mode, with Maintain Current Active Gateway Set After Failover fw1 fw2 Hub Hub Hub ISProuter PDC 192.168.11.131 hme0 MAC=08:00:20:94:20:67 192.168.11.132 hme0 MAC=08:00:20:a1:32:f3 195.166.16.131 qfe0 MAC= 08:00:20:ca:64:f8 195.166.16.132 qfe0 MAC= 08:00:20:a4:99:ec 192.168.1.131 qfe3 MAC=08:00:20:ca:64:fb 192.168.1.132 qfe3 MAC= 08:00:20:a4:99:ef 192.168.1.200 Default route = 192.168.1.130 MAC= 08:00:20:a4:99:ef Out to the Internet 195.166.16.129 Secured Network 192.168.11.0 /24 No VIP External Network 195.166.16.0/24 VIP = 195.166.16.130 Internal Network 192.168.1.0/24 VIP = 192.168.1.130 State table sync on Secured network STANDBY ACTIVE interface back online 259_ChkPt_VPN_06.qxd 4/4/03 10:39 AM Page 234 [...]... 19 5 .16 6 .16 .13 0 VMAC= 01: 50 :5a:a6 :10 :82 ISP Router cpmgr 19 5 .16 6 .16 .13 4 19 5 .16 6 .16 .12 9 Hub 19 5 .16 6 .16 .13 1 eth-s1p1c0 MAC=00:c0: 95: e0 : 15 :dc 19 5 .16 6 .12 .13 1 eth-s1p3c0 MAC=00:c0: 95: e0 : 15 :de fw1 19 2 .16 8 .11 .13 1 eth-s1p2c0 MAC=00:c0: 95: e0 : 15 :dd 19 2 .16 8 .11 .13 2 eth-s2p2c0 Hub MAC=00:c0: 95: e2:b1: 41 State sync Network 19 2 .16 8 .1. 1 31 192 .16 8 .11 .0 /24 19 2 .16 8 .1. 132 eth-s1p4c0 MAC=00:c0: 95: e0 : 15 :df 19 5 .16 6 .16 .13 2... 19 5 .16 6 .16 .13 0 Hub no static manual proxy arp entries 19 5 .16 6 .16 .13 1 qfe0 MAC= 08:00:20:ca:64:f8 19 2 .16 8 .12 .13 1 qfe2 MAC=08:00:20:ca:64:fa 19 5 .16 6 .16 .12 9 fw1 19 2 .16 8 .11 .13 1 Hub hme0 MAC=08:00:20:94:20:67 ACTIVE DMZ Network 19 2 .16 8 .12 .0/24 VIP = 19 2 .16 8 .12 .13 0 19 5 .16 6 .16 .13 2 qfe0 MAC= 08:00:20:a4:99:ec 19 2 .16 8 .11 .13 2 fw2 hme0 MAC=08:00:20:a1:32:f3 Secured Network 19 2 .16 8 .11 .0 /24 No VIP 19 2 .16 8 .12 .13 2... entered on fw2 Hub Secured Network 19 2 .16 8 .11 .0 /24 No VIP 19 5 .16 6 .16 .13 1 qfe0 MAC= 08:00:20:ca:64:f8 19 2 .16 8 .12 .13 1 qfe2 MAC=08:00:20:ca:64:fa 19 5 .16 6 .16 .12 9 External Network 19 5 .16 6 .16 .0/24 VIP = 19 5 .16 6 .16 .13 0 fw1 19 5 .16 6 .16 .13 2 qfe0 MAC= 08:00:20:a4:99:ec 19 2 .16 8 .11 .13 2 19 2 .16 8 .11 .13 1 Hub fw2 hme0 hme0 MAC=08:00:20:a1:32:f3 MAC=08:00:20:94:20:67 19 2 .16 8 .12 .13 2 qfe2 MAC= 08:00:20:a4:99:ee Offline... 19 5 .16 6 .16 .13 2 eth-s2p1c0 MAC=00:c0: 95: e2:b1:40 fw2 eth-s2p4c0 MAC=00:c0: 95: e2:b1:43 Cluster control Network 19 2 .16 8 .12 .0/24 VIP = 19 2 .16 8 .12 .13 0 VMAC =1: 50 :5a:a8:c:82 Hub Internal Network 19 2 .16 8 .1. 0/24 VIP = 19 2 .16 8 .1. 130 VMAC= 01: 50 :5a:a8: 01: 82 Hub 19 2 .16 8 .1. 200 Default route = 19 2 .16 8 .1. 130 PDC www.syngress.com 19 2 .16 8 .12 .13 2 eth-s2p3c0 MAC=00:c0: 95: e2:b1:42 259 _ChkPt _VPN_ 06.qxd 4/4/03 10 :39 AM Page 253 High Availability... host route entry forcing the NAT IP address to forward to the cluster VIP For example, on our ClusterXL example, if you had a manual static destination NAT rule to NAT 19 5 .16 6 .16 .13 3 to 19 2 .16 8 .12 .13 3, you would add a route on the ISP router that looks something like: 19 5 .16 6 .16 .13 3 , netmask 255 . 255 . 255 . 255 gateway 19 5 .16 6 .16 .13 0 This states that to get to IP address 19 5 .16 6 .16 .13 3 as a host route,... |Trusted|Shared| -|hme0 |19 2 .16 8 .11 .13 1|Up | 0| 1| 0| |qfe0 |19 5 .16 6 .16 .13 1|Up | 0| 0| 0| |qfe2 |19 2 .16 8 .12 .13 1|Up | 0| 0| 0| |qfe3| 19 2 .16 8 .1. 1 31| Down | 32000| 0| 0| Problem Notification table -|Name |Status|Priority|Verified|Descr| -|Synchronization|OK | 0| 16 18| | |Filter |OK | 0| 16 18| | |cphad |OK | 0| 0| | |fwd |OK... DMZ Network 19 2 .16 8 .12 .0/24 VIP = 19 2 .16 8 .12 .13 0 Hub 19 2 .16 8 .12 .13 3 www Default route = 19 2 .16 8 .12 .13 0 This will cause undesirable effects in an HA environment.You have no easy way of determining which MAC address the ISP router will have cached for the IP address 19 5 .16 6 .16 .13 3 If you are unlucky, it could be the member that is in Standby mode www.syngress.com 259 _ChkPt _VPN_ 06.qxd 4/4/03 10 :39 AM Page... reason, Check Point suggest that New mode is used in FP3 installations www.syngress.com 259 _ChkPt _VPN_ 06.qxd 240 4/4/03 10 :39 AM Page 240 Chapter 6 • High Availability and Clustering Figure 6.46 Using Static Routes on the ISP Router for NATed IP Addresses Out to the Internet ISP Router Route added here to forward 19 5 .16 6 .16 .13 3 to gateway 19 5 .16 6 .16 .13 0 (the VIP) External Network 19 5 .16 6 .16 .0/24 VIP = 19 5 .16 6 .16 .13 0... fw1 # cphaprob state Working mode: Load Sharing Number Unique Address State 1 (local) 19 2 .16 8 .11 .13 1 active 2 19 2 .16 8 .11 .13 2 active fw1 # Note that both members have a state of active as opposed to active/standby in a ClusterXL HA New mode cluster Should there be a failure on one of the members, you would see something like: fw1 # cphaprob stat Working mode: Load Sharing Number Unique Address State 1. .. of 19 5 .16 6 .16 .13 3, which the firewall modules will have a NAT rule to translate to the real IP address of 19 2 .16 8 .12 .13 3 Figure 6. 45 Possible Scenario If Manual ARP Entries Are Used for NAT Out to the Internet ISP Router arp -s 19 5 .16 6 .16 .13 3 08:00:20:ca:64:f8 pub manually entered on fw1 ARP cache on ISP router ? arp -s 19 5 .16 6 .16 .13 3 08:00:20:a4:99:ec pub manually entered on fw2 Hub Secured Network 19 2 .16 8 .11 .0 . Router PDC 19 2 .16 8 .11 .13 1 hme0 MAC=08:00:20:94:20:67 19 2 .16 8 .11 .13 2 hme0 MAC=08:00:20:a1:32:f3 19 5 .16 6 .16 .13 1 qfe0 MAC= 08:00:20:ca:64:f8 19 5 .16 6 .16 .13 2 qfe0 MAC= 08:00:20:a4:99:ec 19 2 .16 8 .1. 1 31 qfe3 MAC=08:00:20:ca:64:fb 19 2 .16 8 .1. 132 qfe3 MAC=. NAT rule to NAT 19 5 .16 6 .16 .13 3 to 19 2 .16 8 .12 .13 3, you would add a route on the ISP router that looks something like: 19 5 .16 6 .16 .13 3 , netmask 255 . 255 . 255 . 255 gateway 19 5 .16 6 .16 .13 0 This states. the Internet 19 5 .16 6 .16 .12 9 Secured Network 19 2 .16 8 .11 .0 /24 No VIP External Network 19 5 .16 6 .16 .0/24 VIP = 19 5 .16 6 .16 .13 0 DMZ Network 19 2 .16 8 .12 .0/24 VIP = 19 2 .16 8 .12 .13 0 ACTIVE STANDBY no static