LDP over RSVP-TE Tunnelling 431 LDP session are now associated with the TE tunnel and then used for traffic forwarding. When Paris advertises a label back to its local POP routers, then a SWAP/PUSH state on the forwarding plane is generated. The label of the TE tunnel is PUSHed as top level label and the label learned via the multi-hop LDP session is the SWAPed label. The edge routers send their traffic down the LDP established paths and do not even know that their traffic is being engineered in the core topology. As soon as the traffic arrives at the ingress of the TE tunnel (Paris), an additional label is PUSHed on top of the label stack and the traffic is sent down the TE tunnel. The penultimate TE tunnel router (New York) removes the top label and the LDP label underneath becomes visible and is used for fur- ther relaying traffic towards the LDP egress router. Configuration of LDP over RSVP-TE tunnelling is done using the mpls ldp neighbor <address> targeted ldp keyword in IOS and the ldp-tunneling keyword in JUNOS. IOS configuration In IOS LDP tunnelling is a global option which can be configured using the mpls ldp neighbor <address> targeted statement. London# show running-config [… ] mpls ldp neighbor 192.168.1.1 targeted ldp [… ] ! If the multi-hop LDP session comes up and there is an RSVP-TE tunnel to this des- tination, then the resolver will automatically set up the SWAP/PUSH state. In JUNOS LDP over RSVP-TE tunnelling is a property of the TE tunnel and is configured under the protocols mpls label-switched-path <name> {} stanza. JUNOS configuration In JUNOS the ldp-tunneling keyword automatically sets up a session between two ends of a TE tunnel. [edit] hannes@Frankfurt# show [… ] protocols { mpls { label-switched-path to-London { to 192.168.0.8; ldp-tunneling; } interface so-1/2/0; interface l00.0; } [… ] 432 14. Traffic Engineering and MPLS It is imperative that the loopback interface lo0.0 or interface all is listed when config- uring LDP tunnelling. LDP multi-hop sessions are sourced using the IP address of the lo0.0 interface. If it is not listed, then the tunnelled LDP session stays down. LDP over RSVP-TE tunnelling is a good example of how label stacking contributes to better scalability of the network. The LDP over RSVP-TE tunnelling example just needed to set up one additional forwarding state at the TE tunnel ingress router. The rest of the core topology was unaffected by the LDP tunnelling change. An additional advantage of the clear layering is that once the tunnel goes down, immediately alternate paths (that is, the LDP-only paths that are available) are available. Also, the churn for changing the label state is almost zero, because only the TE ingress routers need to change forwarding state to use the IGP guided paths. Unfortunately LDP over RSVP-TE tunnelling does not solve the label selection issue for all topologies. Typically, it only attracts traffic being sourced from directly attached routers in the POP. For any edge router that is at least 2 hops upstream, it is not possible to force traffic onto a certain path. Figure 14.20 illustrates the problem. The links from Paris to Washington, Washington to New York, and Paris to Frankfurt are congested. Frankfurt and Paris are major traffic sources. There is a TE tunnel between Frankfurt and New York (1) – LDP tunneling is turned on. Now, all the Frankfurt POP traffic is using the tunnel. What would be best is to also attract the traffic from Paris. But this is not possible in this topology because Paris selects the label switched path to New York via Washington, which is the shortest path. In this simple topology the easiest fix would be to install another TE tunnel from Paris to New York. However, in complex topologies, often the administrative overhead of man- aging several local tunnels outweighs the convenience of having fewer tunnels to manage. It would be nice if there was a tool where networks could gradually suck traffic to the RSVP 2 1 LDP POP POP POP POP POP Frankfurt London Washington NewYork Paris 120% load 95% load 25% load4 4 4 6 4 1-10 30% load 20% load 110% load4 2 FIGURE 14.20. For traffic engineering of upstream routers forwarding adjacencies need to be configured LDP over RSVP-TE Tunnelling 433 head-end of a TE tunnel. But to affect non-local forwarding decisions, the network needs to find a way to modify the route computation. And there is one. Forwarding adjacencies are a way of re-advertising a label switched path in the IS-IS database. 14.7 Forwarding Adjacencies The Edge MPLS routers, which speak LDP, have to rely on the IGP (IS-IS) to find the shortest path to the destination. Recall that the general problem of traffic engineering is that the shortest path is not always the best path. The tunnel must be made somewhat attractive to the edge systems’ traffic. One way of doing this is to model the core TE tunnel as a direct link and make the tunnels cost a little better than the resulting IGP cost. Because of that slight difference, the edge systems will prefer to load traffic onto the tunnel. A decade ago it was common to run IGPs over a tunnel. But running dynamic routing protocols over a tunnel is almost always a recipe for disaster. Things behave really badly if the total IGP cost over the tunnel undermines the total topologies’ cost. What happens next is that the tunnel “wraps” around itself, ultimately causing a meltdown of the entire network. Having those glorious meltdowns in mind, designers put a few restrictions on re-advertising a TE tunnel as part of the IS-IS topology. First of all, no IS-IS Hellos are sent down a tunnel. The router considers this forwarding adjacency to be up when the tunnel is up. If there is a change in topology and the tunnel goes down, then the forwarding adja- cency will go down as well. Because no Hellos are sent down the tunnel there is no infin- ite recursion problem as there was when tunnelling IGPs in the 1990s. Still, there are some things to watch out for. If the cost of the forwarding adjacency becomes too low (that is, more attractive to the rest of the topology) then too much traffic is sucked towards that tunnel. This could even totally mess up the IGP routing. Reconsider Figure 14.20. If the TE tunnel is advertised with IS reach information in the IS-IS database, then it seems as if there is now a direct, additional link between Frankfurt and New York (2). The nice thing is that the metric of this “virtual” adjacency can be configured arbitrarily. It can be set to a metric of 10, which makes the link totally unattractive because there are shorter paths available. However, if the forwarding adja- cency metric is set (for instance) to 1, then even non-local traffic is sucked into the tun- nel, including all the POP traffic from (for example) Paris. Depending on the IGP metric design, the power of forwarding adjacencies can do severe damage to the network. Consider the IGP metric proposal in Figure 12.10. A metric of 1 is not used today, mainly to leave some headroom for high speed links like OC768/STM-256 pipes. But if the advertised metric of the tunnel is 1, even regional traffic between cities can be sucked across the Atlantic. A common design rule is to keep the IGP cost slightly above the cost of the real topology, and it should not exceed the typical link-metric inside the POP. The idea is to suck the entire POP destinations across the tunnel, but keep the sucking- distance low enough not to affect other region’s traffic. In IOS forwarding adjacencies are a property of a TE tunnel and can be configured using the tunnel mpls traffic-eng forwarding-adjacency parameter in the tunnel interface configuration. 434 14. Traffic Engineering and MPLS IOS configuration In IOS you need to tell the tunnel interface that it has to re-advertise the TE tunnel into IS- IS using the tunnel mpls traffic-eng forwarding-adjacency statement. Additionally the resulting IS-IS metric needs to be specified using the regular isis metric <*> statement. London# show running-config [… ] interface Tunnel0 mpls traffic-eng tunnels tag-switching ip tunnel mode mpls traffic-eng tunnel mpls traffic-eng forwarding-adjacency isis metric 200 level-2 ! In JUNOS, a forwarding adjacency is an IS-IS property and is configured under the protocols isis label-switched-path {} stanza. JUNOS configuration In JUNOS you need to reference a valid label switched path which needs to exist under the protocols mpls {} stanza plus the IS-IS level and metric. hannes@Frankfurt> show configuration [… ] protocols { isis { [… ] label-switched-path Paris-to-London { level 2 metric 200; } } } [… ] How do the other routers know that an IS-IS adjacency is real (over physical links) or the result of a forwarding adjacency (over a TE tunnel)? In order not to run into recursive tun- nel loop problems, there is a differentiation. If you consider the tcpdump output, then you can easily see the difference between a physical link adjacency and a forwarding adjacency. Tcpdump output A forwarding adjacency enabled IS reachability information does not carry any traffic engi- neering sub-TLVs. 00:28:20.760649 OSI, IS-IS, length: 156 hlen: 27, v: 1, pdu-v: 1, sys-id-len: 6 (0), max-area: 3 (0), pdu-type: L2 LSP lsp-id: 1921.6800.1014.00-00, seq: 0x0000df31, lifetime: 1196s chksum: 0x2674 (correct), PDU length: 156, L1L2 IS [… ] Hostname TLV #137, length: 5 Hostname: Paris Extended IS Reachability TLV #22, length: 86 physical link → IS Neighbor: 1921.6800.1008.00, Metric: 10, sub-TLVs present (64) IPv4 interface address subTLV #6, length: 4, 172.16.0.2 IPv4 neighbor address subTLV #8, length: 4, 172.16.0.1 Unreserved bandwidth subTLV #11, length: 32 priority level 0: 2488.320 Mbps priority level 1: 2488.320 Mbps priority level 2: 2488.320 Mbps priority level 3: 2488.320 Mbps priority level 4: 2488.320 Mbps priority level 5: 2488.320 Mbps priority level 6: 2488.320 Mbps priority level 7: 2488.320 Mbps Reservable link bandwidth subTLV #10, length: 4, 2488.320 Mbps Maximum link bandwidth subTLV #9, length: 4, 2488.320 Mbps Administrative groups subTLV #3, length: 4, 0x00000000 Forw. Adjacency → IS Neighbor: 1921.6800.1012.00, Metric: 200, no sub-TLVs present Extended IPv4 Reachability TLV #135, length: 18 IPv4 prefix: 62.0.0.1/32, Distribution: up, Metric: 0 IPv4 prefix: 172.16.0.0/30, Distribution: up, Metric: 10 The forwarding adjacency gets advertised as simply Extended IS Reach Adjacency with no sub-TLVs at all attached to it. Therefore, the adjacency does not get moved to the TED. It is almost as if this virtual “link” does not exist for the TED. If a link does not exist in the TED, then no adjacency can be established over it, and the tunnel recursion prob- lem is fixed. One of the key concepts of forwarding adjacencies is that the resulting vir- tual link should always be worse than a real link. Chapter 17, “Future of IS-IS”, will extend the forwarding adjacency concept to several switching layers and examine how forwarding adjacencies can be utilized for G-MPLS applications. Forwarding adjacencies are a nice tool to offload traffic from the shortest path with minimal configuration and maximum impact. However, one problem remains: if the path’s physical characteristics change, the delay characteristics of that path may also change. In order to modify traffic paths for only some classes of traffic, DiffServ Traffic Engineering needs to be deployed. 14.8 DiffServ Aware Traffic Engineering Originally, traffic engineering was used to offload just best-effort traffic. This was fine, because at that time, only best-effort traffic was routed. In recent years, however, there has Forwarding Adjacencies 435 been a shift from routing pure best-effort traffic toward multi-service networks,which could route voice, video and data. It turned out that the granularity of traffic engineering may be a bit too coarse, because it affects all the traffic between a pair of locations and does not offer a per-class granularity that would take voice traffic over a different path than for best-effort traffic. Consider Figure 14.21 for an illustration of the dilemma. There is a capacity problem between Frankfurt and Washington, so a TE tunnel between via London is applied. However, now the throughput was optimized at the expense of delay. Now voice traffic from London to Washington has an extra delay of 20 ms due to the traffic-engineering imposed delay. The good news is that typically voice applications do not consume much bandwidth. What is needed is to keep the London to Washington voice traffic untouched, and only route best-effort traffic across the alternate path. DiffServ TE allows the network operator to engineer class-specific LSPs. In the above example, the traffic engineer could leave voice traffic on the direct link between Frankfurt and Washington and make sure that, as long the traffic does not violate the traf- fic contract, it gets through even if the interface is 120 per cent congested in the best- effort class (this means bandwidth guarantees get enforced). DiffServ TE-capable software has not yet been released by the two major router vend- ors. Meanwhile, there are pre-standard implementations available, but given that the IETF has not yet agreed on a single TE standard and the necessary code-points, it will probably take another year until interoperable software is available. 14.9 Changed IS-IS Flooding Dynamics IS-IS is the driver behind the TED. It carries dynamic information about the current net- work reservation state. In order to make good routing decisions one needs to make sure that a change in bandwidth reservation is propagated as quickly as possible. On the other hand, frequent LSP updates stress the flooding sub-system. Vendors need to make sure 436 14. Traffic Engineering and MPLS RSVP 45% load 40% load 30% load 80% load4 4 4 6 4 120% load4 20% load Frankfurt London Washington NewYork Paris 1 2 FIGURE 14.21. Bandwidth utilization optimization often makes delay characteristics of a path worse that in the event of many LSPs flapping (down/up), the update in reservation does not churn the network. JUNOS and IOS support throttling mechanisms in order to hold down quick reservation changes. In IOS, those throttling timers are configurable on a per inter- face basis. JUNOS Configuration Snippet In JUNOS the update-threshold parameter to control IGP updates based on reservation is a function of the rsvp interface {} stanza. hannes@Frankfurt# show configuration [… ] protocols { rsvp { interface so-7/0/0.0 { update-threshold 10; } } } You can verify your setting by issuing the show rsvp interface <if-name> detail command. 14.10 Conclusion IS-IS has become far more than just an IP routing protocol. It is now being used for traffic engineering purposes by sending a lot of extra topology-related information like band- width, link colours and additional TE metrics. All this information is conveyed using sub- TLVs to the Extended IS Reach TLV #22 and stored in a unified form in the TED. Based on the TED contents, Explicit Route Objects (ERO) are pre-determined, which will then be embedded in RSVP-TE path messages. The calculations are done using a CSPF calculation, which is a modified version of the SPF algorithm. The constrained SPF algorithm first prunes off links that do not follow certain constraints before doing a regular SPF. Setting up just the tunnel is not enough; one needs to care also about loading traffic on the tunnel. Re-advertising TE tunnels back into IS-IS is the way to attract enough traffic from other hops further downstream from the head of the TE tunnel. Optimizing for the best network utilization may not be desirable, especially for voice traffic, which requires low and stable delays. DiffServ TE allows the calculation and setup of per-traffic-class tunnels for opti- mizing the different goals of each class: best-effort traffic is optimized for bandwidth utilization and voice traffic is optimized for the shortest delay. Changes in bandwidth reservations also cause an extra amount of LSP updates, which need to be taken into account for assessing the overall scalability of large networks. Meanwhile, code supporting traffic engineering has been in the field now for 3–4 years and is in a mature state now. DiffServ Traffic Engineering Code has not been deployed yet in production networks, but is expected to be the “Swiss Army knife” of traffic engineering, fulfilling the purpose for all traffic types. DiffServ Aware Traffic Engineering 437 15 Troubleshooting 439 Now you know almost everything there is to know about IS-IS and you can go and con- figure it in your network. Seriously, IS-IS is quite easy to configure on router OSs. Most likely, a pair of routers will immediately start exchanging PDUs and then do an SPF cal- culation on the common set of entries in the IS-IS link-state database. If they don’t, well, just continue reading. This chapter is all about IS-IS troubleshooting. First, this chapter will develop a common flowchart-like methodology that allows us to hunt down almost any IS-IS related problem. Next, the chapter explores the troubleshooting tools that are available on the router OS for gathering the information required by the troubleshooting methodology. Finally, the chapter will end by exploring real-world case studies of IS-IS- related problems in the areas of adjacency management, database overloading, and SPF calculations. 15.1 Methodology Any good troubleshooting process starts with a step-by-step plan and an iterative comparison between the desired state of the network and the actual state. For example, it does not make sense to examine the contents of the link-state database if no adjacency is up to supply the LSDB with data. The starting assumption for this troubleshooting plan of attack is that the router has a working physical circuit, free of bit errors and other transmission-related faults. Figure 15.1 illustrates a generic troubleshooting plan for IS-IS problems. The heart of the plan is an iterative comparison between results from the router’s observed operation to the intended behaviour, typically the behaviour defined in the configuration file. In the illustrations in this chapter, configuration-related information (what the router should be doing) is indicated by light bubbles. Facts gathered from command outputs confirms what the router is actually doing, which is indicated by dark bubbles. You start a troubleshooting session by verifying that adjacencies to other routers are up. If they are not, then there are three main sources of trouble. First, the problem could be related to the interface configuration (1). For example, an interface might not be included in the router’s IS-IS interface list or it carries wrong IS-IS level information. The problem could even be buried at lower layers in the OSI reference stack: too small a protocol MTU or lack of OSI-CP support on point-to-point interfaces prevent IS-IS con- trol planes from exchanging any information. Next, a good place to look is the address- ing information like IPv4, IPv6 and OSI NLRIs. Most importantly, you first need to look 440 15. Troubleshooting OK OK OK Troubleshooting Start Problem resolved Verify Interface Config Verify Adjacency Table Verify IPv4/IPv6/OSI Protocol Config Verify Topology Config Verify Link-State Database Verify Protocol Preference Config Verify SPF results Verify Authenti- cation Config Verify IP Routing Table 1 2 3 4 FIGURE 15.1. At the heart of the troubleshooting process are iterations between comparisons of configuration information and the actual state of the network at the OSI router configuration, because that encompasses IS-IS Area- and System-ID configuration. If the System-ID and or Area-IDs are not configured, or wrongly defined, then any IS-IS network is condemned to failure. Unfortunately, adjacency management is no longer a protocol-neutral function in IS-IS. Many IS-IS implementations require a match for the IPv4 or IPv6 sub-nets in order to complete the IS-IS adjacency handshake. Also, a broken authentication configuration is a frequent reason why an adjacency does not come up. The next major step is to check to see if the link-state database is filled with informa- tion (2). The major stumbling blocks here are duplicate System-IDs and mismatched authentication strings. After the router executes the SPF calculation, it shows you the cal- culated topology and the attached prefixes. If a node is missing in the SPF result list (3), we examine the link-state database to check for node isolation or unidirectional links and may even spot a problem on a remote router in the network. In the age of multi-topology IS-IS, it may be necessary to revisit the topology information and check to see if a given interface is in the required topology. A double check in the link-state database helps to spot missing IS Reachability Announcements and the like. If a node finally shows up in the SPF result list, and it has attached prefixes, those prefixes need to compete with prefixes from other routing protocols to become active routes. Because of its role as a topology discovery tool, IS-IS does not carry customer routes, but rather infrastructure routes. So IS-IS routes are typically preferred to routes from other routing protocols like BGP. However, in IOS, for example, external BGP paths have an administrative dis- tance, which has a route preference of 20 versus an IS-IS route that ranks only at distance 115 of the route selection process. Particularly during routing protocol migrations (for example from EIGRP to IS-IS) it may be the case that the other routing protocol is still masking the routes from the new routing protocol. A modification of the protocol prefer- ence configuration helps to unveil the IS-IS routes. If the IP routes finally show up in the main IP routing table, the troubleshooting process may end. In our explanation of the troubleshooting process, we already mentioned two tools for information gathering: Examining the router configuration and looking at the output of show commands are important tools during the troubleshooting process. In the next sec- tion we will explore those tools and more, tools that help operators examine, understand and resolve IS-IS problems. 15.2 Tools IS-IS troubleshooting quality and the accuracy of problem assessments is based on one main factor: the tools that supply the network engineer with information. Based on the information derived from good tools, the network engineer makes a decision regarding the parts of the configuration and the environment that are broken. The first and most important troubleshooting tools are the show commands that the router OS offers. Show command outputs are used on a daily basis and therefore it is imperative that each network engineer knows the meaning of each output field of these commands. Tools 441 . in the core topology. As soon as the traffic arrives at the ingress of the TE tunnel (Paris), an additional label is PUSHed on top of the label stack and the traffic is sent down the TE tunnel. The. longer a protocol-neutral function in IS-IS. Many IS-IS implementations require a match for the IPv4 or IPv6 sub-nets in order to complete the IS-IS adjacency handshake. Also, a broken authentication. that the other routing protocol is still masking the routes from the new routing protocol. A modification of the protocol prefer- ence configuration helps to unveil the IS-IS routes. If the IP routes