1-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. Lab 8.3.4.1: Troubleshooting Frame Relay 1 Frame Relay Atlas 550 192.168.192.0/24 1/2 2/1 S0/0 S0/0 DLCI 17 DLCI 18 .2 .4 Fa0/0 192.168.200.1/24 Fa0/0 192.168.232.1/24 London Singapore Objective Apply the troubleshooting method to a simple Frame Relay network problem. Scenario A new Frame Relay connection was installed between London and Singapore over the weekend. On Monday, both network operators at London and Singapore call to let you know they are unable to reach each other with this new Frame Relay connection. You check connectivity between the London router and the Singapore router and find you cannot ping or telnet between the sites. It is your responsibility to get London and Singapore communicating. Adtran Atlas 550 Note If using the Adtran Atlas 550 as your Frame Relay switch, you will see information about DLCIs (such as DLCI 16) that are not shown in sample outputs of this lab. These DLCIs are configured on the Atlas 550, but are not being used for this specific lab. You would not normally see these extra DLCIs so any information pertaining to them should be disregarded for this lab. Step 1 Cable the lab as shown in the diagram. Load the configuration file called Lab8-3-4-1-LondonBrokenConfig.txt into the London router. Load the file called Lab8-3-4-1- SingaporeBrokenConfig.txt into the Singapore router. 2-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. Step 2 Use ping and telnet to check connectivity between London and Singapore. All ping and telnet attempts from either site to the other should fail. From each router, ping its directly connected LAN interface. This ping should be successful. 1. Define the problem you are experiencing. Step 3 Gather the facts about the situation at hand. The commands listed below are useful for gathering factual information about each router. • show interfaces • show frame-relay lmi • show frame-relay pvc • show frame-relay map Gather information about the London router first. Console into the London router and issue a show interfaces serial 0/0 command. The highlighted information in the sample output below is useful for troubleshooting and should be the same as the information in your output. London#show interfaces serial 0/0 Serial0/0 is up, line protocol is up Hardware is PowerQUICC Serial Internet address is 192.168.192.2/24 MTU 1500 bytes, BW 128 Kbit, DLY 20000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation FRAME-RELAY, loopback not set Keepalive set (10 sec) LMI enq sent 152, LMI stat recvd 152, LMI upd recvd 0, DTE LMI up LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE Broadcast queue 0/64, broadcasts sent/dropped 33/0, interface broadcasts 20 Last input 00:00:01, output 00:00:01, output hang never Last clearing of "show interface" counters 00:25:24 Input queue: 0/75/0 (size/max/drops); Total output drops: 0 Queueing strategy: weighted fair Output queue: 0/1000/64/0 (size/max total/threshold/drops) Conversations 0/1/32 (active/max active/max total) Reserved Conversations 0/0 (allocated/max allocated) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 3-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. 187 packets input, 8033 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 1 input errors, 0 CRC, 1 frame, 0 overrun, 0 ignored, 0 abort 185 packets output, 8366 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions DCD=up DSR=up DTR=up RTS=up CTS=up Upon examining the output, everything looks fine here. Notice the output “LMI DLCI 0.” If the LMI type is “ANSI,” as it is in this case or Q933a, DLCI 0 is used for that local LMI usage. If the LMI type is “CISCO”, the DLCI would be 1023 for local LMI purposes. This is good Support Exam knowledge. Issue a show frame-relay lmi command on the London router. The highlighted information in the sample output below is useful for troubleshooting and should be similar to the information in your output. London#show frame-relay lmi LMI Statistics for interface Serial0/0 (Frame Relay DTE) LMI TYPE = ANSI Invalid Unnumbered info 0 Invalid Prot Disc 0 Invalid dummy Call Ref 0 Invalid Msg Type 0 Invalid Status Message 0 Invalid Lock Shift 0 Invalid Information ID 0 Invalid Report IE Len 0 Invalid Report Request 0 Invalid Keep IE Len 0 Num Status Enq. Sent 16 Num Status msgs Rcvd 16 Num Update Status Rcvd 0 Num Status Timeouts 0 Notice that the router and Frame Relay switch are exchanging LMIs, “Num Status Enq. Sent” and “Num Status msgs Rcvd.” When this is the case the DLCI and LMI type are most likely configured correctly. Issue a show frame-relay pvc command on the London router. The highlighted information in the sample output below is useful for troubleshooting and should be the same as the information in your output. (Remember to ignore information about other DLCIs.) London#show frame-relay pvc PVC Statistics for interface Serial0/0 (Frame Relay DTE) Active Inactive Deleted Static Local 1 1 0 0 Switched 0 0 0 0 Unused 0 0 0 0 <output omitted> DLCI = 17, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0 input pkts 1 output pkts 11 in bytes 30 out bytes 3683 dropped pkts 0 in FECN pkts 0 in BECN pkts 0 out FECN pkts 0 out BECN pkts 0 in DE pkts 0 out DE pkts 0 out bcast pkts 11 out bcast bytes 3683 pvc create time 00:02:54, last time pvc status changed 00:02:54 4-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. Notice the PVC for DLCI 17 is ACTIVE, which indicates that the PVC between the router and the switch is configured correctly. If the PVC status was inactive, it would most likely be a problem with the Frame Relay cloud or might also indicate a configuration problem between the Frame Relay cloud and the remote site. Issue a show frame-relay map command on the London router. The highlighted information in the sample output below is useful for troubleshooting and should be the same as the information in your output. (Remember to ignore information about other DLCIs.) Since everything else seems to be working correctly, you should expect to see a successful mapping of London’s DLCI to a remote IP address. London#show frame-relay map Serial0/0 (up): ip 0.0.0.0 dlci 16(0x10,0x400) broadcast, CISCO, status defined, active Serial0/0 (up): ip 192.168.192.4 dlci 17(0x11,0x410), static, broadcast, CISCO, status defined, active The DLCI has been successful mapped to the remote IP address via a “static” frame-relay map statement. “CISCO” identifies the Frame Relay encapsulation used with frame-relay map statement, which is the default Frame Relay encapsulation on Cisco routers. Listed below is a summary of pertinent factual information gathered about the London router. • The WAN interface serial 0/0 interfaces and line protocol are up; the encapsulation is Frame-Relay; the LMI is up; and the LMI type is ANSI. • LMIs are being sent and received so the router and Frame Relay switch are exchanging LMIs. • The PVC for DLCI 17 is active indicating the PVC between the router and switch is configured correctly. • The Frame Relay map is configured correctly. So far there does not appear to be any configuration problems on the London router. Now, gather information about the Singapore router. Console into the Singapore router and issue a show interfaces serial 0/0 command. 2. What pertinent factual information about the interface did you gather and is there any indication of a possible problem? (Hint: The information should be the same as gathered for the London router.) 5-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. Issue a show frame-relay lmi command. 3. What pertinent factual information about the LMI did you gather and is there any indication of a possible problem? (Hint: The information should be the same as gathered for the London router.) Issue a show frame-relay pvc command. 4. What pertinent factual information about PVCs did you gather and is there any indication of a possible problem? (Hint: The DLCI 17 PVC status should be “deleted”.) Note: If the DCLI status is INACTIVE it is usually a problem at the remote router. If the DLCI is INACTIVE the service provider’s switch is programmed to handle/forward frames that your router sends with that DLCI number, but it is not aware/informed that your destination router is ready to receive them. The destination router may be down, non-existent, or having Layer 1 problems. Also, if it is INACTIVE there might be an LMI type mismatch between the local router and the Frame Relay switch. Issue a show frame-relay map command. 5. What pertinent factual information about the frame-relay map did you gather and is there any indication of a possible problem? (Hint: The status should be “deleted”.) Step 4 6. Examine the facts you have just gathered. Based on your observations, what are the possible causes of the problem? (Hint: If the DLCI status is 6-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. DELETED, it is usually a problem at the local router. Compare the DLCI in the network diagram with the DLCI in your Frame Relay map statement.) Step 5 7. Create an action plan to solve this problem. Step 6 Implement the action plan. Based upon the network diagram and output of the show frame-relay map command on the Singapore router, you now know DLCI 17 is incorrect and it should be 18. However, issue a show running-config command on the Singapore router to see the entire frame-relay map statement, which should appear as shown below. interface Serial0/0 ip address 192.168.192.4 255.255.255.0 encapsulation frame-relay frame-relay map ip 192.168.192.2 17 broadcast frame-relay lmi-type ansi Remove the incorrect frame-relay map statement. Singapore(config-if)#no frame-relay map ip 192.168.192.2 17 broadcast Reenter the frame-relay map statement with the correct DLCI. It is a good idea to shutdown, and no shutdown the interface after entering the frame- relay map statement. Observe the results of your implementation. Verify the results of your change using the same commands you used to troubleshoot to make sure that it not only corrected the problem, but that it didn’t cause any new problems. • show interfaces • show frame-relay lmi • show frame-relay pvc 7-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. • show frame-relay map The outputs below are only from the commands that were showing problematic output. These commands will now show that the change has worked. Issue a show frame-relay pvc command on the Singapore router. Your output should appear similar to the sample output below. (Remember to ignore information about other DLCIs.) Singapore#show frame-relay pvc PVC Statistics for interface Serial0/0 (Frame Relay DTE) Active Inactive Deleted Static Local 1 1 0 0 Switched 0 0 0 0 Unused 0 0 0 0 DLCI = 16, DLCI USAGE = LOCAL, PVC STATUS = INACTIVE, INTERFACE = Serial0/0 input pkts 0 output pkts 0 in bytes 0 out bytes 0 dropped pkts 0 in FECN pkts 0 in BECN pkts 0 out FECN pkts 0 out BECN pkts 0 in DE pkts 0 out DE pkts 0 out bcast pkts 0 out bcast bytes 0 pvc create time 00:37:16, last time pvc status changed 00:37:16 DLCI = 18, DLCI USAGE = LOCAL, PVC STATUS = ACTIVE, INTERFACE = Serial0/0 input pkts 179 output pkts 71 in bytes 32724 out bytes 3618 dropped pkts 0 in FECN pkts 0 in BECN pkts 0 out FECN pkts 0 out BECN pkts 0 in DE pkts 0 out DE pkts 0 out bcast pkts 71 out bcast bytes 3618 pvc create time 01:23:46, last time pvc status changed 00:40:44 Issue a show frame-relay map command on the Singapore router. Your output should appear similar to the sample output below. Singapore#show frame-relay map Serial0/0 (up): ip 0.0.0.0 dlci 18(0x12,0x420) broadcast, CISCO, status defined, active Serial0/0 (up): ip 0.0.0.0 dlci 16(0x10,0x400) broadcast, CISCO, status defined, inactive Serial0/0 (up): ip 192.168.8.8 dlci 18(0x12,0x420), static, broadcast, CISCO, status defined, active 8. How are the output results different from the first show frame-relay pvc and show frame-relay map outputs on this router? 8-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. Use ping and telnet to check connectivity between London and Singapore. All ping and telnet attempts from either site to the other should now be successful. 9. Did your solution fix the problem? If not, undo your changes and repeat the process. Step 7 10. Document the results. Addendum Although the problem in this lab is now fixed, it is important to become familiar with some of the various debug outputs that could be helpful for troubleshooting other Frame Relay problems. Debug frame-relay lmi: This command displays the status enq. (inquiry) that your router sends out every 10 seconds, and the status message (type 1) that your router receives (in) from the Frame Relay switch every 10 seconds. Also, every 60 seconds the switch sends a FULL LMI message to your router (type 0), which includes a list of PVC DLCIs along with the status of each DLCI. London#debug frame-relay lmi 03:08:01: Serial0/0(in): Status, myseq 1 03:08:01: RT IE 1, length 1, type 0 03:08:01: KA IE 3, length 2, yourseq 23, myseq 1 03:08:01: PVC IE 0x7 , length 0x3 , dlci 17, status 0x0 03:08:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to down London# 03:08:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to up Some debug output has been deleted here. 03:09:41: Serial0/0(out): StEnq, myseq 11, yourseen 32, DTE up 03:09:41: datagramstart = 0x1059A94, datagramsize = 14 03:09:41: FR encap = 0x00010308 03:09:41: 00 75 95 01 01 01 03 02 0B 20 03:09:41: 03:09:41: Serial0/0(in): Status, myseq 11 03:09:41: RT IE 1, length 1, type 1 03:09:41: KA IE 3, length 2, yourseq 33, myseq 11 London# 03:09:51: Serial0/0(out): StEnq, myseq 12, yourseen 33, DTE up 03:09:51: datagramstart = 0x1059A94, datagramsize = 14 03:09:51: FR encap = 0x00010308 03:09:51: 00 75 95 01 01 01 03 02 0C 21 9-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. 03:09:51: 03:09:51: Serial0/0(in): Status, myseq 12 03:09:51: RT IE 1, length 1, type 1 03:09:51: KA IE 3, length 2, yourseq 34, myseq 12 London# 03:10:01: Serial0/0(out): StEnq, myseq 13, yourseen 34, DTE up 03:10:01: datagramstart = 0x1059A94, datagramsize = 14 03:10:01: FR encap = 0x00010308 03:10:01: 00 75 95 01 01 00 03 02 0D 22 03:10:01: 03:10:01: Serial0/0(in): Status, myseq 13 03:10:01: RT IE 1, length 1, type 0 03:10:01: KA IE 3, length 2, yourseq 35, myseq 13 03:10:01: PVC IE 0x7 , length 0x3 , dlci 17, status 0x2 Notice the LMI information being sent and received between the router and the Frame Relay Switch every 10 seconds. If LMIs are being sent (out) and received (in) properly, the myseq, yourseen and youseq fields should be incremented with every new LMI message. Other items to notice is that “DTE up” should be seen if the LMI type is configured properly and if LMI messages are being exchanged. When the status for DLCI 17 is changed from INACTIVE to ACTIVE the router will receive a FULL LMI message with “dlci 17, status 0x2” instead of “dlci 17, status 0x0” for INACTIVE. The FULL LMI messages are the ones that display the DLCI information (first and last messages in the output above). If the CIR was being sent from the provider, this would also be included in the FULL LMI, with “bw cir-value” after the status of the DLCI. Debug serial interface: You will see similar output to the debug frame-relay lmi command with the command debug serial interface. With both commands, notice that the LMI messages are sent and received every 10 seconds. London#debug serial interface Serial network interface debugging is on London# 00:36:06: Serial0/0(out): StEnq, myseq 33, yourseen 65, DTE up 00:36:06: Serial0/0(in): Status, myseq 33 London# 00:36:16: Serial0/0(out): StEnq, myseq 34, yourseen 66, DTE up 00:36:16: Serial0/0(in): Status, myseq 34 London# 00:36:26: Serial0/0(out): StEnq, myseq 35, yourseen 67, DTE up 00:36:26: Serial0/0(in): Status, myseq 35 London# 00:36:36: Serial0/0(out): StEnq, myseq 36, yourseen 68, DTE up 00:36:36: Serial0/0(in): Status, myseq 36 London# Debug frame-relay packet and debug frame-relay verbose: These commands display debugging information about the packets that are being received on the frame relay interfaces. Notice the different IP packet types of 0x2000 for CDP and 0x800 for ICMP pings. London#debug frame-relay packet Frame Relay packet debugging is on London# 03:47:13: Serial0/0: Broadcast on DLCI 17 link 65(CDP) 10-10 Semester 8 Internetwork Troubleshooting v1.0 - Lab 8.3.4.1 Copyright  2001, Cisco Systems, Inc. 03:47:13: Serial0/0(o): dlci 17(0x411), pkt type 0x2000(CDP), datagramsize 286 03:47:13: broadcast dequeue 03:47:13: Serial0/0(o):Pkt sent on dlci 17(0x411), pkt type 0x2000(CDP), datagramsize 286 London#ping 192.168.232.1 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.232.1, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 48/49/52 ms 03:47:30: Serial0/0(o): dlci 17(0x411), pkt type 0x800(IP), datagramsize 104 03:47:30: Serial0/0(i): dlci 17(0x411), pkt type 0x800, datagramsize 104 03:47:30: Serial0/0(o): dlci 17(0x411), pkt type 0x800(IP), datagramsize 104 03:47:30: Serial0/0(i): dlci 17(0x411), pkt type 0x800, datagramsize 104 03:47:30: Serial0/0(o): dlci 17(0x411), pkt type 0x800(IP), datagramsize 104 03:47:30: Serial0/0(i): dlci 17(0x411), pkt type 0x800, datagramsize 104 03:47:30: Serial0/0(o): dlci 17(0x411), pkt type 0x800(IP), datagramsize 104 03:47:30: Serial0/0(i): dlci 17(0x411), pkt type 0x800, datagramsize 104 03:47:30: Serial0/0(o): dlci 17(0x411), pkt type 0x800(IP), datagramsize 104 03:47:30: Serial0/0(i): dlci 17(0x411), pkt type 0x800, datagramsize 104 03:47:38: Serial0/0(i): dlci 17(0x411), pkt type 0x2000, datagramsize 286 Debug frame-relay packet and inverse-arp: In this lab, neither router relied on frame relay inverse-arp to resolve its local DLCI with the remote IP address. But, if inverse-arp had been used, debug frame-relay packet would have displayed this process. Below is output from a debug frame- relay packet showing an inverse-arp process and the resulting information displayed with show frame-relay map. Note: The local DLCI is 17 and the remote IP address is 192.168.8.9. London#debug frame-relay packet Frame Relay packet debugging is on London# 00:06:03: %LINK-3-UPDOWN: Interface Serial0/0, changed state to up 00:06:04: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, changed state to up London# 00:06:13: Serial0/0(o): dlci 17(0x411), pkt encaps 0x0300 0x8000 0x0000 0x806 (ARP), datagramsize 30 00:06:13: FR: Sending INARP Request on interface Serial0/0 dlci 17 for link 7(IP) 00:06:13: broadcast dequeue 00:06:13: Serial0/0(o):Pkt sent on dlci 17(0x411), pkt encaps 0x300 0x8000 0x0 0x806 (ARP), datagramsize 30 00:06:13: Serial0/0(i): dlci 17(0x411), pkt encaps 0x0300 0x8000 0x0000 0x806 (ARP), datagramsize 30 00:06:13: Serial0/0: frame relay INARP received London# 00:06:18: Serial0/0: Broadcast on DLCI 17 link 65(CDP) 00:06:18: Serial0/0(o): dlci 17(0x411), pkt type 0x2000(CDP), datagramsize 341 00:06:18: broadcast dequeue 00:06:18: Serial0/0(o):Pkt sent on dlci 17(0x411), pkt type 0x2000(CDP), datagramsize 341 London#show frame-relay map Serial0/0 (up): ip 192.168.192.4 dlci 17(0x11,0x410), dynamic, broadcast,, status defined, active . Frame Relay 1 Frame Relay Atlas 550 19 2 .16 8 .19 2.0/24 1/ 2 2 /1 S0/0 S0/0 DLCI 17 DLCI 18 .2 .4 Fa0/0 19 2 .16 8.200 .1/ 24 Fa0/0 19 2 .16 8.232 .1/ 24 London Singapore. 22 03 :10 : 01: 03 :10 : 01: Serial0/0(in): Status, myseq 13 03 :10 : 01: RT IE 1, length 1, type 0 03 :10 : 01: KA IE 3, length 2, yourseq 35, myseq 13 03 :10 : 01: PVC