670 DEPLOYMENT CONSIDERATIONS Figure 13.1 A typical carrier backbone network based on SONET/SDH, showing SONET/SDH add/drop multiplexers (ADMs) and digital crossconnects (DCSs), along with optical line terminals (OLTs) and routers. (a) The network topology, which consists of interconnected rings in the backbone, with feeder metro rings. (b) Architecture of a typical node, including OLTs, stacked up SONET ADMs, and DCSs. 13.1 The Evolving Telecommunications Network 671 Figure 13.2 Bandwidth wasted when two rings built using ADMs share the same fiber route. Half the bandwidth on each ring along the shared route is reserved for protection. This network was designed primarily to carry voice and private line traffic. The network provides guaranteed latency and bandwidth, and well-established protec- tion schemes ensure high network availability. SONET/SDH also provides extensive performance monitoring and fault management capabilities. The network is mostly static, with switching provided by the DCSs in order to provision connnections. The switching is done at the time a connection is set up. Once set up, connections remain for months or years, but may have to be switched in the interim to deal with network failures or for maintenance purposes. However, as we see the increasing dominance of data traffic and the emergence of new optical layer equipment, several deficiencies of the SONET/SDH-based network architecture become evident: 9 It consists primarily of static rings where capacity is provisioned in a static manner. It does not allow the rapid provisioning of services end to end across the network on time scales of tens of milliseconds (for fast protection switching) to seconds (for rapid provisioning). 9 The traffic demands themselves are more meshed, and the ring architecture is not the most efficient at supporting an inherently meshed traffic demand for several reasons. Multiple rings need to be interconnected, and the interconnection is fairly complex and done through digital crossconnects. Half the capacity on each ring in the network is reserved for protection. Moreover, if two rings share a common link, as shown in Figure 13.2, the protection capacity is reserved for each ring separately along the overlapping link. This may be useful if the network needs to protect against multiple simultaneous failures, but is otherwise wasteful. 672 DEPLOYMENT CONSIDERATIONS 9 By default, all the traffic is protected. This does not allow carriers to offer a variety of services, some protected and others not protected. Protection is not needed for certain types of traffic, for instance, best-effort IP traffic. 9 Data traffic is increasingly entering the network at higher and higher speeds. If IP routers are hooked into the tributary ports of a SONET/SDH box, the SONET/SDH equipment may need to operate at a higher line speed than the IP router ports. For instance, IP touters with OC-48c ports may need to be connected into OC-192 SONET ADMs. (Recall from Chapter 6 that the "c" stands for concatenated.) This is usually because the SONET equipment is designed to support tributary interfaces at lower speeds than the line interfaces. Also some variants of SONET, such as a two-fiber BLSR, reserve half the bandwidth on each fiber line for protection. For example, a two-fiber OC-48 ring carries only an OC-24 worth of working traffic on each fiber, and SONET does not provide for splitting up a concatenated SONET stream. This situation presents a problem because the IP routers are increasingly able to support ports at rates comparable to SONET/SDH line rates. Thus an IP router with OC-192c ports may need to be mapped into an OC-768 SONET ADM. The rates coming out of IP router ports may soon exceed the line rate of SONET/SDH ADMs. For this reason, it makes sense to connect IP routers with high-speed ports directly into the optical layer without going through intermediate SONET/SDH equipment. [] Some carriers are in the business of delivering high-speed, best-effort IP services. For these carriers, the SONET/SDH layer doesn't provide much of a benefit. The multiplexing and protection offered by the SONET/SDH layer is not needed. Thus significant cost can be saved by eliminating the SONET/SDH equipment for these applications. Note, however, that SONET framing still offers signifi- cant advantages: it provides a commonly used set of transport rates and provides sufficient overheads to allow detailed performance monitoring and fault man- agement. For this reason, while SONET multiplexing and protection may not be required in IP or ATM networks, SONET framing is still widely used by IP and ATM equipment. 9 SONET/SDH does not provide efficient mapping for many signals used in data networks. For example, transporting a 100 Mb/s Ethernet signal across the coun- try requires leasing a 155 Mb/s OC-3 line. 9 Finally, today, carriers lack the management and signaling systems in order to be able to provision connections end to end across their network. The current situation is that different network elements are managed by different manage- ment systems, and provisioning connections on systems already fully equipped is a time-consuming and rather manual process. For instance, each SONET ADM and DCS in the network is provisioned separately, one at a time, using element 13.1 The Evolving Telecommunications Network 673 I IP ] Private lines I I I I OSl voice SONET Optical layer ~' OLT/OADM/OXC Optical DS3 OC-3c L OC-12c i OC_12 c IP router ~1 SONET Private lines ADM/DCS Voice ~ DS0 ~! (a) (b) Figure 13.3 Using SONET/SDH as the common transmission layer. IP packets are encapsulated into PPP frames for link layer functions and then mapped into SONET/SDH frames for transmisison over the fiber. The bit rates indicated are for illustration purposes only. (a) The logical layered view. (b) Example of how equipment is interconnected. 13.1.2 management systems. While there are some umbrella network management sys- tems that do provision end-to-end connections, these still provide limited inter- operability across equipment from multiple vendors. We saw in Section 9.6.2 that signaling standards are being developed to solve this problem. For these reasons, the network architecture is changing in some rather significant ways. The best architecture depends to a large extent on the service mix offered by the carrier, and also the legacy network that is in place in the current network. We will next describe the choices facing carriers as they plan their next-generation transport networks. Architectural Choices for Next-Generation Transport Networks The optical layer has emerged as the main transmission layer for telecommunications backbone networks. The real debate is about what set of technologies to use above the optical layer to deliver services. This in turn decides the set of boxes that will need to be deployed at the network nodes. The choices today include SONET/SDH, IP, and ATM. Figures 13.3, 13.4, and 13.5 show a variety of options available to carriers planning their next-generation networks. Figure 13.3 shows the SONET/SDH layer as the common transmission layer above the optical layer. Other services, including ATM and IP, are carried over 674 DEPLOYMENT CONSIDERATIONS I IP ] Private lines Voice ATM + SONET or wrapper framing Optical IP router Private lines Voice ATM switch gC-12c/OC-48c] Optical layer OLT/OADM/OXC (a) (b) Figure 13.4 Using the ATM layer as the common service layer. IP and other services are brought into the ATM layer. The ATM switches are directly connected to optical layer equipment. The ATM switches embed their cells in frames, typically using SONET/SDH framing. The bit rates indicated are for illustration purposes only. (a) The logical layered view. (b) Example of how equipment is interconnected. IP I PPP I SONET or wrapper framing Optical LOC-48c/OC- 192c[ Optical layer IP router r OLT/OADM/OXC (a) (b) Figure 13.5 Using the IP layer as the common service layer. The routers use a framing protocol to embed the packets before they are transmitted over the optical layer. The bit rates indicated are for illustration purposes only. (a) The logical layered view. (b) Example of how equipment is interconnected. 13.1 The Evolving Telecommunications Network 675 the SONET/SDH layer. Figure 13.3(a) shows a logical view of the layers, while Figure 13.3(b) shows how the equipment is interconnected in a typical configuration. IP packets are typically carried over a link layer protocol such as PPP (point-to-point protocol), which provides link-level integrity of the frames on a link-by-link basis. These packets are then framed into SONET/SDH frames. All these functions are performed by a line card inside the router. The router is connected to a SONET/SDH box, which multiplexes this connection along with others for transmission over the optical layer. We have already pointed out the deficiencies of this architecture. For these rea- sons, it is becoming clear that SONET/SDH will not remain the core transmission layer for much longer. Rather, SONET/SDH will be moved toward the edge of the network and used to multiplex lower-speed circuit-switched lines and bring them into the optical layer. Other network elements such as IP routers and ATM switches will also bring in traffic into the optical layer. Figure 13.4 shows a model where ATM is used as the common link layer (layer 2) with all services riding above the ATM layer. In this case, the ATM switches are directly connected to optical layer equipment. The ATM switches need to use a framing protocol to embed the cells before transmitting them over the optical layer. The framing protocol allows the data to be formatted for transmission over a physical link and allows various overheads to be added for management purposes. SONET/SDH framing is widely used because of its superior management capabilities and because chipsets are widely available. The framing is done at the line cards sitting inside the ATM switch, rather than requiring a separate SONET/SDH box. The ATM layer is relatively more mature than the IP layer in terms of providing quality-of-service (QoS) guarantees such as latency and bandwidth. For example, carriers can deliver guaranteed-bandwidth "virtual" DS 1 and DS3 services using their ATM networks with a technique known as circuit emulation. Protection switching if needed can be provided by the SONET/SDH layer or the optical layer. Several carriers made heavy investments in ATM and are committed to that approach in the near term. In many cases, IP traffic is carried through a frame relay interface into an ATM network. This option is cheaper than having IP routers use private lines such as DS1/DS3 because the frame relay equivalents for these services are less expensive and best-effort traffic doesn't need the QoS guarantees that private lines offer. Another reason for using the ATM layer to carry IP traffic is that ATM virtual circuits can be set up to provide virtual direct connections between routers. For example, in Figure 13.6, traffic flowing from router A to router C can be routed at intermediate node B through the ATM layer without passing through another router. This helps improve the QoS for the IP traffic. The IP layer itself is currently being enhanced to provide this capability using multi-protocol label switching (MPLS). 676 DEPLOYMENT CONSIDERATIONS IP router I I I ATM switch ATM switch IP router 'l[ ATM switch Node A Node B Node C Figure 13.6 An example to illustrate the use of direct virtual connections between routers using the ATM layer. The router at node A sees a direct virtual connection with the router at node C, without having to go through the router at node B. Thus, over time, the need for having an ATM intermediate layer would seem to disappear. Figure 13.5 shows a model where the IP layer resides directly on top of the optical layer. The IP layer classically belongs to layer 3 of the OSI hierarchy. With the advent of MPLS, the IP layer also includes layer 2 functionality. In this case, IP routers are directly connected to optical layer equipment. In the wide-area network, SONET/SDH framing is widely used for the reasons given above, and the framing is done on line cards within the router. It is important to note that in this case, there is no need for a separate SONET/SDH box in the network, which can translate into significant cost savings. Two other framing techniques are also emerging. The first is based on Ethernet, or more specifically, Gigabit and 10-Gigabit Ethernet. The other is based on the digital wrapper standard that we studied in Chapter 9. Ethernet-based framing is likely to proliferate as high-speed Ethernet becomes widely deployed in metro networks and reaches out into long-haul networks. The IP layer today, however, is not yet capable of providing QoS guarantees, such as guaranteed bandwidth and latencies. It also does not provide the same level of availability that SONET/SDH does with its 60 ms restoration times. For these reasons, it cannot yet support voice and private line traffic as well as the SONET/SDH network does. Such private line traffic still constitutes a significant portion of carrier revenues. Another reason is that in the core of the network it is more efficient to switch data in larger granularities than based on individual packets. There is not as much benefit due to statistical multiplexing, because the traffic is already highly aggregated when it reaches the core. As a result, the traffic tends to be more connection oriented, which doesn't match well with the connectionless datagram approach taken by IP networks. MPLS is being developed to address this issue and will likely be widely implemented 13.1 The Evolving Telecommunications Network 677 in core routers. For all these reasons, an IP over optical layer solution is employed today primarily by carriers to transport best-effort IP services and is not a universal solution. There is intense work under way to evolve the IP and optical layers to provide the same capabilities that SONET/SDH does. Adding protection functions in the optical layer and/or IP layer, as well as adding the QoS support in the IP layer, will enable the IP over optical layer architecture to migrate to a more universal solution. IP over WDM Variants We have talked about directly connecting IP routers to the optical layer, in the IP over WDM paradigm. In reality there are multiple ways this can be architected, as shown in Figure 13.7. The differences pertain primarily to the manner in which traffic passing through intermediate nodes is handled and the degree of agility provided in the optical layer. Before going into this in more detail, we look briefly at the capabilities of large IP routers and large optical crossconnects (OXCs). In general the trend to date has been that the total capacity that can be switched by a top-of-the-line router is much smaller than the total switching capacity of an OXC. Likewise, the OXC can be significantly denser (occupy a smaller footprint) than an equivalent router. Furthermore the cost per router port is usually much larger than the cost per equivalent OXC port. None of these are surprising, given the relative differences in functions and resulting complexity between a router and an OXC. The simplest architecture for IP over WDM, shown in Figure 13.7(a), is to connect the IP routers directly into optical line terminals (OLTs). Passthrough traffic at intermediate nodes is handled by the routers. This, however, has the highest cost for dealing with passthrough traffic, since expensive router ports need to be used to handle all this traffic. Also a large number of router ports will be needed in this approach, requiring significant floor space and associated power and cooling issues. Unfortunately, in some carriers, the router network and the transport (optical layer) network are designed and operated by different groups independently. This often leads to a situation not unlike what we see in Figure 13.7(a). The second approach, shown in Figure 13.7(b), is similar to the first, except that the passthrough traffic is handled by connecting patch cables between back-to-back WDM terminals within the optical layer. This approach is the lowest-cost option, as all passthrough traffic is handled without additional equipment or using up router ports. However, it is relatively inflexible in the sense that lightpaths cannot be con- figured dynamically in the network. Also it may be important to perform some demultiplexing and multiplexing of the lightpaths, that is, grooming, at intermediate nodes, if partial signals have to be dropped and added locally, for instance, an OC-12 signal from an OC-192 lightpath. 678 DEPLOYMENT CONSIDERATIONS Figure 13.7 Different architectures for realizing an IP over WDM network. (a) Passthrough traffic is handled by routers. (b) Passthrough traffic is patched through in the optical layer in a static fashion. (c) Passthrough traffic is handled by an optical crossconnect (OXC) providing dynamic reconfiguration and traffic grooming. The third approach, shown in Figure 13.7(c), uses OXCs to handle the passthrough traffic. In terms of cost, it lies between the two approaches discussed above, but provides the flexibility to set up lightpaths dynamically, as well as per- forms partial demultiplexing and multiplexing at intermediate nodes, if needed. As a result, this is the preferred IP over WDM architecture. The Evolving Network As of this writing, a variety of architectures have been implemented by carriers. One carrier has deployed an ATM over SONET/SDH over WDM network and transports lower-speed IP traffic over it. Higher-speed IP interfaces out of routers such as OC-48c interfaces are in most cases carried directly over the optical layer. Voice and private lines are carried over the SONET network. Another carrier that 13.1 The Evolving Telecommunications Network 679 provides just IP services has deployed the IP over WDM architecture. Yet other carriers have deployed an ATM network operating directly over the optical layer and are using it to deliver both virtual circuit and packet-oriented services. For these reasons, the network is migrating gradually to the architecture shown in Figure 13.8. The backbone is a mesh network made up of optical crossconnects, optical add/drop multiplexers (OADMs), and optical line terminals. The network supports a variety of traffic types, including SONET, ATM, and IR High-speed traffic streams are directly connected into the optical layer, whereas lower-speed streams may be multiplexed and brought into the network using one of the common service layers described above. Capacity is provisioned and allocated dynamically in the network by the OXCs and the OADMs. Bandwidth-efficient protection is offered as needed on a circuit-by-circuit basis. SONET/SDH will remain to support voice and private line traffic, as it is the best architecture for this purpose. In fact some of this multiplexing, particularly at the higher speeds, may be done by optical layer equipment, rather than separate SONET/SDH boxes. IP over the optical layer will become more ubiquitous as QoS guarantees are better implemented in the IP layer, MPLS matures to provide direct connections between routers, and protection functions are implemented well in the optical layer and/or the IP layer. At the edges of the network, access will be provided by a new-generation network element that combines lower-rate statistical and fixed SONET-like time division multiplexing over the optical layer. We call this element a multiservice platform (MSP). By combining time division and statistical multiplexing, an MSP has the potential to deliver a variety of circuit-switched and packet-switched services to the end users of the network. The idea is to use a single box in the access part of the network to deliver a variety of services to end users, without having to deploy multiple overlay networks to support each service type. As of this writing, a variety of MSPs are being developed by different equipment manufacturers, with a range of functionalities. At one end of the spectrum, an MSP is simply a SONET ADM, which provides data interfaces, such as Ethernet, in addition to supporting voice (DS0) and private lines (DS1/DS3, etc.). This box maps Ethernet signals into a SONET time slot and is purely a circuit-switched device, with no statistical multiplexing capabilities. Other MSPs are architected using a packet- or cell-switched internal core, which allows them to combine statistical multiplexing with time division multiplexing. These boxes perform statistical aggregation of the incoming data signals before mapping them into SONET time slots on their line sides. Finally there are MSPs that do not have any time division capabilities at all, carrying all incoming traffic over a packet-switched network such as IP or ATM. These rely on using QoS capabilities within the IP or ATM layers to provide circuit-switched-like services. . except that the passthrough traffic is handled by connecting patch cables between back-to-back WDM terminals within the optical layer. This approach is the lowest-cost option, as all passthrough. in data networks. For example, transporting a 100 Mb/s Ethernet signal across the coun- try requires leasing a 155 Mb/s OC-3 line. 9 Finally, today, carriers lack the management and signaling. providing quality-of-service (QoS) guarantees such as latency and bandwidth. For example, carriers can deliver guaranteed-bandwidth "virtual" DS 1 and DS3 services using their ATM networks