7 Measurements NOIAYTON (Know thyself; Inscription on the fa¸cade of Apollo’s temple in Delphi) The traffic engineering process for IP networks includes obtaining feedback from the network as the basis of assessing the need to modify transport network parameters and to optimize its behaviour. Obtaining information in precise and meaningful form is imperative for being able to make the right adjustments to improve the network performance. This chapter deals with the means of monitoring service quality, the analysis methods applied to measurement data, and examples of network performance optimization based on processed measurement information. A formulation of the goals and methods of traffic engineering measurements in IP networks has been given in a recent Inter- net draft [LCT + 02]. This draft will be used as an introduction to the topic in this chapter, quoting other sources and adding issues specific to multi-service networks as necessary. In the draft, ISPs are singled out as one likely user for the methods presented. The scope of the conceptual framework development therein is intra-domain operations, but the definitions are intended to be transferable also across operator domain boundaries. As such, Implementing Service Quality in IP Networks Vilho R ¨ ais ¨ anen  2003 John Wiley & Sons, Ltd ISBN: 0-470-84793-X 212 MEASUREMENTS they should be applicable to different per-domain technologies as well. The need for consistency, precision and effectiveness of traffic engineering methods is cited as the reason for applying an over- arching framework for traffic engineering. The ultimate goal of measurements is to serve the needs of the traffic engineering pro- cess, including forecasting, planning, dimensioning, control, and performance monitoring. The major tasks of traffic engineering measurements are defined in [LCT + 02] as • traffic characterization; • network monitoring; • traffic control. These will be discussed in more detail later in this chapter. Let us note in passing again that in multi-service networks, the char- acterization, monitoring, and traffic control tasks may need to be applied to multiple traffic aggregates on individual links. Three approximate timescales pertaining to the use of data ob- tained with measurements are identified in [LCT + 02], namely: • Months or longer. This timescale relates to network planning and upgrading. Forecast traffic volumes per service class are impor- tant here. • Hours to days. Capacity management is the primary use of mea- surement data at these timescales. Measurement data could be used to control default routing of traffic aggregates and resource allocation in network nodes. • Minutes or less. This timescale belongs to real-time control of the network. In [LCT + 02], the example of temporary rerouting of traffic aggregates to circumvent congestion is cited. As we shall see later, measurements dealing with different timescales have different requirements and analyses associated with them. Some general requirements for measurements are: • Accuracy in capturing important phenomena at different timescales. The shortest timescale of relevance to network performance on the timescale of interest must be known and the measurement methods should be chosen accordingly. In a multi- service network, the required accuracy may be different for 7.1 TRAFFIC CHARACTERIZATION 213 different traffic aggregates. For example, millisecond-accuracy measurements for delay may be desirable for VoIP, but not necessary for best-effort traffic. • Network performance should not be degraded by measurements.This applies both to the elements being measured as well as the effect of measurements and transferring of measurement data to the normal operation of a production network. Interestingly, this is an issue not only for active measurements but passive ones as well. • The amount of data generated should be moderate. Regarding the last two points, these issues are typically more challenging in a multi-service network than in a best-effort network, since there may be multiple quality support aggregates involved for each link. Subsequently, both performing the actual measurements and transmitting measurement data in such a way as not to interfere with the normal network operation need to be carefully planned. Next, we shall take a look at the three tasks of traffic engineering measurements as defined by our IETF framework [LCT + 02], and then will discuss the definition of measured characteristics, sources of information, measurement methods, and the required measurement infrastructure in general. The present chapter will be concluded with case studies. 7.1 TRAFFIC CHARACTERIZATION Traffic characterization is the first task of traffic engineering mea- surementsasdefinedby[LCT + 02], having the following goals: • Identifying variations in traffic patterns using statistical analysis, including development of traffic profiles to capture daily, weekly, or seasonal variations. • Determining traffic distributions in the network on the basis of flows, interfaces, links, nodes, node-pairs, paths, or destinations. • Estimation of the traffic load according to service classes in different routers and the network. • Observing trends for traffic growth and forecasting of traffic demands. 214 MEASUREMENTS The determination of traffic distributions in the network is partly related to the estimation of traffic matrix discussed in Chapter 4. In other words, direct measurement can be used to obtain the topological distribution of traffic in the network. However, per- link volumes need to be linked to traffic aggregates entering and exiting the network domain in order to influence the distribution with routing control. For a best-effort IP network domain, traffic pattern variations may relate to changes in the composition of protocol types in the totality of traffic, as well as information about traffic volumes in topological context. In multi-service networks, such information should be available per service quality support aggregate. In a multi-service network with service mapping onto service quality support aggregates on the network edge, traffic engineering benefits from the ability to compare characterizations of both incoming service types and service quality support aggregates, side by side. Such a comparison makes it possible to effectively evaluate the suitability of both the service/aggregate mapping at the network edge, and the service quality support for aggregates in the network core (see Figure 7.1). Depending on the measurement methods, discussed in more detail below, modelling of data may be needed to interpret the results in a context relevant for traffic engineering. For modelling, the alternatives are fitting of measurement results into an existing model and providing generic modelling for measurement results without reference to the use context. In certain situations modelling has a risk associated with it, since it makes assumptions about what the results should look like. Thus, when modelling is used, consistency checks should be constructed to check that the situation in the measured network ER Service distribution Traffic load for service support aggregates Figure 7.1 Modelling both incoming services at the network edge and loads of service quality support bearers in the network core 7.1 TRAFFIC CHARACTERIZATION 215 is consistent with the assumptions made in the model. Another useful technique is computation of the same characteristics using multiple different methods. According to [LCT + 02], Internet traffic is bandwidth-limited but non-stationary; traffic can be heavy-tailed and have strong correlations at short timescales. This is often the case in best- effort Internet with no per-flow policing. The suggestion of the Internet draft is to use decomposition of measurement results into stationary and trend parts. Obviously the stationary part also needs to account for diurnal variations in traffic intensity. Regarding the scope of this book, this breakdown should be possible per service quality support aggregate. A more fine- grained temporal classification of the problem area of traffic decomposition could be as follows: • trend prediction (days or longer periods of time). This timescale is relevant for capacity management in traffic engineering. • busy-hour traffic characterization; • statistics and correlation at the timescale of seconds. The reported measurement data should be associated with information on the scale of applicability, partly stemming from the details of the actual measurement. This requirement is valid irrespective of the possible use of modelling as a part of the measurement. An example used in the cited draft, modelling of flow burstiness can be performed by fitting measurement results into a token bucket model as the probability that a flow can be accommodated by the token bucket, as estimated from a measurement. The result, in general, can be dependent on the length of the measurement period. Thus, a full description of measurements in this case is as follows: • token bucket rate; • token bucket depth; • probability; • timescale of applicability. The timescale is of importance especially when dealing with self- similar traffic patterns, where the average magnitude of variations ranges with the length of the observation period. 216 MEASUREMENTS 7.2 NETWORK MONITORING The goals of network monitoring, the second traffic engineering measurement task, are: • Determining the operational state of the network, including fault detection. • Monitoring the continuity and quality of network services, to ensure that service quality objectives are met for traffic aggregates. Further, goals include verification of the performance of delivered services. • Evaluating the effectiveness of traffic engineering policies, or triggering certain policy-based actions (such as alarm generation, or path pre-emption) upon threshold crossing; this may make use of past performance data, in addition to latest measurement results. • Verifying peering agreements (SLAs) between service providers by monitoring/measuring the traffic flows over interconnecting links at border routers. This includes the estimation of inter- and intra-network traffic, as well as originating, terminating, and transit traffic that are being exchanged between peers. Performance monitoring, meaning continuous monitoring of the performance of network elements to ensure that they are performing correctly, is a central part of performance management. Possible reasons for suboptimal service performance include route flapping, congestion, hardware or software failures, and element misconfiguration. In a multi-service network domain, the above goals need to be implemented for the different service quality support aggregates. This sets higher requirements for the measurement infrastructure. For example, one needs to be able to detect the malfunctioning at the granularity of individual service quality support classes. Similarly, the triggering of traffic engineering actions needs to work reliably in the multi-service case. Obtaining data for various subtasks of network monitoring is a trade-off between the number of different measurement types and accuracy for individual monitoring purposes. It is clearly desirable to derive multiple levels of performance characteristics from a single set of measurement data, if the methods for performing this are known and sufficient processing power for this is available. Alas, both conditions are not always found. Especially for new 7.2 NETWORK MONITORING 217 types of services, the analysis methods for obtaining service quality description may not yet be available. If services need to be created quickly for commercial reasons, the very act of configuring measurements using low-level data sources may be too time- consuming and it may be easier to perform using separate service- level measurements. Later, when the behaviour of the network is understood better and on more quantitative level, the service- specific performance data can possibly be extracted from a smaller number of measurements. The scheduling of measurements is part of the network monitoring task. One important concept here is the temporal distance between successive measurements, known as the read- out period. The length of the read-out period should be sufficiently short to prevent momentary variations from being averaged out, but long enough to avoid disturbing the router. In a multi- service network, the measurement scheduling periods need to take into account different service quality support types. For measurements directly addressing the service level performance of a particular service support aggregate, the actual lengths of read- out periods may be different from each other. For measurements, the results of which are used for assessing the performance of multiple types of services, on the other hand, the length of the read-out period may be determined by the strictest reporting requirement. An important part of network monitoring is presenting the measurement data within the right context. Taking the load as an example, in a simple routed IP network, load levels might be collected per link, out of which a meaningful traffic matrix needs to be constructed. If MPLS is in use, however, also load levels per LSP are of interest. If a single LSP is shared by multiple traffic aggregates, load levels per DSCP/LSP combination would be useful information for traffic engineering purposes. A router can belong to multiple such contexts. As an example, an MPLS- capable DiffServ router could perform monitoring both per LSP, as well as per network interface. 7.2.1 Troubleshooting measurements for services Troubleshooting-type measurements for service quality can be used as necessary to complement data from normal “traffic 218 MEASUREMENTS engineering” type measurements. Troubleshooting measurements can be based on testing the actual service performance either using realistic protocols is an emulation of application stream, or otherwise measuring the quality of a traffic aggregate used as a bearer for applications. Application emulation – type troubleshooting measurements can be performed end-to-end, per- domain, or narrowed down on smaller areas to identify potential problem locations. For non-end-to-end measurements, one needs to be careful in interpreting the measurement results in terms of impact on end-to-end performance, as has been discussed in Chapter 5. Making an ICMP PING test for a route or for a LSP is an example of a simple test of the ability of an aggregate to transfer any data. Such tests could be carried out periodically to verify the opera- tional status of routes or LSPs. An example of an application emulation troubleshooting mea- surement is transmission of emulated VoIP stream between two laptops, and measuring delay time series and packet loss charac- teristics from the stream. The emulation produces correct stream metrics (packet sizes and transmission intervals can be chosen to correspond to a realistic codec) and uses the same four low- est OSI layers for the measurement stream as a VoIP applica- tion (UDP/IP/LL/PHY). As with active measurements in general, ICMP PING and traceroute may not yield correct delays due to possibly different PDU treatment in routers for ICMP and L3+ traffic. Keeping in mind the related discussion in previous chap- ters, all measurements also take into account the possible effects of measurement endpoints. An example of an end-to-end test is verifying the signalling per- formance of a SIP proxy by using a SIP client, which is able to time the individual signalling events. In this case, provided that the endpoints (client and SIP proxy) are operating correctly, such a test shows an example of the effect of network performance to end-to- end application performance. Similarly, assessing end-to-end VoIP performance could be based on transmission of reference samples over the network using actual VoIP terminal applications having hooks to provide application-level performance characteristics. A possible set-up and its relation to traffic engineering mea- surements in a domain are illustrated in Figure 7.2. The service flow emulation measurements can be made on the same route as the traffic engineering measurements, but may be expressly 7.3 TRAFFIC CONTROL 219 IP domain ER ER Traffic engineering measurements IP domain ER ER Testing of aggregate performance SIP proxy Terminal Testing of signalling performance Terminal Testing of voice quality Figure 7.2 Traffic engineering measurements (left) and different types of troubleshooting-type measurements (right) designed to test the performance relevant to the tested applica- tion. An example of this is performing active measurements with 20 ms inter-packet separation to probe timescales of performance, which are relevant for the AMR codec. 7.3 TRAFFIC CONTROL The third task of traffic engineering measurements, traffic control, is related to interfacing of the control part of traffic engineering to measurements. According to [LCT + 02], some of the relevant subtasks are: • Adaptively optimizing network performance in response to net- work events. Rerouting around points of congestion or failure in the network is given as example of this. • Providing a feedback mechanism in the reverse flow messaging of RSVP-TE or CR-LDP signalling in MPLS to report on actual topology state information such as link bandwidth availability. • Support of measurement-based admission control, i.e., by pre- dicting the future demands of the aggregate of existing flows so that admission decisions can be made on new flows. The examples listed above relate to routing control and admission control means of service quality control. More generally, the examples are partly overlapping with activities in the traffic engineering process, which take measurements as input. Routing control above is an example of this. Another example could be us- ing measurements for triggering configuration optimization and 220 MEASUREMENTS reconfiguration of DiffServ parameters in the network elements using policy management. Other aspects of traffic control, as exemplified by the list above, fall within shorter timescales than what is relevant for the entire measure-model-reconfigure loop. Direct measurement-based feedback to admission control is an example of such shorter timescale processes. Further examples of traffic control include: • Triggering of reconfiguration of policing at the network edge. • Adjustments of link costs in link-state routing protocols based on measurements. This requires a management interface to link cost computation in routers. The first of these two examples is more clearly within the appli- cation of the traffic engineering loop, requiring configuration of multiple network elements. The reaction to a certain set of mea- surement results could still be preconfigured, avoiding the need to perform a full modelling step. Some aspects of dynamic traf- fic control falling within the concept of bandwidth brokers are discussed in the following chapter. It should be noted that in traffic control, modelling of system level behaviour is needed. The means for this have been discussed in Chapter 5. Modelling of system behaviour should be contrasted with modelling of data in the traffic characterization task, the latter providing input for the former. 7.4 DEFINITION OF MEASURED CHARACTERISTICS The measured characteristics, in all, span multiple scopes. A general goal of Service Level Agreements, the characteristics should be to use system-level characteristics that are independent of protocols and platforms, and be uniform across operator boundaries [LCT + 02]. This goal is rather similar to the Per- Domain Behaviours [RFC3086] mentioned in the last chapter. More fine-grained characteristics are needed for detailed performance information. The classification used in this chapter is as follows: characteris- tics can be measured at service level, packet or traffic aggregate level, or with network element granularity. Examples of charac- teristics at each of these levels are given below. [...]... shall be set in accordance to the measured aggregate Instantaneous packet delay variation (IPDV) is based on the definitions of the IPPM working group, and on the ITU-T specification [I380] Measuring IPDV does not require synchronized clocks It is specified that packets should not be fragmented The time-out for IPDV is specified to be two minutes, as for delay and loss measurements QBone domain 2 M M QBone... corrupted packets For passive monitoring, also the time the packet was received • Flow Source and destination addresses, port numbers, protocols, time of start and end of a flow, packet count, IP version (IPv4/IPv6) When the application flows are encrypted, obtaining of some of the information may be limited • Traffic aggregate Example characteristics are per-aggregate buffer occupancy levels, offered... be monitored can relate to packet statistics, traffic aggregates, flows, links, and service events The four first items listed are IP or link level characteristics, whereas the last one is service-level characteristic A subset of IP- level characteristics can be available in any IP device such as a router, depending on the set of counters available The palette of available characteristics (and counters)... Type Example Notes Service quality support aggregate performance Poisson-distributed transmission of packets Defined by IPPM working group Periodic streams Defined by IPPM working group ICMP ping Results may differ from application performance Service emulation HTTP query Emulation of VoIP media stream 230 7.7 MEASUREMENTS TRAFFIC ENGINEERING MEASUREMENT INFRASTRUCTURE Next, let us take a look at the infrastructure... SIP proxy for information about call statistics When the SIP proxy is outside of the network operator’s domain, delivery of such information could be part of the Service Level Agreement Service response time can also be estimated with application-level measurement, in which case modelling may be necessary • Packet/aggregate level: – Bandwidth availability Can be used for traffic control using multiple... traceroutes sum of BE bps None 5 minutes raw 1 minute EF load measurements (no timestamps needed) 5 minutes 5 minutes milliseconds of IPDV at the 99.5th percentile 5 minutes sum of EF bps 5 minutes milliseconds of IPDV at the 90th percentile 5 minutes 5 minutes milliseconds of IPDV at the 50th percentile Summary interval 5 minutes Summary function Image report (*.Gif)/page reports (*.Html) Summary interval... cases, the identity (IP address), location (in a connectivity hierarchy), and capabilities of the measuring entity need to be known by the measurement control system The location depends on which layer the measurement is performed: if the measured characteristics related to link layer properties, it is possible that information about link layer connectivity is needed in addition to IP level connectivity... estimators 7.5 SOURCES OF MEASUREMENT DATA Physically, measurement data can be obtained from network elements and measurement probes Network elements, in turn, can be IP or link layer devices such as routers, or service elements such as HTTP servers or SIP proxies For a transport network operator, direct access to all service elements is not always possible, but service usage information could be included into... per second BeLoad BE Load unsigned integer pair bits per second, packets per second EfTrace EF Traceroute string of dot-notated, comma-delimited IP addresses (e.g “a1.b1.c1.d1, a2.b2.c2.d2, ”) NA BeTrace BE Traceroute string of dot-notated, comma-delimited IP addresses (e.g “a1.b1.c1.d1, a2.b2.c2.d2, ”) NA LinkBW Link Bandwidth unsigned integer ordered pair bits per second, timestamp EfCom EF Commitment... measurement through the MPLS traffic engineering MIBs may be used (4) Active measurements based on IPPM metrics are currently in use for node-pairs; they may be also applied to paths, but without means of controlling the routing no one-toone correspondence necessarily exists (5) Besides active measurements based on IPPM methodologies, path loss may possibly be inferred from the difference between ingress and . of a SIP proxy by using a SIP client, which is able to time the individual signalling events. In this case, provided that the endpoints (client and SIP proxy). expressly 7.3 TRAFFIC CONTROL 219 IP domain ER ER Traffic engineering measurements IP domain ER ER Testing of aggregate performance SIP proxy Terminal Testing