376 Part IV: Application Layer Protocols and Services Figure 18-2 AH Transport mode For AH Transport mode, the ICV calculation is performed over the following: ■ All the fields in the IP header except those that are allowed to change in transit. These fields are the Type of Service (TOS), Flags, Fragment Offset, Time to Live (TTL), and Header Checksum, all of which are set to 0 for the ICV calculation. For source-routed IP traffic, the final destination IP address is predictable, and the appropriate fields within the Loose Source Route and Strict Source Route options are allowed to change. ■ All the fields in the AH (the Authentication Data field is set to 0). ■ The IP packet payload. For AH Transport mode, the AH protects the IP header, except the fields that are allowed to change, and the payload of the original IP datagram. The following is Frame 10 of Capture 18-01 in the \Captures folder on the companion CD-ROM, which shows an AH-protected Domain Name System (DNS) Name Query Request message, as displayed by Network Monitor 3.1: Frame: + Ethernet: Etype = Internet IP (IPv4) - Ipv4: Next Protocol = AH, Packet ID = 1807, Total IP Length = 86 + Versions: IPv4, Internet Protocol; Header Length = 20 + DifferentiatedServicesField: DSCP: 0, ECN: 0 TotalLength: 86 (0x56) Identification: 1807 (0x70F) + FragmentFlags: 0 (0x0) TimeToLive: 128 (0x80) NextProtocol: AH, 51(0x33) Checksum: 11405 (0x2C8D) SourceAddress: 131.107.0.2 DestinationAddress: 131.107.0.1 IP payload IP payload IP header IP header AH Chapter 18: Internet Protocol Security (IPsec) 377 - Ah: Next Protocol = UDP, SPI = 0x48B7D428, Seq = 0x1 NextHeader: UDP, 17(0x11) PayloadLength: 24 bytes Reserved: 0 (0x0) SecurityParametersIndex: 1220006952 (0x48B7D428) SequenceNumber: 1 (0x1) AuthenticationData: 12 UINT8(s) + Udp: SrcPort = 50286, DstPort = DNS(53), Length = 42 + Dns: QueryId = 0xDE8D, QUERY (Standard query), Query for test.contoso.com of type Host A ddr on class Internet AH Tunnel Mode Figure 18-3 shows AH Tunnel mode for an IP datagram. Figure 18-3 AH Tunnel mode In AH Tunnel mode, the entire original IP datagram is encapsulated with a new (outer) IP header and an AH. In the IP header, the Protocol field is set to 51 (0x33) to indicate that an AH is present. For Tunnel mode, the original IP header and payload are unmodified. The outer IP header is constructed from the configuration of the IPsec tunnel. The source IP address is the locally assigned IP address that is the best source to reach the tunnel destina- tion address. For AH Tunnel mode, the ICV calculation is performed over the following: ■ All the fields in the outer IP header except those that are allowed to change in transit (TOS, Flags, Fragment Offset, TTL, Header Checksum), all of which are set to 0 for the calculation ■ All the fields in the AH (the Authentication Data field is set to 0) ■ The original IP packet IP payload IP payload Authenticated AH IP header (new) IP header IP header 378 Part IV: Application Layer Protocols and Services For AH Tunnel mode, the AH protects the entire original IP packet (both the IP header and the payload) at the expense of an additional outer IP header that is not used for AH Transport mode. Encapsulating Security Payload (ESP) Encapsulating Security Payload (ESP) is a header and trailer combination defined in RFC 4303 that provides data origin authentication, data integrity, replay protection, and data con- fidentiality for the ESP-encapsulated portion of the packet. Figure 18-4 shows the structure of the ESP header and trailer and their location relative to the IP packet payload. Figure 18-4 The IPsec Encapsulating Security Payload header and trailer The ESP header consists of the following fields: ■ Security Parameters Index A 4-byte field that identifies, when used in combination with the Destination Address field in the IP header and transform (ESP), the specific SA for this datagram ■ Sequence Number A 4-byte field that is the same field as the Sequence Number field of the AH The ESP trailer consists of the following fields: ■ Padding A variable-length field (0-255 bytes) that is used to pad the encrypted payload to an appropriate length (depending on the encryption algorithm used), align the ESP portion of the packet along 4-byte boundaries, or deliberately obscure the encrypted payload’s length. ■ Padding Length A 1-byte field that specifies the number of bytes in the Padding field. Security Parameters Index Sequence Number Payload Padding Padding Length Next Header Authentication Data Chapter 18: Internet Protocol Security (IPsec) 379 ■ Next Header A 1-byte field used to identify the next header in the payload. This field uses the same values as the Protocol field in the IP header. ■ Authentication Data A variable-length field that contains the ICV calculation of the sender (the HMAC MD5 or HMAC SHA1 value). Because the use of a specific ICV algorithm is negotiated before data with an ESP header and trailer is sent, each peer knows the size of the Authentication Data portion of the ESP trailer and can determine the location of the end of the ESP-encapsulated payload. IPsec in Windows Server 2008 and Windows Vista can use the following encryption algorithms: ■ Advanced Encryption Standard (AES) with a 128-bit key size (AES-128) ■ AES with a 192-bit key size (AES-192) ■ AES with a 256-bit key size (AES-256) ■ Triple Data Encryption Standard (3DES) with three 56-bit keys ■ Data Encryption Standard (DES) with a 56-bit key (not recommended) The following is Frame 11 of Capture 18-02 in the \Captures folder on the companion CD-ROM, which shows an ESP-protected DNS Name Query Request message when using ESP and no encryption, as displayed by Network Monitor 3.1: Frame: + Ethernet: Etype = Internet IP (IPv4) - Ipv4: Next Protocol = ESP, Packet ID = 1542, Total IP Length = 88 + Versions: IPv4, Internet Protocol; Header Length = 20 + DifferentiatedServicesField: DSCP: 0, ECN: 0 TotalLength: 88 (0x58) Identification: 1542 (0x606) + FragmentFlags: 0 (0x0) TimeToLive: 128 (0x80) NextProtocol: ESP, 50(0x32) Checksum: 11669 (0x2D95) SourceAddress: 131.107.0.2 DestinationAddress: 131.107.0.1 - Esp: Next Protocol = UDP, SPI = 0x469021eb, Seq = 0x1 SecurityParameterIndex: 1183850987 (0x469021EB) SequenceNumber: 1 (0x1) - Trailer: PaddingData: Binary Large Object (2 Bytes) PaddingLength: 2 (0x2) NextProtocol: UDP, 17(0x11) AuthenticationData: Binary Large Object (12 Bytes) + Udp: SrcPort = 50202, DstPort = DNS(53), Length = 44 + Dns: QueryId = 0xF341, QUERY (Standard query), Query for test99.contoso.com of type Host A ddr on class Internet 380 Part IV: Application Layer Protocols and Services Note Network Monitor 3.1 displays the fields of the ESP trailer within the ESP header, rather than after the ESP payload. Network Monitor 3.1 cannot interpret the encrypted portions of an ESP-protected packet. ESP Transport Mode Figure 18-5 shows ESP Transport mode for an IP datagram. Figure 18-5 ESP Transport mode For ESP Transport mode, the ESP header is added to the IP datagram just after the IP header and the ESP trailer is added just after the payload. In the IP header, the Protocol field is set to 50 (0x32) to indicate that an ESP header is present. Routers forward this traffic as any other IP packet. Firewalls, on the other hand, might need to be configured to allow the forwarding of IP protocol 50 traffic. The payload is unmodified. Like AH, inserting an ESP header and trailer creates additional packet overhead, which lowers the effective MTU between the two endpoints. Performing the data encryption and calculating the ICV for the ESP trailer imposes additional processing overhead for each protected packet. Using network adapters that can perform cryptographic calculations in hardware, also known as offload adapters, can minimize this overhead. For ESP Transport mode, the following portions of the packet are encrypted: ■ The payload ■ The Padding, Padding Length, and Next Header fields of the ESP trailer Authenticated IP payload Encrypted IP payload IP header IP header ESP header ESP trailer Auth Data trailer Chapter 18: Internet Protocol Security (IPsec) 381 For encryption algorithms that use cipher block chaining (CBC), there is an unencrypted field between the ESP header and the payload. This field is the initialization vector (IV) for the CBC calculation performed at the receiver. This field cannot be encrypted because it is used to begin the decryption process. The inclusion of the IV as plaintext in the packet does not create a security problem. The IV does not provide additional cryptographic strength, only a way to ensure that the encryption of the same block with different IVs does not produce the same ciphertext. A malicious user might be able to view the IV, but without the encryption key, he or she cannot decrypt the ciphertext portion of the packet. To prevent a malicious user from modifying the IV and causing the receiver to produce garbled deciphered data, the IV is protected by the ICV. For ESP Transport mode, the ICV calculation is performed over the following: ■ All the fields in the ESP header ■ The payload (including the plaintext IV, if needed) ■ All the fields in the ESP trailer except the Authentication Data field For ESP Transport mode, the ESP trailer does not provide protection for the IP header and the Authentication Data field of the ESP trailer. To obtain protection for these elements, use both AH and ESP, as shown in Figure 18-6. Figure 18-6 Using both AH and ESP to protect an IP packet With AH and ESP, the ESP header and trailer wraps the payload, which then becomes the pay- load that is wrapped with an AH and the original IP header. Now the entire packet is protected (except the changeable fields in the IP header). ESP trailer Auth Data trailer IP header IP payload Authenticated with ESP Authenticated with AH Encrypted AH IP payload IP header ESP header 382 Part IV: Application Layer Protocols and Services The following is Frame 10 of Capture 18-03 in the \Captures folder on the companion CD-ROM, which shows an AH- and ESP-protected IP payload with ESP encryption, as displayed by Network Monitor 3.1: Frame: + Ethernet: Etype = Internet IP (IPv4) - Ipv4: Next Protocol = AH, Packet ID = 1555, Total IP Length = 120 + Versions: IPv4, Internet Protocol; Header Length = 20 + DifferentiatedServicesField: DSCP: 0, ECN: 0 TotalLength: 120 (0x78) Identification: 1555 (0x613) + FragmentFlags: 0 (0x0) TimeToLive: 128 (0x80) NextProtocol: AH, 51(0x33) Checksum: 11623 (0x2D67) SourceAddress: 131.107.0.2 DestinationAddress: 131.107.0.1 - Ah: Next Protocol = ESP, SPI = 0x43E235D7, Seq = 0x1 NextHeader: ESP, 50(0x32) PayloadLength: 24 bytes Reserved: 0 (0x0) SecurityParametersIndex: 1138898391 (0x43E235D7) SequenceNumber: 1 (0x1) AuthenticationData: 12 UINT8(s) - Esp: SPI = 0x1ef5e304, Seq = 0x1 SecurityParameterIndex: 519430916 (0x1EF5E304) SequenceNumber: 1 (0x1) EncryptedPayload: Binary Large Object (68 Bytes) ESP Tunnel Mode Figure 18-7 shows ESP Tunnel mode for an IP datagram. Figure 18-7 ESP Tunnel mode ESP header ESP trailer Auth Data trailer Authenticated Encrypted IP header (new) IP header IP payload IP payload IP header Chapter 18: Internet Protocol Security (IPsec) 383 In ESP Tunnel mode, the entire original IP datagram is encapsulated with a new (outer) IP header and an ESP header and trailer. In the outer IP header, the Protocol field is set to 50 (0x32) to indicate that an ESP header is present. For Tunnel mode, the original IP header and payload are unmodified. Like AH Tunnel mode, the outer IP header is constructed from the configuration of the IPsec tunnel. For ESP Tunnel mode, the following portions of the packet are encrypted: ■ The original IP datagram (IP header and payload) ■ The Padding, Padding Length, and Next Header fields of the ESP trailer For ESP Tunnel mode, the ICV calculation is performed over the following: ■ All the fields in the ESP header ■ The original IP datagram (IP header and payload), including the plaintext IV, if needed ■ All the fields in the ESP header except the Authentication Data field For ESP Tunnel mode, the ESP trailer provides protection for the original IP header and pay- load, but does not provide protection for the outer IP header and the Authentication Data field of the ESP trailer. IPsec and Security Associations A security association (SA) is the combination of security services, protection mechanisms, and cryptographic keys mutually agreed to by communicating peers. The SA contains the informa- tion needed to determine how the traffic is to be secured (the security services and protection mechanisms) and with which secret keys (cryptographic keys). There are two types of SAs that are created when IPsec peers communicate securely: the Internet Security Association and Key Management Protocol (ISAKMP) SA and the IPsec SA. ISAKMP SA The ISAKMP SA, also known as the main mode SA, is used to protect IPsec security negotia- tions. The ISAKMP SA is created by negotiating the ciphersuite used for protecting future ISAKMP traffic, exchanging key-generation material, and then identifying and authenticating each IPsec peer. When the ISAKMP SA is complete, all future SA negotiations for IPsec SAs are protected. This is an aspect of secure communications known as protected ciphersuite negotiation. Not only is the data protected, but the determination of the protection algorithms negotiated by the IPsec peers is also protected. To break IPsec protection, a malicious user must first determine the ciphersuite protecting the data, which represents another cryptographic barrier. For IPsec, the 384 Part IV: Application Layer Protocols and Services exceptions to complete protected ciphersuite negotiation are the negotiations of the cipher- suites of ISAKMP SAs, which begin as plaintext. IPsec SA The IPsec SA, also known as the quick mode SA, is used to protect data sent between the IPsec peers. The IPsec SA ciphersuite negotiation is protected by the ISAKMP SA. No information about the type of traffic or the protection mechanisms is sent as plaintext. For a pair of IPsec peers, there are always two IPsec SAs: one is negotiated for inbound traffic and one is for out- bound traffic. The inbound SA for one IPsec peer is the outbound SA for the other. Security Parameters Index For each IPsec session, IPsec peers must track the usage of three different SAs: the ISAKMP SA, the inbound IPsec SA, and the outbound IPsec SA. To identify a specific SA, a 32-bit pseudo- random number known as the Security Parameters Index (SPI) is used. The SPI is used for SA management at each IPsec peer and is a field in the IPsec headers protecting IPsec traffic and in the messages negotiating or managing SAs. The node that initiates an IPsec negotiation to perform IPsec protection is known as the initiator. The node that responds to a request to perform IPsec protection is known as the responder. The initiator chooses the ISAKMP SA SPI, and each IPsec peer chooses the IPsec SA SPI for its outbound traffic. Creating SAs An IPsec negotiation and determination of both ISAKMP and IPsec SAs occurs in two phases: the Main mode phase (also known as Phase I) and the Quick mode phase (also known as Phase II). Main Mode Main mode negotiation creates the ISAKMP SA. The initiator and responder exchange a series of ISAKMP messages to negotiate the ciphersuite for the ISAKMP SA (in plaintext), exchange key determination material (in plaintext), and identify and authenticate each other (in encrypted text). For more information about the details of Main mode negoti- ation, see the section “Main Mode Negotiation” later in this chapter. Quick Mode Quick mode negotiation creates the two IPsec SAs. The initiator and responder exchange a series of ISAKMP messages to negotiate the ciphersuite for both the inbound and outbound IPsec SAs. During Quick mode negotiation, keying material is refreshed or, if necessary, new keys are generated. For more information about the details of quick mode negotiation, see the section “Quick Mode Negotiation” later in this chapter. For IPsec for Windows Server 2008 and Windows Vista, a complete IPsec negotiation includ- ing both Main mode and Quick mode requires either 9 or 10 ISAKMP messages exchanged between IPsec peers, depending on security settings. Chapter 18: Internet Protocol Security (IPsec) 385 Internet Key Exchange The Internet Key Exchange (IKE) is a standard that defines a mechanism to establish SAs. IKE, described in RFC 2409, combines ISAKMP and the Oakley Key Determination Protocol. IPsec uses the ISAKMP protocol to negotiate SAs. ISAKMP includes facilities to identify and authenticate peers, manage SAs, and exchange key material. ISAKMP is a framework for nego- tiating secure communications independent of specific key exchange protocols, encryption and integrity algorithms, and authentication methods. To generate secret key material for secure communications, IKE uses the Oakley Key Determi- nation Protocol. Oakley is based on the Diffie-Hellman key exchange algorithm, which allows two peers to determine a secret key by exchanging unencrypted values over a public network. The mutually determined secret key becomes keying material from which secret keys for HMAC or encryption algorithms are derived. More Info The details of the Diffie-Hellman algorithm and the Oakley protocol are outside the scope of this book, but they are described in RFC 2412. ISAKMP Message Structure ISAKMP messages are sent as the payload of UDP messages using UDP port 500. Figure 18-8 shows the format of an ISAKMP message. Figure 18-8 An ISAKMP message The ISAKMP message consists of an ISAKMP header and one or more ISAKMP payloads. The ISAKMP payloads contain negotiation information and are encrypted for most ISAKMP mes- sages. The encryption protects the negotiation from being viewed by malicious users who are capturing ISAKMP traffic. The encrypted portions of ISAKMP messages cannot be viewed with Network Monitor. ISAKMP is defined in RFC 2408. ISAKMP Header The ISAKMP header is a standard header that is present for all ISAKMP messages and con- tains information about the message, including the type of packet. Figure 18-9 shows the for- mat of the ISAKMP header. IP datagram UDP message ISAKMP payloads UDP header ISAKMP header IP header [...]... AuthIP messages To obtain Network Monitor 3.1 components to parse AuthIP messages, see http://www .microsoft. com/about/legal/intellectualproperty /protocols/ mcpp.mspx AuthIP and IKE Coexistence Windows Server 2008 and Windows Vista support both IKE and AuthIP Windows XP and Windows Server 2003 support only IKE An initiator that supports both AuthIP and IKE must 402 Part IV: Application Layer Protocols and. .. Application Layer Protocols and Services 7 Peers 1 and 2 complete IKE Main mode and Quick mode negotiation 8 Subsequent segments sent over the TCP connection are protected with IPsec Windows Vista-based IPsec Peer in Request Mode and a Windows XP-based IPsec Peer in Require Mode In this example, a Windows Vista-based IPsec peer (Peer 1) is the initiator and the responder is running Windows XP (Peer... communication mode (in request mode, an IPsec peer requests IPsec protection but does not require it) ■ Windows Vista-based IPsec peer in request mode and a Windows XP-based IPsec peer in request mode ■ Windows Vista-based IPsec peer in request mode and a Windows XP-based IPsec peer in require communication mode (in require mode, an IPsec peer requires IPsec protection and silently discards unprotected packets)... the network or arrives after Message 2, IPsec peers running Windows Server 2008 or Windows Vista send Message 2 with a vendor ID payload that indicates support for AuthIP If an IPsec peer running Windows Server 2008 or Windows Vista receives Message 2 with the AuthIP-supported vendor ID payload, it waits for the initiating IPsec peer to retransmit Message 1 and then responds to Message 1 The initiator... negotiation is two IPsec SAs: one for inbound traffic and one for outbound traffic 400 Part IV: Application Layer Protocols and Services Quick mode negotiation for IPsec for Windows Server 2008 and Windows Vista consists of four ISAKMP messages The first Quick mode ISAKMP message, sent by the initiator, contains the following payloads: ■ SA The SA payload contains a list of proposals and encryption and hashing... Main mode negotiation 5 Peer 1 and Peer 2 complete IKE Main mode and Quick mode negotiation 6 Peer 1 retransmits the TCP- SYN segment (protected with IPsec) 7 Peer 2 sends the TCP- SYN-ACK segment (protected with IPsec) 8 Peer 1 sends the TCP- ACK segment (protected with IPsec) 9 Subsequent segments sent over the TCP connection are protected with IPsec IPsec NAT Traversal IPsec was designed to provide... AuthIP-based ISAKMP message, and then the IKE-based ISAKMP message as the first three messages of the message exchange Windows Vista-based IPsec Peer in Request Mode and a Windows XP-based IPsec Peer in Request Mode In this example, a Windows Vista-based IPsec peer (Peer 1) is the initiator and the responder is running Windows XP (Peer 2) Both peers are configured with request mode for both inbound and. .. allow IKE negotiation and ESP-encapsulated packets to work over NATs, IPsec for Windows Server 2008 and Windows Vista supports IPsec NAT traversal (NAT-T) as described in RFCs 394 7 and 394 8 NAT-T is especially useful when making L2TP/IPsec connections from a VPN client that is behind a NAT Chapter 18: Internet Protocol Security (IPsec) 405 NAT-T consists of the following elements used during Main mode:... NAT-T-capable IPsec peer and the presence of NATs between the peers 406 Part IV: Application Layer Protocols and Services ■ ISAKMP to complete Main mode and Quick mode negotiation despite the NAT’s presence ■ ESP-protected traffic to traverse a NAT Note Windows Server 2008 and Windows Vista do not support the UDP-EncapsulatedTunnel Encapsulation mode of NAT-T Summary IPsec is the standard method of providing... Reserved, and Payload Length fields) is the Vendor ID field, a variable-length field that contains the Vendor ID value IPsec for Windows Server 2008 and Windows Vista uses the following Vendor ID payloads to indicate that the IPsec peer: ■ Is running a Microsoft operating system and the version of that operating system ■ Supports Network Address Translator (NAT) Traversal capability based on RFC 394 7 ■ . chapter. For IPsec for Windows Server 2008 and Windows Vista, a complete IPsec negotiation includ- ing both Main mode and Quick mode requires either 9 or 10 ISAKMP messages exchanged between IPsec. is set to 0) ■ The original IP packet IP payload IP payload Authenticated AH IP header (new) IP header IP header 378 Part IV: Application Layer Protocols and Services For AH Tunnel mode,. Vendor ID value. IPsec for Windows Server 2008 and Windows Vista uses the following Vendor ID payloads to indicate that the IPsec peer: ■ Is running a Microsoft operating system and the version