VPN: IPsec – ESP and AH

IPsec provides:

1. confidentiality: encryption

2. integrity: by using hash

3. authentication: by using Internet key exchange (IKE).

4. antireplay protection: prevention of duplicated packets by comparing sequence number of the received packets with sliding window of the destination host.
any packets that has sequence number that comes before the sliding window is considered late or duplicated packet and will be dropped.

IPsec VS SSL:

1. Applications:
SSL: Web enabled application, file sharing and email.
IPsec: all ip based application (more flexibility comparing to SSL)

2. Encryption:
SSL: Key length between 40-bit and 128-bit
IPsec: between 56-bit and 256-bit (more secured than SSL as IPsec is able to provide the highest key length as compared with SSL, AES and RC6 both can provide of up to 256-bit key length.)

3. Authentication:
SSL: One way or two-way authentication.
IPsec: Two-way authentication using shared secret and digital certificates. (this provides a stronger authentication method used by SSL. Digital certificates are issued by Certificate Authority and could not be forged, and shared secret keys are calculated with an encryption algorithm such as DH which will be known only by the peers)

4. Ease of use
SSL: Easier than IPsec. Easier to deploy than IPsec.
IPsec: Can be quite difficult for non-technical users.

5. Overall security:
SSL: any device can connect.
IPsec: only specific device can be connected through the use of ACL.

If security is the primary concern use IPsec, if ease of deployment is the primary concern use SSL.

IPsec Framework:

1. IPsec protocol/negotiation protocol: Authentication Header (AH), Encapsulating Security Payload (ESP), ESP+AH

2. Encryption: DES, 3DES, AES

3. Integrity/authenticity: MD5, SHA-1

4. Protection(Diffie-Hellman): DH1, DH2, DH5, DH7

5. authentication: Pre-Shared Key (PSK), digital certificate.

Above 5 departments are scalable; that means if there is a new method of doing one of the 5 departments, the method will be implemented without changing the whole IPsec standard.

Negotiation protocol (authentic user check)
A. Authentication Header AH (IP protocol 15)
The message digest and the original data are transmitted to the destination host in clear text hence no confidentiality is provided.
AH supports HMAC-MD5 and HMAC-SHA1.

AH using HMAC in action:
1. The original payload (data) and the ip header are hashed together with a secret key that’s shared between the source and destination host

2. the message digest builds a AH header and appended to the original packet.

3. the new packet is transmitted to the peer router.

4. peer router will hash the ip header and the original payload, strips out the AH from the payload and compare its digest with the digest in AH. AH digest and peer router’s digest must match exactly, if not match the packet will be dropped.

NAT is incompatible with AH, because the ip header will be modified.

AH is used when confidentiality is not a concern, AH provides Integrity, authentication and antireplay protection.

Sidetrack:
Hashed Message Authentication Codes (HMAC) must provide a secret key as an input to the hash function.
original payload + secret key = HMAC value.
The secret key is a pre-shared key of the peers. The destination peer received the HMAC payload will hash the payload with its share secret key, the answer must match with the source HMAC digest.
There are two types of HMAC: HMAC-MD5 and HMAC-SHA-1
HMAC MD5 has a fixed length of 128-bit digest
HMAC SHA-1 has a fixed length of 160-bit digest

B. Encapsulating Security Payload (ESP)
To address the confidentiality issue with AH; Encapsulating Security Payload (ESP) is created.

ESP supports several symmetric encryption: DES, 3DES, AES. The confidentiality is achieved through encryption of the payload (ip datagram). After encryption ESP trailer and header are added to the encrypted payload.

The encrypted payload will be hashed using HMAC-MD5 or HMAC-SHA-1. The encrypted payload (ip datagram), ESP trailer and header are included into the hash function.

Optionally antireplay protection can be enforced by requiring the receiving host to turn on the replay bit in the ESP header to indicate that the packet has been seen.

A new ip header will be prepended over the encrypted ip datagram (ESP header onwards until ESP trailer all are encrypted) to be routed over the internet, this is actually known as the tunnel mode, whereby the original ip header is encrypted and only the new ip header is in clear text.

ESP and AH both can use Transport mode or Tunnel mode.

Transport mode:
Transport layer 4 and above will be encrypted, the original ip header will be in clear text. This mode works well with Generic Routing Encapsulation (GRE). GRE is a cisco proprietary tunneling protocol which will hide its original IP header and adds a new IP header.
This mode is also suitable for LAN.

Tunnel mode:
Layer 3 and above will be encrypted. The original IP address, port numbers including the data will be encrypted. A new IP header which is in clear text will be prepended for routing. This is suitable when your data is transported through the internet.