The underlying protocol of how VPNs work

IPSec

• If the network layer is secure, the upper layers of transport and application don't need to be secure.
• Similar to encrypted telephone lines.
• ipv4 was designed without security in mind. ipv6 has necessary authentication and encryption.

The basic principle is fairly straightforward.

For any machine/router that's sending packets within the network, do nothing and run IP packets as they are.

For anything that's trying to send the packets outside, we turn the IP header into an IPSec header.

IPSec transport mode

• Both OSs should support IPSec in their network stack.
• Routing protection is not provided, since the original IP header is never encrypted.

Original IP hdr

ESP hdr

IP payload

ESP trailer

ESP Auth

The ESP header is the new addition.

IPSec tunnel mode

• This is used when the IPSec protocol is dealt with at the router level.
• There are gateways at both ends that allow you to transparently communicate without supporting IPSec yourself.
• Encrypts both the header and the payload.

New IP Header

ESP Header

Old IP Header

IP payload

ESP trailer

ESP Auth

ESP header is added on top of the old IP header, and a new IP header is added before the ESP header.

ESP header mechanism

ESP encrypts and optionally authenticates

ESP header is the same as the IP header, with the notable exception of its protocol number, which is 50 in this case. The payload itself might be enriched with optional padding (both for obfuscation and alignment), the followed by a pad length and the next-header field, which identifies the type of data contained in the enclosed protocol.

Note that the paddings, pad length, and the next-header field are after the payload. These are then followed by an integrity check.

IPSec security policy

• This is applied on each packet travelling from source to destination.
• Three policies, each holding up to its namesake:
  • bypass
  • secure
  • drop
• Stored in the association database and policy database

Security Association Database

A security association is a one-way logical connection between a sender and a receiver.

Uniquely identified by 3 params:

• Security Parameter Index
• Dest IP
• Protocol Identifier (in our case always ESP)

The database stores cryptographic parameters associated with each association.

The parameters are negotiated by using IKE.

Security Policy Database

This is effectively an IP based packet filter lookup table alike.

Maps a subset of IP traffic to an association for that traffic.

May create entries in SAD

Internet Key Exchange

• Four keys, two each for transmit and receive. Each of the keys is used for integrity and confidentiality.
• Can be manual or automated.
• IKE messages go via UDP over ports 500/4500
• Message format is IKE header followed by one or more IKE payloads (sort of like a linked list).
• Each payload has a critical bit, which means that when unable to understand, reject if 1, ignore otherwise.
• Uses a cookie scheme to prevent DDoS and drop superfluous connections if made. (Check for cookie values before starting computation)
• Authenticated DH exchange to prevent MITM
  • Digital signature
  • Public Key encryption
  • Symmetric key encryption

The actual exchange goes something like this:

• Init -- Responder: header + DH group offered algo + pubkey of DH + Nonce
• Resp -- Initiator: header + DH group offered algo + pubkey of DH + Nonce + Cert request
• Init -- Responder: header + MAC-ENC (ID + Cert + Cert Req + ID + Auth + DH chosen algo + traffic selector for both)
• Resp -- Initiator: header + MAC-ENC (ID + Cert + Auth + DH chosen algo + traffic selector (narrowed))

Other forms of exchanges using child nodes and notifs etc are also there.