I could see that, an explanation to IPSEC is not explained well in all the articles I’ve read. I’ll try to explain IPSEC as simple as possible.
IPSEC is having two Phases. First is IKE Phase 1.
IKE Phase 1:
The two end points need to Authenticate first. Authentication can be performed using 3 methods, what you have, what you are and what you know. In most IPSEC deployments, we use a Preshared key for Authentication.
Next, this key is exchanged securely using a wonderful technique called Diffie-Hellman algorithm. This algo minimises MITM by salting the Preshared key with a secret key and then exchanging the salted Preshared key and then adding the secret key again at the end points to get the Common Key to be used. I know, a picture is more explanatory than a thousand words, Here’s the technique for you below. There are various DH groups used depending upon the length of the key used (DH 1, 2, 5 etc).
ISAKMP Security Association will have 1 bidirectional SAs that creates policy to manage Hash, Encryption algo etc to be used. It uses the UDP 500 port.
IKE Phase 2:
The second SA created is called IPSEC SA, which have 2 unidirectional SA’s, which creates transform sets for the actual transfer of data. IPSEC SA happens in the IP layer, and no port is assigned to it, instead it have Security Parameter Index (SPI) which is recognised by some Cisco Firewalls. Otherwise, it can be wrapped in UDP 4500 too.
IPSec uses two distinct protocols, Authentication Header (AH) and Encapsulating Security Payload (ESP), which are defined by the IETF.
The AH protocol provides a mechanism for authentication only. AH provides data integrity, data origin authentication, and an optional replay protection service.
The ESP protocol provides data confidentiality (encryption) and authentication (data integrity, data origin authentication, and replay protection). ESP can be used with confidentiality only, authentication only, or both confidentiality and authentication. When ESP provides authentication functions, it uses the same algorithms as AH, but the coverage is different. AH-style authentication authenticates the entire IP packet, including the outer IP header, while the ESP authentication mechanism authenticates only the IP datagram portion of the IP packet.
In the process of applying either AH or ESP to an IP packet, the original IP packet is modified. Outbound packets are rebuilt with additional IPSec headers in a process known as encapsulation, while inbound packets are stripped of their IPSec headers in a process known as decapsulation. Before leaving a host, outbound packets are encapsulated using a cryptographic key that is known to both communicating hosts. Inbound packets are decapsulated on the receiving side using the same cryptographic key, thereby recovering the original datagram. If encryption is used, any packet that is intercepted on the IP network is unreadable to anyone without the encryption key. Any modifications to the IP packet while in transit are detected by authentication processing at the receiving host and discarded.
Transport mode and tunnel mode
The manner in which the original IP packet is modified depends on the encapsulation mode used. There are two encapsulation modes used by AH and ESP, transport and tunnel.
Transport mode encapsulation retains the original IP header. Therefore, when transport mode is used, the IP header reflects the original source and destination of the packet. Transport is most often used in a host-to-host scenario, where the data endpoints and the security endpoints are the same. A transport mode encapsulated datagram is routed, or transported, in the same manner as the original packet.
IPv4 packet encapsulated using AH in transport mode
IPv4 packet encapsulated using ESP in transport mode
Tunnel mode encapsulation builds a new IP header containing the source and destination address of the security endpoints. When tunnel mode is used, the outer IP header reflects the source and destination of the security endpoints, which might or might not be the same as the original source and destination IP address of the data connection. The choice of transport or tunnel mode depends on the structure of the network and relies heavily on logical connections between the endpoints. Tunnel mode is required if one of the IKE peers is a security gateway that is applying IPSec on behalf of another host or hosts. A datagram that is encapsulated in tunnel mode is routed, or tunneled, through the security gateways, with the possibility that the secure IPSec packet will not flow through the same network path as the original datagram.
IPv4 packet encapsulated using AH in tunnel mode
IPv4 packet encapsulated using ESP in tunnel mode
In Conclusion, A new IP header is added in the Tunnel Mode. This inturn hides the Private IPs and shows only Public IPs on either end of the channel. Where as, in Transport mode, it encrypts only the Data not the IP header and its mainly used in Host-Host communication.