IPSec’s protocol objective is to provide
security services for IP packets such as encrypting sensitive data,
authentication, protection against replay and data confidentiality.
As outlined in our IPSec protocol article,
Encapsulating Security Payload (ESP) and Authentication Header (AH) are
the two IPSec security protocols used to provide these security
services. Analysing the ESP and AH protocols is out of this article’s
scope, however you can turn to our IPSec article where you’ll find an in-depth analysis and packet diagrams to help make the concept clear.
Understanding IPSec Modes –Tunnel Mode & Transport Mode
IPSec
can be configured to operate in two different modes, Tunnel and
Transport mode. Use of each mode depends on the requirements and
implementation of IPSec.
IPSec Tunnel Mode
IPSec tunnel mode is the default mode.
With tunnel mode, the entire original IP packet is protected by IPSec.
This means IPSec wraps the original packet, encrypts it, adds a new IP
header and sends it to the other side of the VPN tunnel (IPSec peer).
Tunnel
mode is most commonly used between gateways (Cisco routers or ASA
firewalls), or at an end-station to a gateway, the gateway acting as a
proxy for the hosts behind it.
Tunnel mode is used to encrypt traffic between secure IPSec Gateways,
for example two Cisco routers connected over the Internet via IPSec VPN.
Configuration and setup of this topology is extensively covered in our Site-to-Site IPSec VPN article. In this example, each router acts as an IPSec Gateway for their LAN, providing secure connectivity to the remote network:
Another example of tunnel mode is an IPSec tunnel between a Cisco VPN Client and an IPSec Gateway (e.g ASA5510 or PIX Firewall). The client connects to the IPSec Gateway. Traffic from the client is encrypted, encapsulated inside a new IP packet and sent to the other end. Once decrypted by the firewall appliance, the client’s original IP packet is sent to the local network.
In tunnel mode, an IPSec header (AH or ESP header)
is inserted between the IP header and the upper layer protocol. Between
AH and ESP, ESP is most commonly used in IPSec VPN Tunnel
configuration.
The packet diagram below illustrates IPSec Tunnel mode with ESP header:
ESP is identified in the New IP header with an IP protocol ID of 50.
The packet diagram below illustrates IPSec Tunnel mode with AH header:
The
AH can be applied alone or together with the ESP, when IPSec is in
tunnel mode. AH’s job is to protect the entire packet. The AH does not
protect all of the fields in the New IP Header because some change in
transit, and the sender cannot predict how they might change. The AH
protects everything that does not change in transit. AH is identified in
the New IP header with an IP protocol ID of 51.
IPSec Transport Mode
IPSec
Transport mode is used for end-to-end communications, for example, for
communication between a client and a server or between a workstation and
a gateway (if the gateway is being treated as a host). A good example
would be an encrypted Telnet or Remote Desktop session from a
workstation to a server.
Transport
mode provides the protection of our data, also known as IP Payload, and
consists of TCP/UDP header + Data, through an AH or ESP header. The
payload is encapsulated by the IPSec headers and trailers. The original
IP headers remain intact, except that the IP protocol field is changed
to ESP (50) or AH (51), and the original protocol value is saved in the
IPsec trailer to be restored when the packet is decrypted.
IPSec
transport mode is usually used when another tunneling protocol (like
GRE) is used to first encapsulate the IP data packet, then IPSec is used
to protect the GRE tunnel packets. IPSec protects the GRE tunnel
traffic in transport mode.
The packet diagram below illustrates IPSec Transport mode with ESP header:
Notice
that the original IP header at the front is the IP header from the
original IP packet. Placing the sender’s IP header at the front (with
minor changes to the protocol ID), proves that transport mode does not
provide protection or encryption to the original IP header and ESP is
identified in the New IP header with an IP protocol ID of 50.
The packet diagram below illustrates IPSec Transport mode with AH header:
The AH can be applied alone or together with the ESP when IPSec is in transport mode. AH’s job is to protect
the entire packet, however, IPSec in transport mode does not create a
new IP header in front of the packet but places a copy of the original
with some minor changes to the protocol ID therefore not providing
essential protection to the details contained in the IP header (Source
IP, destination IP etc). AH is identified in the New IP header with an IP protocol ID of 51.
In both ESP and AH cases with IPSec Transport mode, the IP header is exposed.
