| ]

This chapter will focus on how to troubleshoot IPsec sessions on Cisco PIX and ASA security appliances. The layout of this chapter is similar to that found in Chapter 19, “Troubleshooting Router Connections.” I’ve broken the chapter into two areas on troubleshooting: ISAKMP/IKE Phase 1 and ISAKMP/IKE Phase 2 issues. With these two areas, I’ll show you how ISAKMP/IKE Phase 1 and 2 connections are built, and what to look for when there is a problem with either of these phases.

This chapter by no means covers all possible problems you’ll experience with IPsec sessions on Cisco security appliances. However, I hope to provide you with the basic background knowledge so that troubleshooting IPsec sessions on the appliances is a simpler process.

ISAKMP/IKE Phase 1 Connections

In the first part of this chapter I’ll focus on troubleshooting ISAKMP/IKE Phase 1 connections. If you recall from Chapter 3, “IPsec,” the management connection built during Phase 1 is used to pass IPsec management traffic; no user data traverses this connection. This connection is important, however, because it is used to build the two data connections for Phase 2. I’ve broken this part of the chapter into three areas:

  • An overview of the ISAKMP/IKE Phase 1 troubleshooting commands

  • Examining your management connections

  • Examining the building of L2L and remote access management connections

  • Troubleshooting Easy VPN connections on a Remote

Overview of the Phase 1 Commands

You can use several commands to troubleshoot ISAKMP/IKE Phase 1 connections on the security appliances, including the following:

  • show isakmp sa [detail] Displays the status of any management connections.

  • show [crypto] isakmp stats Displays the statistics of the management connections (FOS 7.0 only).

  • show [crypto] isakmp ipsec-over-tcp stats Displays the statistics of any IPsec over TCP connections the management connection is managing (FOS 7.0 only).

  • debug crypto isakmp Displays the steps taken to build a management connection and data connections via the management connection.

  • debug crypto vpnclient Displays the interaction between the appliance, acting as an Easy VPN Remote, and the Easy VPN Server (FOS 6.3 only).

  • debug crypto ca [messages | transactions] Displays the interaction between the appliance and CA for certificate enrollment and authentication functions; the optional parameters are new in FOS 7.0. The 7.0 version of this command produces similar output compared to the debug crypto pki command discussed in Chapter 19; therefore, I won’t cover it in this chapter.

  • debug crypto engine Displays events related to the encryption/decryption problems on the appliance.

  • clear [crypto] isakmp sa [SA_ID_#]— Deletes all the management SAs or a specific management connection by specifying the SA ID number.

As you can see from the above list, not all commands are supported in all FOS versions. The following sections will discuss some of the more important commands, related to troubleshooting connectivity processes, in more depth.


Note

Before FOS 7.0, I found the output of debug commands less administrator-friendly than the debug output from IOS routers. In FOS 6.3 and earlier, I tended to try to troubleshoot IPsec problems from the remote peer and would look at the PIX’s debug output only when I was still having problems trying to pinpoint the problem. However, Cisco has rectified most of my concerns in regard to this in FOS 7.0. In FOS 7.0, the debug output is much more similar to the debug output of IOS-based routers.

The show isakmp sa Command

Example 23-1 illustrates the use of the show isakmp sa command with an appliance running FOS 6.3. The output of this command is very similar to the show crypto isakmp sa command in Chapter 16, “Router ISAKMP/IKE Phase 1 Connectivity.” Table 16-1 in that chapter explains the states. If you recall, QM_IDLE indicates the successful setup of the connection to the associated peer. If you’re seeing MM_NO_STATE or AG_NO_STATE, this indicates that there is a problem with the initial setup of the connection. The two most common problems that might cause this are:

  • You forgot to activate the crypto map or profile on the remote peer router’s interface.

  • There is no matching ISAKMP/IKE Phase 1 policy on the remote peer.

If you see a state of MM_KEY_EXCH or AG_INIT_EXCH, then probably the culprit is failed device authentication. For pre-shared keys, be sure you’ve configured the keys correctly. For certificates, verify that they haven’t expired, that the date and time are correct on the peers, and that they haven’t been revoked.


Tip

You can use the debug [crypto] isakmp sa command for more detailed troubleshooting based on the output of the show crypto isakmp sa command.

Example 23-1: The show crypto isakmp sa Command in 6.3
Image from book
pix63(config)# show isakmp sa
Total : 1
Embryonic : 0
dst src state pending created
192.1.1.101 192.1.1.40 QM_IDLE 0 0

Image from book

In FOS 7.0, the output of the command is different, as shown in Example 23-2. Instead of seeing “QM_IDLE” when the management connection has completed, you’ll see either “MM_Active” or “AG_Active,” depending on whether main mode or aggressive mode was used to build the management connection, respectively.

Example 23-2: The show crypto isakmp sa Command in 7.0
Image from book
pix70(config-general)# show isakmp sa
Active SA: 1
Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1 IKE Peer: 192.1.1.40
Type : L2L Role : responder
Rekey : no State : MM_ACTIVE

Image from book

The debug crypto isakmp Command

In most instances, you’ll use the debug crypto isakmp command to assist in detailed troubleshooting of building ISAKMP/IKE Phase 1 management connections as well as Phase 2 data connections the management connection builds. Deciphering the output of this command is not that simple. The following two sections will take a look at a few examples of L2L and remote access sessions.


Note

Because the output of the debug commands in FOS 6.3 and earlier is somewhat similar to that of Cisco routers, the following sections will focus on the use of the commands in FOS 7.0.

L2L Sessions

To understand how an L2L session is successfully set up, view the output from the debug crypto isakmp command in Example 23-3. In this example, the output is from a simple L2L configuration where the appliance is accepting a session setup request from a remote L2L peer. I’ve added steps to the right of some of the output, which are explained below the example.

Example 23-3: Successful Building of the Management Connection in FOS 7.0
Image from book
[IKEv1 DEBUG]: IP = 192.1.1.40, processing SA payload             (1)
[IKEv1 DEBUG]: IP = 192.1.1.40, Oakley proposal is acceptable
output omitted
[IKEv1 DEBUG]: IP = 192.1.1.40, Received NAT-Traversal ver 03 VID (2)
output omitted
[IKEv1 DEBUG]: IP = 192.1.1.40, processing IKE SA (3)
[IKEv1 DEBUG]: IP = 192.1.1.40, IKE SA Proposal # 1, (4)
Transform # 1 acceptable Matches global IKE entry # 2
[IKEv1 DEBUG]: IP = 192.1.1.40, constructing ISA_SA for isakmp (5)
output omitted
[IKEv1 DEBUG]: IP = 192.1.1.40, processing ke payload
[IKEv1 DEBUG]: IP = 192.1.1.40, processing ISA_KE
[IKEv1 DEBUG]: IP = 192.1.1.40, processing nonce payload
[IKEv1 DEBUG]: IP = 192.1.1.40, processing VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Received Cisco Unity client VID
[IKEv1 DEBUG]: IP = 192.1.1.40, processing VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Received DPD VID
[IKEv1 DEBUG]: IP = 192.1.1.40, processing VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Processing IOS/PIX Vendor ID payload
(version: 1.0.0, capabilities: 0000077f)
[IKEv1 DEBUG]: IP = 192.1.1.40, processing VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Received xauth V6 VID
[IKEv1 DEBUG]: IP = 192.1.1.40, constructing ke payload
[IKEv1 DEBUG]: IP = 192.1.1.40, constructing nonce payload
[IKEv1 DEBUG]: IP = 192.1.1.40, constructing Cisco Unity VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, constructing xauth V6 VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Send IOS VID
[IKEv1 DEBUG]: IP = 192.1.1.40, Constructing ASA spoofing IOS Vendor
ID payload (version: 1.0.0, capabilities: 20000001)
[IKEv1 DEBUG]: IP = 192.1.1.40, constructing VID payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Send Altiga/Cisco
VPN3000/Cisco ASA GW VID
[IKEv1]: IP = 192.1.1.40, Connection landed on tunnel_group (6)
192.1.1.40
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Generating keys
for Responder...
[IKEv1]: IP = 192.1.1.40, IKE DECODE SENDING Message (msgid=0) with
payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13)
+ VENDOR (13) + VENDOR (13) + NONE (0) total length : 256
[IKEv1]: IP = 192.1.1.40, IKE DECODE RECEIVED Message (msgid=0) with
payloads : HDR + ID (5) + HASH (8) + IOS KEEPALIVE (14) +
NOTIFY (11) + NONE (0) total length : 112
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Processing ID (7)
[IKEv1 DECODE]: ID_IPV4_ADDR ID received 192.1.1.40
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, processing hash
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, computing hash
[IKEv1 DEBUG]: IP = 192.1.1.40, Processing IOS keep alive payload:
proposal=30/10 sec.
[IKEv1 DEBUG]: IP = 192.1.1.40, Starting IOS keepalive monitor:
80 sec.
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Processing
Notify payload
[IKEv1]: IP = 192.1.1.40, Connection landed on tunnel_group
192.1.1.40
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, constructing ID
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, construct hash
payload
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, computing hash
[IKEv1 DEBUG]: IP = 192.1.1.40, Constructing IOS keep alive (8)
payload: proposal=32767/32767 sec.
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40,
constructing dpd vid payload
output omitted
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, PHASE 1 COMPLETED (9)
[IKEv1]: IP = 192.1.1.40, Keep-alive type for this connection: DPD
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Starting
phase 1 rekey timer: 82080000 (ms)
[IKEv1 DECODE]: IP = 192.1.1.40, IKE Responder starting QM:
msg id = 4a9a7c8b
[IKEv1]: IP = 192.1.1.40, IKE DECODE RECEIVED Message (10)
(msgid=4a9a7c8b) with payloads : HDR + HASH (8) + SA (1) +
NONCE (10) + ID (5) + ID (5) + NONE (0) total length : 172
output omitted
[IKEv1 DECODE]: ID_IPV4_ADDR_SUBNET ID received-- (11)
192.168.0.0--255.255.255.0
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Received remote IP
Proxy Subnet data in ID Payload: Address 192.168.0.0,
Mask 255.255.255.0, Protocol 0, Port 0
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Processing ID
[IKEv1 DECODE]: ID_IPV4_ADDR_SUBNET ID received--
192.168.2.0--255.255.255.0
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Received local IP Proxy
Subnet data in ID Payload: Address 192.168.2.0,
Mask 255.255.255.0, Protocol 0, Port 0
[IKEv1]: QM IsRekeyed old sa not found by addr
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Static Crypto Map (12)
check, checking map = mymap, seq = 10...
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Static Crypto Map
check, map mymap, seq = 10 is a successful match
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, IKE Remote Peer
configured for SA: mymap
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, processing IPSEC SA
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, IPsec SA (13)
Proposal # 1, Transform # 1 acceptable Matches global IPsec
SA entry # 10
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, IKE: requesting SPI!
[IKEv1 DEBUG]: IKE got SPI from key engine: SPI = 0xcc3dcb5a
output omitted
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Transmitting (14)
Proxy Id: Remote subnet: 192.168.0.0 Mask 255.255.255.0
Protocol 0 Port 0 Local subnet: 192.168.2.0
mask 255.255.255.0 Protocol 0 Port 0
output omitted
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, loading all (15)
IPSEC SAs
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Generating
Quick Mode Key!
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Generating
Quick Mode Key!
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Security (16)
negotiation complete for LAN-to-LAN Group (192.1.1.40)
Responder, Inbound SPI = 0xcc3dcb5a, Outbound SPI = 0x382e1cb2
[IKEv1 DEBUG]: IKE got a KEY_ADD msg for SA: SPI = 0x382e1cb2
[IKEv1 DEBUG]: pitcher: rcv KEY_UPDATE, spi 0xcc3dcb5a
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Starting P2 Rekey timer
to expire in 3420 seconds
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, PHASE 2 COMPLETED (17)
(msgid=4a9a7c8b)
[IKEv1 DEBUG]: Group = 192.1.1.40, IP = 192.1.1.40, Sending (18)
keep-alive of type DPD R-U-THERE (seq number 0x3252ed2c)

Image from book

Here’s a brief description of the references in Example 23-3 (the output the debug crypto isakmp command is very verbose, so I’ve omitted some of it):

  1. Main mode exchange is beginning; no policies have been shared yet and the peers are still in an MM_NO_STATE.

  2. The remote peer is testing for the use of NAT-T.

  3. The comparison of ISAKMP/IKE policies begins here.

  4. This message indicates that a matching policy has been found.

  5. The management connection is being built.

  6. The peer is associated with the “192.1.1.40” L2L tunnel group and the encryption and hash keys are being generated.

  7. This is where authentication begins with pre-shared keys: remember that authentication occurs on both peers, and thus you’ll see two sets of corresponding authentication processes.

  8. DPD is being negotiated.

  9. Phase 1 is complete.

  10. Phase 2 (quick mode) begins.

  11. The remote subnet (192.168.0.0/24) is received and compared to the local subnet (192.168.2.0/24).

  12. A matching static crypto entry is looked for and found.

  13. The appliance finds a matching data transform for the data connections.

  14. A check is performed for mirrored crypto ACLs.

  15. Keys are generated for the data SAs.

  16. SPIs are assigned to the data SAs.

  17. Phase 2 completes.

  18. A DPD keepalive is being sent to the remote peer on the management connection.


Caution

Also in 7.0, you control the debugging level by specifying a number from 1–255 after the debug command: this affects the amount of output you see from the debug command. A level of 1 will give you little information, if any, and 255 will show partial packet contents (the most in-depth). Therefore, you’ll want to specify a number like 100 or 150 for the debug level to give you a reasonable amount of output to troubleshoot problems. Also, if you enter the debug command without specifying a level number, it defaults to level 1; therefore, if you don’t see any output from your debug command, this is the first thing I would check: use the show debug command to determine what debug functions you’ve enabled and for what output level they’ve been configured.

If there is a mismatch in the ISAKMP/IKE Phase 1 policy, between the peers, your debug output will look like that in Example 23-4.

Example 23-4: Mismatch ISAKMP/IKE Phase 1 Policies in 7.0
Image from book
[IKEv1 DEBUG]: IP = 192.1.1.40, processing SA payload
[IKEv1]: IP = 192.1.1.40, IKE DECODE SENDING Message (msgid=0)
with payloads : HDR + NOTIFY (11) + NONE (0) total length : 100
[IKEv1 DEBUG]: IP = 192.1.1.40, All SA proposals found unacceptable
[IKEv1]: IP = 192.1.1.40, Error processing payload: Payload ID: 1
[IKEv1 DEBUG]: IP = 192.1.1.40, IKE MM Responder FSM error
history (struct &0x19f49a0) , : MM_DONE,
EV_ERROR-->MM_START, EV_RCV_MSG-->MM_START, EV_
START_MM-->MM_START, EV_START_MM
[IKEv1 DEBUG]: IP = 192.1.1.40, IKE SA MM:2d31c23f terminating:
flags 0x01000002, refcnt 0, tuncnt 0
[IKEv1 DEBUG]: sending delete/delete with reason message

Image from book


Tip

I’ve found out, the hard way, in certain FOS releases that even if the ISAKMP policies match on the two peers, you still could get the dreaded “All SA proposals found unacceptable” message. Apparently, certain combinations of policies, especially for the two data connections, will not work, even though you can configure them on the appliance, like AES-128 and SHA. Playing around, I’ve found out that in most instances, any encryption algorithm and MD5 will work; but only certain encryption algorithms and SHA will work. Therefore, before you waste your time troubleshooting this problem with either a Phase 1 or, more likely, Phase 2 connection, first I would try a proposal that supported MD5 with your selected encryption algorithm.

If there is a mismatch in a key used for pre-shared key authentication, the output of the debug crypto isakmp command will look like that found in Example 23-5.

Example 23-5: Mismatched Pre-shared Key Illustration in 7.0
Image from book
[IKEv1 DEBUG]: IP = 192.1.1.40, processing SA payload
[IKEv1 DEBUG]: IP = 192.1.1.40, Oakley proposal is acceptable
←output omitted→
[IKEv1 DEBUG]: IP = 192.1.1.40, IKE SA Proposal # 1,
Transform # 1 acceptable Matches global IKE entry # 3
←output omitted→
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Received
encrypted Oakley Main Mode packet with invalid payloads,
MessID = 0
[IKEv1]: IP = 192.1.1.40, IKE DECODE SENDING Message (msgid=0)
with payloads : HDR + NOTIFY (11) + NONE (0) total length : 136
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, ERROR, had problems
decrypting packet, probably due to mismatched pre-shared key.
Aborting
[IKEv1]: Group = 192.1.1.40, IP = 192.1.1.40, Duplicate Phase 1
packet detected. Retransmitting last packet.
output omitted
Jun 29 17:39:09 [IKEv1 DEBUG]: sending delete/delete with reason message
output omitted

Image from book


Tip

One of the problems I’ve seen with the output of the FOS debug commands is that the nomenclature and verbiage has a tendency of changing from one FOS release to another. This is very apparent if you compare the 6.3 output to that of 7.0. Therefore, you have to scrutinize the output carefully to determine the exact problem. In certain cases, you might want to look at the debug output from a connection that works from the same FOS revision that you’re using on a connection that is failing. You can use the successful connection output as a baseline when comparing this debug output to the debug output from a failed connection attempt. I’ve also found the FOS 7.0 debug output more user-friendly in deciphering its messages than with the 6.x and earlier releases.

Remote Access Sessions

The debug output from setting up a remote access session can be very verbose—about 20 pages in length! Example 23-6 shows the output of the debug crypto isakmp command from a 7.0 Easy VPN Server, where I’ve omitted much of the output to keep it brief. I explain the numbered references below the example output.

xample 23-6: Establishing a Remote Access Connection to an Easy VPN Server Running 7.0
Image from book
[IKEv1 DEBUG]: IP = 192.1.1.77, processing SA payload             (1)
output omitted
[IKEv1 DEBUG]: IP = 192.1.1.77, IKE Peer included IKE
fragmentation capability flags: Main Mode: True
Aggressive Mode: False
[IKEv1 DEBUG]: IP = 192.1.1.77, processing VID payload
[IKEv1 DEBUG]: IP = 192.1.1.77, Received Cisco Unity client VID (2)
[IKEv1]: IP = 192.1.1.77, Connection landed on tunnel_
group salesgroup
[IKEv1 DEBUG]: Group = salesgroup, IP = 192.1.1.77, processing
IKE SA
[IKEv1 DEBUG]: Group = salesgroup, IP = 192.1.1.77, IKE SA (3)
Proposal # 1, Transform # 5 acceptable Matches global
IKE entry # 1
[IKEv1 DEBUG]: Group = salesgroup, IP = 192.1.1.77, constructing
ISA_SA for isakmp
[IKEv1 DEBUG]: Group = salesgroup, IP = 192.1.1.77, constructing
nonce payload
output omitted
[IKEv1 DEBUG]: Processing MODE_CFG Reply attributes. (4)
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, IKEGetUserAttributes: primary DNS = 4.2.2.1
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, IKEGetUserAttributes: secondary DNS = cleared
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, IKEGetUserAttributes: primary WINS = cleared
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, IKEGetUserAttributes: secondary WINS = cleared
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, IKEGetUserAttributes: IP Compression = disabled
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, IKEGetUserAttributes: Split Tunneling
Policy = Disabled
[IKEv1]: Group = salesgroup, Username = salesuser, (5)
IP = 192.1.1.77, User (salesuser) authenticated.
output omitted
[IKEv1 DEBUG]: Processing cfg Request attributes (6)
[IKEv1 DEBUG]: MODE_CFG: Received request for IPV4 address!
[IKEv1 DEBUG]: MODE_CFG: Received request for IPV4 net mask!
[IKEv1 DEBUG]: MODE_CFG: Received request for DNS server address!
[IKEv1 DEBUG]: MODE_CFG: Received request for WINS server address!
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Received unsupported transaction mode
attribute: 5
[IKEv1 DEBUG]: MODE_CFG: Received request for Banner!
[IKEv1 DEBUG]: MODE_CFG: Received request for Save PW setting!
[IKEv1 DEBUG]: MODE_CFG: Received request for Default Domain Name!
[IKEv1 DEBUG]: MODE_CFG: Received request for Split Tunnel List!
[IKEv1 DEBUG]: MODE_CFG: Received request for Split DNS!
[IKEv1 DEBUG]: MODE_CFG: Received request for PFS setting!
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Received unknown transaction mode attribute: 28683
[IKEv1 DEBUG]: MODE_CFG: Received request for backup ip-sec peer
list!
[IKEv1 DEBUG]: MODE_CFG: Received request for Application (7)
Version!
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Client Type: WinNT Client Application
Version: 4.6.01.0019
[IKEv1 DEBUG]: MODE_CFG: Received request for FWTYPE!
[IKEv1 DEBUG]: MODE_CFG: Received request for DHCP hostname for
DDNS is: i7500!
[IKEv1 DEBUG]: MODE_CFG: Received request for UDP Port!
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser, (8)
IP = 192.1.1.77, constructing blank hash
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, constructing qm hash
[IKEv1]: IP = 192.1.1.77, IKE DECODE SENDING Message
(msgid=e9f26b16) with payloads : HDR + HASH (8) + ATTR (14)
+ NONE (0) total length : 170
[IKEv1 DECODE]: IP = 192.1.1.77, IKE Responder starting QM:
msg id = d9fcc34b
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Delay Quick Mode processing, Cert/Trans
Exch/RM DSID in progress
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Resume Quick Mode processing, Cert/Trans
Exch/RM DSID completed
[IKEv1]: Group = salesgroup, Username = salesuser, (9)
IP = 192.1.1.77, PHASE 1 COMPLETED
output omitted
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser, (10)
IP = 192.1.1.77, constructing blank hash
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, constructing qm hash
[IKEv1]: IP = 192.1.1.77, IKE DECODE SENDING Message
(msgid=3b776e14) with payloads : HDR + HASH (8) +
NOTIFY (11) + NONE (0) total length : 92
[IKEv1]: IP = 192.1.1.77, IKE DECODE RECEIVED Message
(msgid=d9fcc34b) with payloads : HDR + HASH (8) + SA (1)
+ NONCE (10) + ID (5) + ID (5) + NONE (0) total length : 1026
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, processing hash
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, processing SA payload
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, processing nonce payload
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Processing ID
[IKEv1 DECODE]: ID_IPV4_ADDR ID received 192.168.2.200 (11)
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Received remote Proxy Host data in ID
Payload: Address 192.168.2.200, Protocol 0, Port 0
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Processing ID
[IKEv1 DECODE]: ID_IPV4_ADDR_SUBNET ID received--
0.0.0.0--0.0.0.0
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Received local IP Proxy Subnet data in ID
Payload: Address 0.0.0.0, Mask 0.0.0.0, Protocol 0, Port 0
[IKEv1]: QM IsRekeyed old sa not found by addr (12)
[IKEv1]: Group = salesgroup, Username = salesuser, (13)
IP = 192.1.1.77, Static Crypto Map check, checking
map = mymap, seq = 10...
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Static Crypto Map check, map = mymap,
seq = 10, ACL does not match proxy IDs src:192.168.2.200
dst:0.0.0.0
[IKEv1]: Group = salesgroup, Username = salesuser, (14)
IP = 192.1.1.77, IKE Remote Peer configured for SA: dynmap
[IKEv1]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, processing IPSEC SA
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser, (15)
IP = 192.1.1.77, IPsec SA Proposal # 11, Transform # 1
acceptable Matches global IPsec SA entry # 1
output omitted
[IKEv1]: Group = salesgroup, Username = salesuser, (16)
IP = 192.1.1.77, Overriding Initiator's IPsec rekeying
duration from 2147483 to 28800 seconds
output omitted
[IKEv1]: Group = salesgroup, Username = salesuser, (17)
IP = 192.1.1.77, Security negotiation complete for
User (salesuser) Responder, Inbound SPI = 0x46ffd888,
Outbound SPI = 0xfc4dd2f3
[IKEv1 DEBUG]: IKE got a KEY_ADD msg for SA: SPI = 0xfc4dd2f3
[IKEv1 DEBUG]: pitcher: rcv KEY_UPDATE, spi 0x46ffd888
output omitted
[IKEv1]: Group = salesgroup, Username = salesuser, (18)
IP = 192.1.1.77, Adding static route for client address:
192.168.2.200
[IKEv1]: Group = salesgroup, Username = salesuser, (19)
IP = 192.1.1.77, PHASE 2 COMPLETED (msgid=d9fcc34b)
output omitted
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser, (20)
IP = 192.1.1.77, Received keep-alive of type DPD R-U-THERE
(seq number 0xa780a31f)
[IKEv1 DEBUG]: Group = salesgroup, Username = salesuser,
IP = 192.1.1.77, Sending keep-alive of type DPD R-U-THERE-ACK
(seq number 0xa780a31f)
output omitted

Image from book

Here’s an explanation of the debug output from Example 23-6:

  1. The Remote (192.1.1.77) initiates a session to the appliance (acting as a Server).

  2. The Remote sends its identity type to the Server, along with the group it wants to connect to (“salesgroup”).

  3. A matching Phase 1 policy is found: policy 5 of the Remote matches the first policy of the Server).

  4. The Remote initiates IKE Mode Config and the appliance is determining which parameters it has configured for the associated group.

  5. The group authentication is successful, as is the XAUTH authentication via the user account “salesuser”; notice that this message appears here rather than before IKE Mode Config, because the appliance needs to verify whether or not the user is allowed access to the group.

  6. The Remote sends an IKE Mode Config request for the policies defined for the salesgroup group.

  7. During IKE Mode Config, the appliance learns the client type and version.

  8. The Server sends back the IKE Mode Config parameters.

  9. This completes ISAKMP/IKE Phase 1.

  10. Quick mode begins with an exchange of policies.

  11. The internal address of the client is 192.168.2.200 and the proxy message it sends indicates that all of its traffic is to be protected (the group policy is split tunneling disabled).

  12. A check is performed to make sure that the client isn’t reconnecting (the Initial Contact feature for Easy VPN); in this example, the client is initiating a new connection.

  13. The appliance compares the proxy information with its first crypto map entry (which is a static one) and finds that it doesn’t match this entry (the proxy information doesn’t match).

  14. The appliance compares the proxy information with its second crypto map entry, which is a dynamic crypto map for remote access users.

  15. A matching data transform is found.

  16. There is a difference in the data SA lifetime values between the two devices: the lower one (28,800 seconds) is negotiated.

  17. The two IPsec data SAs (inbound and outbound) are created and SPIs are assigned.

  18. Because RRI is enabled, a static route for the Remote’s internal address (192.168.2.200) is added to the Server’s local routing table.

  19. Phase 2 has completed.

  20. Because DPD was negotiated in Phase 1, DPD now takes place; in this instance, the Remote is initiating DPD (however, both sides of the tunnel will do this periodically based on their local keepalive counters).

The debug crypto vpnclient Command

For 6.x PIXs configured as Easy VPN Remotes, you can use the debug crypto vpnclient command to troubleshoot client-specific configuration and connection setup issues. Example 23-7 illustrates the use of this command, where the client is using network extension mode. Below the example, I explain the numbered references found in the example.

Example 23-7: Establishing a Remote Access Connection from an Easy VPN Remote Running 6.3
Image from book
VPNC CFG: transform set unconfig attempt done                     (1)
VPNC CLI: no isakmp keepalive 10 5
VPNC CLI: no isakmp nat-traversal 20
VPNC CFG: IKE unconfig successful
VPNC CLI: no crypto map _vpnc_cm
VPNC CFG: crypto map deletion attempt done
VPNC CFG: crypto unconfig successful
VPNC CLI: no global (outside) 65001
VPNC CLI: no nat (inside) 0 access-list _vpnc_acl
VPNC CFG: nat unconfig attempt failed
VPNC CLI: no http 192.168.3.1 255.255.255.0 inside
VPNC CLI: no http server enable
VPNC CLI: no access-list _vpnc_acl
VPNC CFG: ACL deletion attempt failed
VPNC CLI: no crypto map _vpnc_cm interface outside
VPNC CFG: crypto map de/attach failed
VPNC CFG: transform sets configured (2)
VPNC CFG: crypto config successful
VPNC CLI: isakmp keepalive 10 5
VPNC CLI: isakmp nat-traversal 20
VPNC CFG: IKE config successful
VPNC CLI: http 192.168.3.1 255.255.255.0 inside
VPNC CLI: http server enable
VPNC CLI: aaa-server _vpnc_nwp_server protocol tacacs+
VPNC CLI: aaa-server _vpnc_nwp_server (outside) host 192.1.1.100
VPNC CLI: access-list _vpnc_nwp_acl permit ip any any
VPNC CLI: aaa authentication match _vpnc_nwp_acl outbound
vpnc_nwp_server
VPNC CLI: no access-list _vpnc_acl
VPNC CFG: ACL deletion attempt failed
VPNC CLI: access-list _vpnc_acl permit ip host 192.1.1.101 (3)
host 192.1.1.100
VPNC CLI: crypto map _vpnc_cm 10 match address _vpnc_acl
VPNC CFG: crypto map acl update successful
VPNC CLI: no crypto map _vpnc_cm interface outside
VPNC CLI: crypto map _vpnc_cm interface outside
VPNC INF: IKE trigger request done (4)
VPNC INF: Constructing policy download req
VPNC INF: Packing attributes for policy request
VPNC INF: Attributes being requested
VPNC ATT: INTERNAL_IP4_DNS: 4.2.2.1
VPNC ATT: ALT_PFS: 0
VPNC INF: Received application version 'Cisco Systems, Inc (5)
PIX-515 Version 7.0(1) built by builders on
Thu 31-Mar-05 14:37'
VPNC ATT: ALT_CFG_SEC_UNIT: 0
VPNC ATT: ALT_CFG_USER_AUTH: 0
VPNC CLI: no aaa authentication match _vpnc_nwp_acl outbound _
vpnc_nwp_server
VPNC CLI: no access-list _vpnc_nwp_acl permit ip any any
VPNC CLI: no aaa-server _vpnc_nwp_server
VPNC CLI: no access-list _vpnc_acl
VPNC CLI: access-list _vpnc_acl permit ip (6)
192.168.3.0 255.255.255.0 any
VPNC CLI: access-list _vpnc_acl permit ip
host 192.1.1.101 any
VPNC CLI: access-list _vpnc_acl permit ip
host 192.1.1.101 host 192.1.1.100
VPNC CFG: _vpnc_acl no ST define done
VPNC CLI: crypto map _vpnc_cm 10 match address _vpnc_acl
VPNC CFG: crypto map acl update successful
VPNC CLI: no crypto map _vpnc_cm interface outside
VPNC CLI: crypto map _vpnc_cm interface outside
VPNC CLI: no global (outside) 65001 (7)
VPNC CLI: no nat (inside) 0 access-list _vpnc_acl
VPNC CFG: nat unconfig attempt failed
VPNC CLI: nat (inside) 0 access-list _vpnc_acl
VPNC INF: IKE trigger request done (8)
output omitted

Image from book

Here is an explanation of the numbered references in Example 23-7:

  1. This is the first time the VPN Remote functionality was enabled on the PIX, so the PIX is first removing any VPN commands that could cause any type of conflict.

  2. After attempting to remove all VPN-related commands, the Remote then configures the necessary VPN commands.

  3. Add a note hereAn ACL is built to allow communications between this PIX (192.1.1.101) and the Easy VPN Server (192.1.1.100).

  4. The PIX Remote initiates its connection to the Server and sends its policies.

  5. The Server is a PIX 515 running FOS 7.0.

  6. Based on the split tunneling policy passed to it by the Server, the client PIX builds an appropriate crypto ACL.

  7. Based on the split tunneling policy, the appropriate address translation policy is configured.

  8. The tunnel is now established to the Server.


Tip

Unfortunately, the debug crypto vpnclient command is not that useful for troubleshooting the setup of an IPsec session. If something is misconfigured on the Remote or Server, you’ll see something like that in Example 23-8 repeated over and over; however, as you’ll notice in the output, there is nothing that indicates what the problem is. In this example, the Remote was configured for network extension mode, but the group on the Server didn’t have this policy defined. For example, with the output of the debug crypto isakmp command on the Server, you would see a message like this: “[IKEv1]: Group = salesgroup, Username = salesuser, IP = 192.1.1.101, Hardware Client connection rejected! Network Extension Mode is not allowed for this group!” Unfortunately, the debug output from the same command on a 6.3 Remote isn’t as verbose, making the troubleshooting of this problem more difficult, if not impossible, from the Remote end using the debug crypto vpnclient command.

Example 23-8: A Failed Remote Access Connection from an Easy VPN Remote Running 6.3
Image from book
VPNC INF: Constructing policy download req
VPNC INF: Packing attributes for policy request
VPNC INF: Attributes being requested
VPNC CLI: no access-list _vpnc_acl
VPNC CLI: access-list _vpnc_acl permit ip host 192.1.1.101 host 192.1.1.100
VPNC CLI: crypto map _vpnc_cm 10 match address _vpnc_acl
VPNC CFG: crypto map acl update successful
VPNC CLI: no crypto map _vpnc_cm interface outside
VPNC CLI: crypto map _vpnc_cm interface outside
VPNC INF: IKE trigger request done

Image from book