Chapter 23: Troubleshooting PIX and ASA Connections (Part 2)
[3/22/2011 02:44:00 PM | ]

ISAKMP/IKE Phase 2 Connections

In this section I’ll discuss some security appliance commands you can use to troubleshoot ISAKMP/IKE Phase 2 connections. I’ll begin by briefly describing the commands you can use and then, in later sections, I’ll discuss some of these commands in more depth.

Overview of the Phase 2 Commands

If you’re experiencing problems with establishing IPsec data connections with an IPsec peer, you could use several PIX/ASA commands to help pinpoint the problem. Here’s a brief summary of these commands:

  • show crypto engine [verify]— Displays the usage statistics for the appliance’s crypto engine (FOS 6.x only); the verify parameter runs the Known Answer Test (KAT), which checks the integrity of the cryptography engine used by the appliance.

  • show crypto interface [counters]— Displays the VAC/VAC+ card installed in the appliance and, optionally, traffic statistics for the card (FOS 6.x only).

  • show crypto accelerator statistics— Displays the VAC/VAC+ card installed in the appliance and, optionally, traffic statistics for the card (FOS 7.0 only).

  • show crypto protocol statistics {ikev1 | ipsec}— Displays general traffic statistics about the management or data connections (FOS 7.0 only).

  • show [crypto] ipsec sa— Displays the data SAs established between two IPsec peers, and the components used to protect the connection and packet statistical information.

  • debug crypto isakmp— Displays the steps taken to build a management connection and data connections via the management connection (see “The debug crypto isakmp Command” section previously in the chapter).

  • debug crypto ipsec— Displays the actual creation of the two unidirectional data SAs between two peers.

  • clear crypto [ipsec] sa [counters | mapmap_name|peer IP_address| entry IP_address{ah | esp} SPI_#]— Clears the statistics (counters), all data SAs associated with a crypto map (map), all data SAs associated with a peer (peer), or a particular data SA to a particular peer (entry).

The following sections will discuss some of the above troubleshooting commands in more depth; these commands discussed below are the more common ones used to troubleshoot data SA problems.

The show crypto ipsec sa Command

The show crypto ipsec sa command displays the crypto map entry information used to build data connections, and any existing data connections to remote peers. There are two forms of the command, depending on which FOS version your appliance is running.

In FOS 6.x and earlier, the following syntax applies:

pix63# show crypto [ipsec] sa [map map_name | address | identity]

Without any specified parameters, the crypto map information used to create the data SAs is displayed, in addition to traffic statistics, and the inbound and outbound connections and their SPI numbers. The map parameter allows you to display only the SAs associated with the specified crypto map name. The address parameter sorts the output based on the IP address of the SA. The identity parameter sorts the display by SA flows.

In FOS 7.0, the following syntax can be used:

pix70# show crypto [ipsec] sa [entry | identity| map map_name |
                       peer peer_IP_address ] [detail]

As you can see from the syntax of this command, it is similar to the 6.x version. The entry parameter performs the same function as the address parameter in the 6.x command; it sorts the SAs based on IP addresses. The identity parameter sorts the display by SA flows. The map parameter displays only the SAs associated with the specified crypto map. The peer parameter allows you to display only the SAs established to the specified peer. The detail parameter displays more detailed information about the SAs, including error information.

In the case of either of these two commands, the output produced, shown in Example 23-9, is very similar to a router’s output from the same command. The output in Example 23-9 is from a security appliance running 7.0, even though the 6.x output is almost the same. The crypto map called “mymap” has an entry associated with a remote peer, 192.1.1.40. The local ident and remote ident entries display the traffic that is to be protected (traffic between 192.168.2.0/24 and 192.168.0.0/24). The #pkts encaps and #pkts decaps display the number of packets encapsulated/deencapsulated using IPsec (AH and/or ESP); likewise, you can see the number of packets encrypted and decrypted, and the number of packets where a hash function was created or verified. Given that there are nonzero numbers in these fields, a connection is currently established to the remote peer (192.1.1.40).

Example 23-9: Using the show crypto ipsec sa Command
Image from book 
pix70(config)# show crypto ipsec sa
interface: outside
   Crypto map tag: mymap, local addr: 192.1.1.100
     local ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
     remote ident (addr/mask/prot/port): (192.168.0.0/255.255.255.0/0/0)
     current_peer: 192.1.1.40

     #pkts encaps: 4, #pkts encrypt: 4, #pkts digest: 4
     #pkts decaps: 4, #pkts decrypt: 4, #pkts verify: 4
     #pkts compressed: 0, #pkts decompressed: 0
     #pkts not compressed: 4, #pkts comp failed: 0, #pkts decomp failed: 0
     #send errors: 0, #recv errors: 0

     local crypto endpt.: 192.1.1.100, remote crypto endpt.: 192.1.1.40
     path mtu 1500, ipsec overhead 76, media mtu 1500
     current outbound spi: 2ED644AD

inbound esp sas:
 spi: 0x76DFE868 (1994385512)
    transform: esp-aes esp-sha-hmac
    in use settings ={L2L, Tunnel, }
    slot: 0, conn_id: 1, crypto-map: mymap
    sa timing: remaining key lifetime (kB/sec): (4274999/3586)
    IV size: 16 bytes
    replay detection support: Y
outbound esp sas:
 spi: 0x2ED644AD (785794221)
    transform: esp-aes esp-sha-hmac
    in use settings ={L2L, Tunnel, }
    slot: 0, conn_id: 1, crypto-map: mymap
    sa timing: remaining key lifetime (kB/sec): (4274999/3584)
    IV size: 16 bytes
    replay detection support: Y
Image from book

The SAs are displayed in separate sections in Example 23-9. In this example, only ESP is used, so you can see the SPI values, transforms, tunnel type, and other connection particulars in the inbound esp sas and outbound esp sas sections of the output. If you don’t see anything under these subsections, no data connections have been established. Common problems that might cause this situation are:

  • Mismatch in transforms

  • Mismatch in crypto ACLs

  • Mismatch in addresses that the two peers will use for IPsec communications

Further troubleshooting can be done with the debug crypto ipsec command. I’ll discuss that in the next section.

The debug crypto ipsec Command

If you’re experiencing problems establishing the two IPsec data connections between peers, the most common appliance commands to troubleshoot this problem are the debug crypto isakmp and the debug crypto ipsec commands. I discussed the former command earlier in “The debug crypto isakmp Command” section. The debug crypto ipsec command is supported by both 6.x and 7.0; the only difference is that in 7.0, you need to specify a debug level from 1–255, where 255 displays partial packet contents. Normally I set it to 150, which gives me enough information to troubleshoot a problem. With 6.3 and earlier, there is no option of specifying a debug level, as is the case with Cisco routers.

Example 23-10 illustrates the use of this command in 6.3, where the two data connections between two peers are established successfully. In this example, an appliance is accepting an L2L connection request from a remote peer. The local appliance’s address is 192.1.1.101 with a local subnet of 192.168.3.0/24. Below the example is an explanation of the debug output reference numbers.

Example 23-10: Successful IPsec Data SAs Established on a 6.3 Appliance
Image from book 
IPSEC(key_engine): got a queue event...
IPSEC(key_engine_delete_sas): rec'd delete notify from ISAKMP     (1)
IPSEC(key_engine_delete_sas): delete all SAs shared with
   192.1.1.40
IPSEC(validate_proposal_request): proposal part #1,               (2)
 (key eng. msg.) dest= 192.1.1.101, src= 192.1.1.40,
   dest_proxy= 192.168.3.0/255.255.255.0/0/0 (type=4),
   src_proxy= 192.168.0.0/255.255.255.0/0/0 (type=4),
   protocol= ESP, transform= esp-aes esp-md5-hmac ,
   lifedur= 0s and 0kb,
   spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x4
IPSEC(key_engine): got a queue event...
IPSEC(spi_response): getting spi 0xffc3de48(4291026504) for SA    (3)
       from 192.1.1.40 to 192.1.1.101 for prot 3
IPSEC(key_engine): got a queue event...
IPSEC(initialize_sas): ,                                          (4)
 (key eng. msg.) dest= 192.1.1.101, src= 192.1.1.40,
   dest_proxy= 192.168.3.0/255.255.255.0/0/0 (type=4),
   src_proxy= 192.168.0.0/255.255.255.0/0/0 (type=4),
   protocol= ESP, transform= esp-aes esp-md5-hmac ,
   lifedur= 3600s and 4608000kb,
   spi= 0xffc3de48(4291026504), conn_id= 2, keysize= 128,
   flags= 0x4
IPSEC(initialize_sas): ,                                          (5)
 (key eng. msg.) src= 192.1.1.101, dest= 192.1.1.40,
   src_proxy= 192.168.3.0/255.255.255.0/0/0 (type=4),
   dest_proxy= 192.168.0.0/255.255.255.0/0/0 (type=4),
   protocol= ESP, transform= esp-aes esp-md5-hmac ,
   lifedur= 3600s and 4608000kb,
   spi= 0x378ef8b8(932116664), conn_id= 1, keysize= 128,
   flags= 0x4
Image from book

Here’s a brief explanation of the numbered references in the output from Example 23-10:

  1. Any existing data SAs are being deleted before the new ones are added between the two peers: 192.1.1.101 is the local appliance and 192.1.1.40 is the remote peer.

  2. A matching transform set (ESP with AES-128 and MD5) and crypto ACL are found; in this example, the remote peer initiated the connection, requesting that traffic from 192.168.0.0/24 to 192.168.3.0/24 be protected. This can be determined by the “validate proposal request” message.

  3. The remote peer assigns an SPI for the SA from the remote peer to the local appliance.

  4. The SA from the local appliance to the remote peer is initialized and assigned an SPI value.

  5. The SA from the remote peer to the local appliance is initialized.

As you can see from this output, it is much less verbose than what you would see with the debug crypto isakmp command; this makes sense because only the particulars of the building of the data connection are displayed, whereas with the debug crypto isakmp command, you see everything that ISAKMP/IKE builds, which includes both the management and data connections.

The following sections will cover some common problems with establishing data connections, including:

  • Mismatched Data Transforms

  • Mismatched Crypto ACLs

  • Matching on the Incorrect Crypto Map Entry

Mismatched Data Transforms

As you can see from the output in Example 23-10, the debug output from the debug crypto ipsec command is fairly straightforward to interpret. Of course, not all data SAs are built successfully. For example, if you don’t have a matching transform for the data connections, you’ll see the output in Example 23-11 from the debug crypto isakmp and the debug crypto ipsec commands. The first part is from the former debug command (begins with “ISKAMP”) and the last part is from the latter command (begins with “IPSEC”). Use the show crypto ipsec transform-set command on the local appliance to determine which transforms have been created already, and the show crypto map command to see which transform set has been associated with the remote peer’s crypto map entry.

Example 23-11: Mismatched IPsec Data Transforms
Image from book 
←output omitted→
ISAKMP (0): processing SA payload. message ID = 2686916944
ISAKMP : Checking IPsec proposal 1
ISAKMP: transform 1, ESP_AES
ISAKMP:   attributes in transform:
ISAKMP:      encaps is 1
ISAKMP:      SA life type in seconds
ISAKMP:      SA life duration (basic) of 3600
ISAKMP:      SA life type in kilobytes
ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0
ISAKMP:      authenticator is HMAC-SHA
ISAKMP:      key length is 128IPSEC(validate_proposal):
   transform proposal (prot 3, trans 12, hmac_alg 2) not supported
ISAKMP (0): atts not acceptable. Next payload is 0
ISAKMP (0): SA not acceptable!
←output omitted→
IPSEC(key_engine): got a queue event...
IPSEC(key_engine_delete_sas): rec'd delete notify from ISAKMP
IPSEC(key_engine_delete_sas): delete all SAs shared with 192.1.1.40
IPSEC(validate_proposal): transform proposal (prot 3, trans 12,
   hmac_alg 2) not supported
Image from book

Add a note hereAt the beginning of Example 23-11, you can see the output from the debug crypto isakmp command. The first two highlighted lines display that there is a problem with an ESP proposal that the remote peer wants to use: ESP with AES-128 and SHA. The last part of the output shows that the transform proposed by the remote peer wasn’t accepted because of a conflict in the HMAC function defined on both peers. Obviously, the local peer wants to use MD5, but the remote peer has been configured with only one transform that has SHA.

Mismatched Crypto ACLs

Add a note hereIf the crypto ACLs are not mirrored on the two peers, you’ll see debug output from the debug crypto ipsec and debug crypto isakmp commands shown in Example 23-12. The proxy identities not supported message indicates that the crypto ACLs (if routers or PIXs), or network lists (if concentrators), do not match (are not mirrored) on the two IPsec peers. This misconfiguration is commonly called an “invalid proxy ID.” When this error occurs, examine the crypto ACLs (or network lists, if the peer’s a concentrator) to see where the ACL entries are not mirrored.

Example 23-12: Mismatched Crypto ACLs: Not Mirrored
Image from book 
←output omitted→
ISAKMP (0): processing SA payload. message ID = 2620452987
ISAKMP : Checking IPsec proposal 1
ISAKMP: transform 1, ESP_AES
ISAKMP:   attributes in transform:
ISAKMP:      encaps is 1
ISAKMP:      SA life type in seconds
ISAKMP:      SA life duration (basic) of 3600
ISAKMP:      SA life type in kilobytes
ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0
ISAKMP:      authenticator is HMAC-MD5
ISAKMP:      key length is 128
ISAKMP (0): atts are acceptable.IPSEC(validate_proposal_request):
   proposal part #1,
 (key eng. msg.) dest= 192.1.1.101, src= 192.1.1.40,
   dest_proxy= 192.168.3.0/255.255.255.0/0/0 (type=4),
   src_proxy= 192.168.1.0/255.255.255.0/0/0 (type=4),
   protocol= ESP, transform= esp-aes esp-md5-hmac ,
   lifedur= 0s and 0kb,
   spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x4
IPSEC(validate_transform_proposal): proxy identities not supported
IPSEC(validate_proposal_request): proposal part #1,
 (key eng. msg.) dest= 192.1.1.101, src= 192.1.1.40,
   dest_proxy= 192.168.1.0/255.255.255.0/0/0 (type=4),
   src_proxy= 192.168.3.0/255.255.255.0/0/0 (type=4),
   protocol= ESP, transform= esp-aes esp-md5-hmac ,
   lifedur= 0s and 0kb,
   spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x4
IPSEC(validate_transform_proposal): proxy identities not supported
ISAKMP: IPsec policy invalidated proposal
ISAKMP (0): SA not acceptable!
←output omitted→
Image from book


Tip

From practical experience, I’ve learned that in certain cases you’ll get the mismatched crypto ACL error even if you have the entries mirrored on the same side, but the crypto ACL entries happen to be in a different order on the two peers. Therefore, I recommend mirroring each statement and also matching up the statements in the same order.

Matching on the Incorrect Crypto Map Entry

Another uncommon problem you might experience is if there are overlapping crypto ACLs, where a match is found for a peer for the wrong crypto ACL in a wrong crypto map entry. For example, an appliance might have two crypto ACLs with overlapping entries like that found in Example 23-13. In this example, crypto ACLs 101 and 102 overlap. If the security appliance has an IPsec tunnel to 192.1.1.2, but not 192.1.1.1, and 192.1.1.2 forwards a packet from 192.168.2.1 to 192.168.1.1, it will match the crypto ACL in the first entry, thus causing the error similar to the one shown previously in Example 23-12. To solve this problem, put the crypto map entry with the more specific crypto ACL entry or entries before the less specific one; in other words, give the former a lower entry number.

Example 23-13: Overlapping Crypto ACL Entries Example
Image from book 
security(config)# access-list 101 permit ip 192.168.0.0 0.0.0.255
                       192.168.3.0 0.0.0.255
security(config)# access-list 102 permit udp host 192.168.0.10
                       host 192.168.3.25 eq 69
security(config)# crypto map mymap 10 ipsec-isakmp
security(config)# crypto map mymap 10 match address 101
security(config)# crypto map mymap 10 set peer 192.1.1.1
security(config)# crypto map mymap 10 set transform-set trans1
security(config)# crypto map mymap 20 ipsec-isakmp
security(config)# crypto map mymap 20 match address 102
security(config)# crypto map mymap 20 set peer 168.3.25
security(config)# crypto map mymap 20 set transform-set trans2
Image from book

Add a note hereIn some older FOS versions of the PIXs, you might see an “ACL = deny; no sa created” message, which also could indicate that you have a proxy mismatch:

  • Add a note hereA nonmatching mirrored crypto ACL condition exists.

  • Add a note hereThe same ACL is being used with a nat 0 access-list command and a crypto ACL.

  • Add a note hereA match on the wrong crypto map entry with overlapping crypto ACLs occurred.

Add a note hereI’ve never seen the message in the first bullet point; I’ve always seen the messages displayed (or something very similar) in Example 23-12. With a crypto ACL mismatch, you’ll see one of the following:

  • Add a note hereThe connection fails (with a proxy ID mismatch).

  • Add a note hereAn asymmetric tunnel startup condition, where the tunnel will come up when started from one peer, but not from the other.

  • Add a note hereAn asymmetric transfer condition, where data will flow across the tunnel depending on which peer brought up the tunnel, but not in the reverse direction.

Add a note hereWith the last two bullet points, you might commonly see error statements about packets being dropped that should be protected: you can also view the output of the show crypto ipsec sa command and look for packet errors or drops.

Add a note hereA bug in earlier FOS versions prevented you, in certain situations, from using the same ACL for address translation with the nat 0 access-list command and as a crypto ACL. Cisco highly encourages its customers to not use the same ACL for crypto and address translation functions; doing so might cause the proxy process, and thus the data connections, to fail.

Add a note hereThe most common problem with overlapping crypto ACLs is when two entries in your crypto map match on the same ACL reference, but in two different ACLs. At first this might sound impossible, but it is quite plausible based on your security policy. For example, you might have an L2L session between two appliances, protecting traffic between 192.168.0.0/24 and 192.168.3.0/24. However, on the remote peer, you have a second crypto map entry that’s a transport connection, protecting syslog traffic from a remote appliance (192.168.0.10) to a local TFTP server (192.168.3.25). In this situation, if the crypto map entry that represented this connection appeared after the L2L connection, it would never match and thus never would be used. Depending on how the TFTP server was set up, the syslog messages from the appliance might or might not be dropped. Reordering the crypto map entries would fix this problem. In Example 23-13, renumber crypto map entry 20 to something like 5.


Tip

Add a note hereThe crypto map entry with the more specific ACL should have a lower number than an overlapping crypto map entry with a less specific ACL.