IPSec VPN Fundamentals

With the explosive growth and popularity of the Internet, more and more enterprises are looking towards building their network infrastructure across the Internet without having to spend a lot on private leased lines. However, with the increasing usage of the Internet to build networks, more and more "evil ways" of breaking into the network to gather sensitive information are also evolving. Therefore, security on the Internet has been a main concern for each enterprise. Enter IP Security or IPSec. IPSec provides the necessary infrastructure to extend an enterprise's private network across the Internet to reach out to customers and business partners, in other words, to build what is called a "virtual private network (VPN)".

IPSec is a set of protocols and standards developed by the Internet Engineering Task Force (IETF) to support secure communication at the IP layer. Used widely to implement VPNs either as a full-fledged security gateway sitting between the corporate network and the router connected to the Internet, or as part of the router itself, IPSec and an associated suite of protocols offer complete cryptographic security.

• Remote-Access VPNs: These let individual users, such as telecommuters, connect to a corporate network. The user's laptop usually contains a VPN client that creates a secure tunnel to the security gateway at the corporate headquarters. Another flavor of this application is offered via creating an L2TP/PPTP session that is protected through IPSec.
  
  
• Intranet VPNs: This type connects branch offices to the corporate headquarters, thus creating a transparent Intranet.
  
  
• Extranet VPNs: These let companies connect with their business partners (for example, suppliers, customers, and joint ventures).

Figure 1:  IPsec VPN applications

• Encryption/Decryption: This forms the basis of confidentiality. Encryption is the process of taking a plaintext M and converting it to a ciphertext C of equal length. The ciphertext C is transmitted on the wire that protects the message M from any eavesdropper who overhears the communication. On the receiving end, decryption is performed that converts the ciphertext C back to plaintext M.
  
  There are two kinds of cryptographic transforms available that differ on the usage of the key used to encrypt/decrypt:
  
  
  1. Symmetric-Key Cryptography: Here only one shared key K is known to both the peers, but it is not known to any third party. Encryption is the function of two variables (M, K), and decryption is the function of (C, K). Many symmetric-key algorithms exist, but the popular ones are DES, 3DES, and AES. Most of these algorithms process one block of data at a time and transform it into a ciphertext block. These block-transform functions can be operated in different modes to add extra security. One such popular mode is cyclic block chaining (CBC), where each processed ciphertext block is added (in binary arithmetic) to the next plaintext block before it is input to the block transform function. An initialization vector (IV), random data of equal length, is used as the initial data for this mode. describes these algorithms and various modes in detail.
     
     
  2. Asymmetric-Key Cryptography (also called public-key cryptography): Here a pair of keys is assigned per peer: a public key K₁ that can be published and a private key K₂ known only to the owner of the key pair. A text M encrypted by K₁ can be decrypted only by K₂ and vice versa, but not by any other means. The most popular public-key algorithm is RSA. Refer to Mohapatra for an introduction to public-key cryptography.
  
  Note that symmetric key algorithms are computationally much faster than public key algorithms. For efficiency, symmetric-key algorithms like DES or 3DES are used for preserving confidentiality. Public-key algorithms are used in a "hybrid" or "key-management" mode to achieve authentication and non-repudiation.
  
  
• Message Digest or Integrity Checksum Value (ICV): This forms the basis of data integrity. A message digest is a small "fingerprint" of a long message M that is obtained by applying a one-way hash function, H - a function that is easy to compute but for which it is computationally not feasible to find two messages M and M' such that H(M) = H(M'). If the message digest is included in the IPSec packet, the receiving end can compute the message digest on the received message and compare it with the included digest to find out if the message has been compromised. Popular hash functions are MD5 and SHA1.
  
  
• Digital Signature: This forms the basis of non-repudiation. Typically a public-key algorithm, like RSA, is combined with a one-way hash function, like MD5 or SHA1. For a message M, a message digest H(M) is computed and encrypted (signed) by the sender's private key. This is called the digital signature for message M that is sent across the wire. Because only the owner knows the private key, the sender cannot repudiate the fact that he sent the message M.
  
  
• Certificate: This forms the basis of authentication. Certificate is some data in a well-defined format that binds a person's identity to his/her public key and is signed by a "trusted authority." This certificate is "self-authenticating" because both the parties trust the "trusted authority" and have its public key available.

• Transport Mode: This mode is applicable only for host-to-host security. Here protection extends to the payload of IP data. The IP addresses of the hosts must be public IP addresses.
  
  
• Tunnel Mode: This mode is used to provide data security between two networks. It provides protection for the entire IP packet and is sent by adding an outer IP header corresponding to the two tunnel end-points. The unprotected packets generated by hosts travel through the protected "tunnel" created by the gateways on both ends. The outer IP header in Figure 2 corresponds to these gateways. Both intranet and extranet VPNs are enabled through this mode. Since tunnel mode hides the original IP header, it facilitates security of the networks with private IP address space.

Figure 2:  IPsec modes of operation—tunnel and transport

Figure 3:  Security association structure

An SA indicates all the security services to the traffic carried by it (pointed to by the authentication transform, encryption transform, and the replay protection—a replay attack is a denial of service attack where an eavesdropper saves already traversed packets and sends them at a later point of time). It also has a lifetime associated with it, by the end of which the SA state becomes expired. An expired SA can no longer be used to protect traffic; a new SA for the same tuple has to be created. Because SAs are simplex, for bi-directional communication between two IPSec systems, there must be two SAs defined, one for each direction. SAs can be created either manually or through an automatic key-exchange protocol, IKE.

Security Parameter Index (SPI) is a 32-bit value used to identify an SA. The SPI is carried in the IPSec header of the protected packet and has local significance only. SAD contains all the SAs that are either created manually or automatically through negotiation, indexed by the tuple: .

1. Discard the packet
   
   
2. Process by the IPSec module, in which case the SPD entry points to an SA
   
   
3. Pass through (pass the packet to the IP stack for normal forwarding).

Figure 4:  IPsec architecture

The "Policy Manager" module is the interface between the user adding a security policy and the SPD. The "IKE Daemon" module does the automatic SA negotiation between two IPSec peers. The "Certificate Manager" verifies and enrolls certificates for authentication purposes.

In short, a typical packet flow inside this architecture proceeds as follows:

• A packet is received through the receive queue and passed to the IPSec packet processing module.
  
  
• The IPSec packet processing module extracts the "selector" from the packet and looks up the SPD for a policy. If the policy is "discard", the packet is discarded. If the policy is "pass through", the forwarding engine forwards the packet normally.
  
  
• If the policy is "IPSec", the SPD entry should point to an SA in SAD. The module then fetches the corresponding SAD entry and checks for validity. If the SA state is expired, the module indicates the IKE Daemon for another SA negotiation.
  
  
• All the transforms depicted in the SA are performed on the packet with the help of the "cryptography" module.
  
  
• The transformed packet is sent to the "transmit queue" for transmission.

Figure 5:  IPsec (a) AH format and (b) ESP format

Figure 6 shows how AH and ESP are used in tunnel mode to protect an IP packet.

Figure 6:  AH and ESP in tunnel mode

IKE uses the Internet Security Association and Key Management Protocol (ISAKMP) as the framework and incorporates Oakley and SKEME as the key-exchange protocol. All the parameter values to be negotiated are defined in the IPSec domain of interpretation (DOI) document. The IKE Daemon runs as an UDP application with UDP port number 500. The protocol operates in two phases:

1. Phase I (Authentication Phase)
   When two peers over the Internet wish to communicate, it is assumed that no secure channel exists. Therefore, the objective of "phase I" is to establish a secure channel, authenticate the negotiating parties, and generate shared keys to protect IKE protocol messages. Two modes are available for use in "phase I": main mode and aggressive mode. They both accomplish the same thing"establishing an IKE SA that will be used to protect the negotiating messages of phase II. Aggressive mode is a little faster, but it does not provide identity protection for the negotiating parties. Also, for some authentication methods used (like the public key encryption), aggressive mode is known to be vulnerable to denial-of-service (DoS) attacks.
   
   
   • Main Mode
     Figure 7 shows the basic main-mode message exchanges. In main mode, the negotiating parties use six messages: the first two messages to negotiate the security policy that will be used to protect the phase II messages. The next two messages perform a Diffie-Hellman key exchange and pass nonces (random numbers sent for signing) to each other. The last two messages are used to authenticate the peers.
     
     There are four methods specified in the IKE standard to authenticate the negotiating peers. They are:
     
     
     • Pre-Shared Keys (PSK): As the name suggests, a shared secret is distributed out-of-band to the peers. The peers use this information and nonce parameters to create a hash that is used to authenticate messages.
       
       
     • Digital Signatures (RSA or DSS): Here, certificates of the peers are exchanged in the last two messages and hashes are calculated over these certificates to authenticate each other. There are many protocols and standards available now that ease the process of certificate enrollment, certificate request, and certificate status checking. Some popular ones are RSA Labs' PKCS #7, PKCS #10, Cisco's Simple Certificate Enrolment Protocol (SCEP), and Online Certificate Status Protocol (OCSP).
       
       
     • RSA Public Key Encryption: With this, each negotiating party has a public-private key-pair that they use to encrypt/decrypt messages starting from message #3. Nonces exchanged are secured through this and the parties calculate hashes over these nonces to authenticate each other.
       
       
     • Revised RSA Public Key Encryption: This method is similar to the prior method, but reduces the number of public key operations from four to two and, instead, incorporates two symmetric-key operations.
     
     The authentication data payload shown in Figure 7 is different for each of the previously described modes. For example, for PSK mode the previously created hash is used. For an RSA digital-signature mode, the certificate and the digital signature are sent as authentication data.
     
     
     

     
     Figure 7:  IKE main mode message exchange

     
     
     
   • Aggressive Mode
     In aggressive mode, you need only three messages to establish the IKE SA. However, the identities of the parties involved are revealed. Figure 8 illustrates aggressive mode exchanges.
     
     
     

     
     Figure 8:  IKE aggressive mode message exchange

     
     
     
2. Phase II (Key Exchange)
   Phase II, also called as the "Quick Mode," is used to establish the IPSec SA and to generate new keying material. A full Diffie-Hellman key exchange may be done to provide perfect forward secrecy (PFS), otherwise the keys are derived from the phase I keying material. The basic quick-mode message exchanges are illustrated in Figure 9.

Figure 9:  IKE quick mode message exchange

For scenarios where there is a NAT device in-between (NAT-in-the-middle problem), the IPSec standards group at IETF has proposed a new protocol called "NAT traversal (NAT-T)". Here, the IPSec packet is encapsulated within a UDP packet using the IKE UDP port 500. Negotiation of NAT-T support of the peers as well as detection of NAT presence in the path is done during IKE phase-I.

• Processors with Basic Algorithm Support
  These processors perform basic symmetric-key operations such as DES, 3DES, and AES, and hash operations such as SHA1, MD5, and others.
  
  
• Packet Processors
  These take in a packet along with an SA and do the complete packet processing (for example, the addition of the AH or ESP header, as required) in addition to supporting the prior functionality.
  
  
• Inline Security Coprocessors
  These processors can handle SA lookup and packet handling, as well as SPD verification.

You can choose the right kind of crypto accelerators with the right price/performance as needed for your system. For example, if you are building a 1-Gbps security gateway and want to offload all of the security processing from your main network processor, you should use an inline processor.

The protocol is as strong as the underlying algorithms it employs, so proper algorithm selection is important to network security. Furthermore, for smooth operation, IPSec requires a public-key infrastructure (PKI). Such infrastructures are still in their infancy, and wide-scale key infrastructures are just emerging on the Internet. All-in-all, the management of the security and access policies is a complicated field and a thorough knowledge of several concepts is necessary to properly administer these policies. Finally, as IPSec and IKE are evolving standards, interoperability between different vendors is still a problem.

Still, the overall impact of IPSec will be tremendous in the years to come. IPSec makes it possible to securely connect offices, users, and partners to the network and safely transmit information as a very cost-effective solution. IPSec does this in a manner completely transparent to the end users. As the networks migrate to Ipv6, IPSec will become an integral component of those networks as well.