IPSec has become the defacto standard protocol for secure Internet communications, providing traffic integrity, confidentiality and authentication. Although IPSec supports a rich set of protection modes and operations, its policy configuration remains a complex and error-prone task. The complex semantics of IPSec policies that allow for triggering multiple rule actions with different security modes/operations coordinated between different IPSec gateways in the network increases significantly the potential of policy misconfiguration and thereby insecure transmission. Successful deployment of IPSec requires thorough and automated analysis of the policy configuration consistency for IPSec devices across the entire network.

In this paper, we present a generic model that captures various filtering policy semantics using Boolean expressions. We use this model to derive a canonical representation for IPSec policies using Ordered Binary Decision Diagrams. Based on this representation, we develop a comprehensive framework to classify and identify conflicts that could exist in a single IPSec device (intra-policy conflicts) or between different IPSec devices (inter-policy conflicts) in enterprise networks. Our testing and evaluation study on different network environments demonstrates the effectiveness and efficiency of our approach.