This paper introduces a novel system (architecture and techniques) that aims to secure overlay networks by detecting anomalies in Voice over IP networks. It is particularly designed for the signaling protocol SIP. The proposed system mainly consists of two parts. The first one determines the different features that are extracted from the specification of the SIP protocol. In fact, these features should highly characterize the behavior of the signaling traffic so that the evidence of the intrusion is not lost when only these attributes are considered for the attack detection goal. After the attributes extraction step, a detection algorithm is used to classify new SIP profiles in their appropriate class (either as normal, or as an anomaly). Another feature of this system is its adaptability since a feedback from the detected attacks is possible.