A vision-based detecting and tracking vehicle in video streams is an important research in computer vision, and it plays an important role in ITS. The aim of motion detection is to get the changed region from the background image in video sequences, and it is also very important for targets classification and tracking motion objects. In this paper, a self-adaptive background subtraction method for vehicle segmentation was proposed. To indicate motion mask regions in a scene, instead of determining the threshold value manually, we use an adaptive thresholding method to automatically choose the threshold value. In order to accurately locate vehicle, we combined the projection of the difference image and the projection of the edge map from coarse to refine accurately locate vehicles. This proposed method could locate vehicle well. We formed an association graph between the regions from the previous frame and the regions from the current frame, so we modeled the vehicle tracking problem as a problem of finding maximal weight association graph. Very promising experimental results are provided using real-time video sequences, Experimental results demonstrate the validity of the approach in term of robustness, accuracy and time responses.