In this paper, we propose a successive method for adaptively estimating background components using Kalman filters, and a novel method for detecting objects using margined sign cross correlation (MSC). MSC is a natural extension of sign cross correlation, in other words, peripheral increment sign correlation. MSC has a margin to deal with observation noise. By applying MSC to our adaptive background model, our proposed system can robustly and accurately perform object detection. We show experimental results using real images to demonstrate the performance of the proposed system.