Foreground segmentation with moving camera is a challenging task due to the presence of parallax effect, registration error, scene variations in out door, and etc. Currently, background modeling techniques either assumes correspondence among pixels in concurrent frames or do not model it explicitly. The contribution by this paper is in two folds. First, we achieve a new background model by introducing correspondence into it. Second, we pose foreground segmentation and correspondence estimation as a labeling problem. Spatial context is enforced in shape of tree structure and global optimal label at each node is computed using dynamic programming. Finally, based on the optimal correspondence, background model is updated. Resultantly, parallax effect and registration error are reduced significantly. Primary experiments proved our algorithm to be robust in performance.