This paper proposes a new method for the extraction of background objects, especially the extraction of randomly moving objects such as curtains and plants. For this purpose we use an Optical Flow method combined with Split and Merge algorithm to avoid the mentioned background effects. The Split and Merge algorithm is based on the variance of regions and the Optical Flow method is implemented conform to the proposition of Horn and Schunck [1]. To further increase the robustness and utility in surveillance circumstances, illumination invariance has to be provided by the algorithm. Hence we combined the mentioned methods with an illumination invariant change detector.Tests have been performed on indoor and outdoor surveillance sequences. Especially in indoor sequences the proposed method provided a very robust tool against illumination changes, reflections, shadows and randomly moving objects like curtains and plants. Nevertheless in all circumstances it detected the moving 'semantic' objects such as persons.