Recovering the 3D structure and motion of nonrigid object from a monocular image sequence is an important and difficult task in computer vision. Many previous methods on this problem utilize the extension technique of SVD factorization based on rank constraint to the tracking matrix. In this paper, we propose a constrained power factorization (CPF) algorithm that combines the orthonormal constraint and the replicated block structure of the motion matrix directly into the iterations. The proposed algorithm overcomes the limitations of previous SVD based methods. It is easy to implement and can even cope with the tracking matrix with missing data. Based on the solutions of the CPF, a novel sequential factorization technique is proposed to recover the shape and motion of new frames in realtime. Extensive experiments on synthetic data and real sequences validate the effectiveness of the algorithm and show noticeable improvements over the previous methods.