We propose an energy-based joint motion and disparity estimation algorithm with an anisotropic diffusion operator to yield correct and dense displacement vectors. The model estimates the left and right motions simultaneously in order to increase accuracy. We use the Euler-Lagrange equation with variational methods and solve the equation with the finite difference method (FDM). Then, the method computes the initial disparity in the current frame with joint estimation constraint, and regularizes this disparity by using our energy model. Experimental results show that the proposed algorithm provides accurate motion-disparity maps, and preserve the discontinuities of the object boundaries well.