Based on the ordered Hadamard transform and probability model, we propose a fast partial distortion elimination algorithm. In this paper, we intuitively derive DC and AC constraints to model local block complexities using ordered Hadamard transform from pixel based gradient method. For lossless motion estimation, we obtain the optimized search order in the matching error calculation by descending order of local constraints using sum of these constraints. And for lossy motion estimation, using probability model which is approximated by the cumulative distribution function of constraints, we also propose the accurate total matching error prediction algorithm in the middle of calculation without all matching error calculations. Compared with the conventional full search algorithm, the proposed algorithm reduces the motion estimation complexity up to 92.4% for lossy motion estimation and 89.4% for lossless motion estimation on average.