Robustness of parameter estimation relies on discriminating inliers from outliers within the set of correspondences. In this paper, we present a method using tensor voting to eliminate outliers and estimating affine transformation parameters directly from covariance matrix of selected inliers without additional parameter estimation processing. Our approach is based on the representation of the correspondences in a decoupled joint image space and the use of the metric associated with the affine transformation. We enforce the metric property in a joint image space for tensor voting, detect several inlier groups corresponding distinct affine motions and directly estimate affine parameters from each set of inliers. The proposed approach is illustrated by a set of challenging examples.