Regeneration of biometric templates from match scores has security and privacy implications related to any biometric based authentication system. In this paper, we propose a novel non-iterative scheme to reconstruct face templates from match scores. We use an affine transformation of the images to approximate the behavior of the given face recognition system based on an independent set of face templates termed as ?break-in? set. Selected templates from the ?break-in? set are matched only once with the enrolled template of the target account and match scores are recorded. These scores are then embedded in the approximating affine space along with break-in set templates to compute the co-ordinates of the target template. The inverse transformation is used to reconstruct the original target template. We present the reconstruction of templates for three different face recognition algorithms: Bayesian intrapersonal/extrapersonal classifier, Elastic Bunch Graph Matching (EBGM) Algorithm and baseline algorithm used in Face Recognition Grand Challenge (2005). We also report the initial result to break an user?s account using the reconstructed template for Bayesian algorithm. Bayesian algorithm was set to operate at 0.1% False Acceptance Rate and 98% True Acceptance Rate with 100 enrollments. We observed that with proposed scheme, at most 600 attempts is required to achieve a 0.99 probability of break-in the Bayesian face recognition algorithm.