Face pose estimation forms an important part in a face recognition system. However, fully automated and accurate pose determination still remains an unsolved problem in the research community. In this paper, we propose a novel Elastic Energy Model to automatically estimate face poses. Our method employs statistical energy contributions of a set of feature points, which can avoid over-trusting selected anchor points. It provides a robust solution to the feature localisation inaccuracy problem, which is inevitable in practical applications with cluttered backgrounds. As a general configuration, our model can be easily implemented and extended to other non-rigid objects. Its effectiveness and robustness are revealed in our experiments.