Khaled El Emam Fraunhofer Institute for Experimental Software Engineering The development of binary classifiers to identify highly error-prone or high maintenance cost components is increasing in the software engineering quality modeling literature and in practice. One approach for evaluating these classifiers is to determine their ability to predict the classes of unseen cases, i.e., predictive validity. A chi-square statistical test has been frequently used to evaluate predictive validity. In this paper we illustrate that this test has a number of disadvantages. The disadvantages include a difficulty in using the results of the test to determine whether a classifier is a good predictor, demonstrated through a number of examples, and a rather conservative Type I error rate, demonstrated through a Monte Carlo simulation. We present an alternative test that has been used in the social sciences for evaluating agreement with a "gold standard". The use of this alternative test is illustrated in practice by developing a classification model to predict maintenance effort for an object oriented system, and evaluating its predictive validity on data from a second object-oriented system in the same environment.