In this paper, we propose a combination of rule-based and neural classifiers to recognize unconstrained handwritten numerals, 0 to 9. During training, the rule-based classifier identifies the candidate set for each character class. The candidate set of a character class i comprises the character classes with which a pattern of i is most likely to be confused. For each candidate set, a neural net is then trained to distinguish patterns within the candidate set, but to reject all patterns that do not belong to the candidate set. During testing, based upon the output of the rule-based classifier, appropriate neural nets are invoked to confirm or reject the decision of the rule-based classifier.