A new dimensionality reduction method is proposed which is used to extract salient features for pattern classification problems. The method is used jointly with a classifier of smooth response. It performs a PCA-like operation to a set of vectors defined using directional derivatives of the classifier's response in the original feature space of the training patterns. The method is implemented using a smooth variation of the K-nearest neighbor classifier. The efficiency of the method is evaluated in three benchmark classification tasks. Efficient dimensionality reduction is observed without adverse effects on classification ability.