The selection of relevant and non-redundant features or variables from a larger set is an ubiquitous problem in many disciplines. Numerous automated methods have been introduced, however, the important issue of selection stability is still largely uncovered. It can be observed, that small changes in the data can lead to dramatic changes in the selection. This compromises both statistical reliability and recognition rates as well as knowledge extraction. In our work, we pursue an approach employing data sampling techniques, e.g., leave-one-out method, and generate statistics of selection to determine a stability factor and identify stable features. In this paper, we introduce improved selection techniques from first and second order statistics and demonstrate their effectiveness for three benchmark problems of increasing complexity.