We investigate an alternative way of combining classification and clustering techniques for sparse binary data in order to reduce the amount of training samples required. Initializing EM from the available labels also reduces the algorithms known dependency on the initialization, which is more evident in the case of sparse data. In addition, the two-valued Poisson class-model is proposed in this paper as a sparse variant of the usual Bernoulli assumption. Our method can be seen as a fusion between generalized logistic regression and parametric mixture modeling. Comparative simulation results on subsets of the 20 Newsgroups binary coded text corpora and binary handwritten digit data demonstrate the potential usefulness of the suggested method.