This work addresses the data-balancing problem of the existing neural network based speaker verification methods, and proposes new method using modular neural network. In this method, each expert network is trained with the balanced number of genuine speaker data and imposter speaker data. In our experiments, we obtained high performance results for the unknown imposter speakers. High performance and the modular nature of the proposed method enables building a large scalable speaker verification system.