In this paper, a neural network based framework for semantic event detection in soccer videos is proposed. The framework provides a robust solution for soccer goal event detection by combining the strength of multimodal analysis and the ability of neural network ensembles to reduce the generalization error. Due to the rareness of the goal events, the bootstrapped sampling method on the training set is utilized to enhance the recall of goal event detection. Then a group of component networks are trained using all the available training data. The precision of the detection is greatly improved via the following two steps. First, a pre-filtering step is employed on the test set to reduce the noisy and inconsistent data, and then an advanced weighting scheme is proposed to intelligently traverse and combine the component network predictions by taking into consideration the prediction performance of each network. A set of experiments are designed to compare the performance of different bootstrapped sampling schemes, to present the strength of the proposed weighting scheme in event detection, and to demonstrate the effectiveness of our framework for soccer goal event detection.