A new method to recognize continuous sign language based on Hidden Markov Model (HMM) is proposed in this paper. According to the dependence of linguistic context, connections between elementary subwords are classified as strong connection and weak connection. The recognition of strong connection is accomplished with the aid of subword trees, which describe the connection of subwords in each sign language word; In weak connection, the main problem is how to extract the best matched subwords and find their end-points with little help of context information. The proposed method improves the summing process of viterbi decoding algorithm which is constrained in every individual model and compares the end score at each frame to find the ending frame of a subword. Experimental results show an accuracy of 70% for continuous sign sentences that comprise no more than 4 subwords.