Signs produced by gestures (such as in American Sign Language) can have a basic meaning coupled with additional meanings that are layered over the basic meaning of the sign. These layered meanings are conveyed by temporal and spatial modification of the basic form of the gesture movement. The work reported in this paper seeks to recognize temporal and spatial modifiers of hand movement and integrates them with the recognition of the basic meaning of the sign. To this end, a Bayesian network framework is explored with a simulated vocabulary of 4 basic signs which give rise to 14 different combinations of basic meanings and layered meanings. Recognition accuracies of upto 88.2% were obtained.