Semantic Web Services are generally considered as the evolution of conventional Web Services. Semantic information included in the service descriptions enables the development of advanced matchmaking schemes, capable of assigning degrees of match to the discovered services. In this paper, we address issues related to the evaluation of the retrieval effectiveness of semantic matchmaking systems. Our main position is that conventional evaluation schemes do not fully capture the added value of service semantics nor do they take into account the assigned degrees of match, supported by the majority of discovery engines. Through some preliminary experiments, we show that a generalization of the evaluation process based on fuzzy set theory techniques can lead to more accurate and meaningful evaluation results.