JMS is an API specification that defines a standard way for Java applications to access messaging services. All JMS products promise good performance and to properly support the QoS attributes specified in the standard, making it hard to choose between them. Customers who want to determine which JMS product best meets their requirements need a simple, effective and fair methodology for evaluating and comparing competing implementations. This paper presents an empirical methodology for evaluating the QoS implementation of a JMS product. We present a number of test scenarios and define metrics for measuring performance and message persistence. We then illustrate this methodology by using it to evaluate two leading JMS products. Our evaluation results show differences between these products in terms of their overall performance and the impact of various QoS attributes. The case study demonstrates that our empirical methodology is an effective and practical way to test the performance of JMS and other messaging systems.