In hospital practice, several diagnostic hysteroscopy videos are produced daily. These videos are continuous (non-interrupted) video sequences, usually recorded in full. However, only a few segments of the recorded videos are relevant from the diagnosis/prognosis point of view, and need to be evaluated and referenced later. This paper proposes a new technique to identify clinically relevant segments in diagnostic hysteroscopy videos, producing a rich and compact video summary which supports fast video browsing. Also, our approach facilitates the selection of representative key-frames for reporting the video contents in the patient records. The proposed approach requires two stages. Initially, statistical techniques are used for selecting relevant video segments. Then, a post-processing stage merges adjacent video segments that are similar, reducing temporal video over-segmentation. Our preliminary experimental results indicate that our method produces compact video summaries containing a selection of critically relevant video segments. These experimental results were validated by specialists.