There is a large and rapidly increasing amount of video data on the Internet and in personal or organizational collections. Fast and accurate video search emerges to be an important issue. The need and main technical challenges for video retrieval are similar to those for the content-based image retrieval (CBIR) problem. Lack of meaningful and comprehensive text annotation means that an approach based on content similarity can be promising; and the differences between an often high-level search intention and the low-level features used in content-based search techniques suggest that content-based video retrieval (CBVR) may also suffer from "semantic gap" issues. In this paper, we analyze the problem of CBVR from related work in the literature as well as some current work in our team, focusing on the relationship between CBIR and CBVR, open yet well-defined research issues and practical applications of CBVR.