Relevance feedback has been shown as an important tool to boost the retrieval performance in content-based image retrieval. In the past decade, various algorithms have been proposed to formulate relevance feedback in contentbased image retrieval. Traditional relevance feedback techniques mainly carry out the learning tasks by focusing lowlevel visual features of image content with little consideration on log information of user feedback. However, from a long-term learning perspective, the user feedback log is one of the most important resources to bridge the semantic gap problem in image retrieval. In this paper we propose a novel technique to integrate the log information of user feedback into relevance feedback for image retrieval. Our algorithm?s construction is based on a coupled support vector machine which learns consistently with the two types of information: the low-level image content and the user feedback log. We present a mathematical formulation of the problem and develop a practical algorithm to solve the problem effectively. Experimental results show that our proposed scheme is effective and promising.