The interpretation of medical evidence is normally presented in terms of a controlled, but diversely expressed specialist vocabulary and natural language phrases. Such informally expressed data require human intervention to ascertain its relevance in any specific case. In order to facilitate machine-based reasoning about the evidence gathered, additional interpretive semantics must be attached to the data; a shift from amerely data-intensive approach to a semantics-rich model of evidence. In this paper, we present a system to formally annotate medical images captured to aid the diagnosis and management of breast cancer, that enables a series of semantics-based operations to be performed. Our approach is grounded upon an imaging ontology specifying the domain knowledge and a Description Logic (DL) taxonomic inferential engine responsible for semantics-based reasoning and image retrieval.