In this paper we introduce a technique called polishing to reduce variations in cached layer-encoded video by identifying those segments which can be removed without diminishing the perceptual quality. In fact, polishing even allows to increase the perceptual quality of a layer-encoded video. This can be used for perceptual quality improved playout from a cache towards a client as well as for perceptual quality aware cache replacement decisions. We devise an optimal scheme to polish layer-encoded videos but show that a simple heuristic achieves similar performance in simulations. By means of simulations we furthermore show how polishing can improve the quality of the cached content.