Soft Color Segmentation and Its Applications
|
We propose an automatic approach to soft color segmentation, which produces soft color segments with appropriate amount of overlapping and transparency essential to synthesizing natural images for a wide range of image-based applications. While many state-of-the-art and complex techniques are excellent at partitioning an input image to facilitate deriving a semantic description of the scene, to achieve seamless image synthesis, we advocate to a segmentation approach designed to maintain spatial and color coherence among soft segments while preserving discontinuities, by assigning to each pixel a set of soft labels corresponding to their respective color distributions. We optimize a global objective function which simultaneously exploits the reliability given by global color statistics and flexibility of local image compositing, leading to an image model where the global color statistics of an image is represented by a Gaussian Mixture Model (GMM), while the color of a pixel is explained by a local color mixture model where the weights are defined by the soft labels to the elements of the converged GMM. Transparency is naturally introduced in our probabilistic framework which infers an optimal mixture of colors at an image pixel.To adequately consider global and local information in the same framework, an alternating optimization scheme is proposed to iteratively solve for the global and local model parameters. Our method is fully automatic, and is shown to converge to a good optimal solution. We perform extensive evaluation and comparison, and demonstrate that our method achieves good image synthesis results for image-based applications such as image matting, color transfer, image deblurring, and image colorization.
[1] 1520 S. Belongie, C. Carson, H. Greenspan, and J. Malik, “Color- and Texture-Based Image Segmentation Using the Expectation-Maximization Algorithm and Its Application to Content-Based Image Retrieval,” Proc. Int'l Conf. Computer Vision, pp. 675-682, 1998.
[2] A. Berman, P. Vlahos, and A. Dadourian, “Comprehensive Method for Removing from an Image the Background Surrounding a Selected Object,” US Patent 6,134,345, 2000.
[3] J.C. Bezdek and R.J. Hathaway, “Convergence of Alternating Optimization,” Neural, Parallel Scientific Computing, vol. 11, no. 4, pp. 351-368, 2003.
[4] M.J. Black and A. Rangarajan, “The Outlier Process: Unifying Line Processes and Robust Statistics,” Proc. Computer Vision and Pattern Recognition, pp. 15-22, 1994.
[5] Y.-Y. Chuang, B. Curless, D.H. Salesin, and R. Szeliski, “A Bayesian Approach to Digital Matting,” Proc. Computer Vision and Pattern Recognition, vol. II, pp. 264-271, 2001.
[6] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach Toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[7] I. Csiszar and G. Tusnady, “Information Geometry and Alternating Minimization Procedures,” Statistics and Decisions, Supplement Issue, pp. 205-237, 1984.
[8] A.P. Dempster, N.M. Laird, and D.B. Rubin, “Maximal Likelihood from Incomplete Data via the EM Algorithm,” Royal Statistics, vol. B, no. 39, pp. 1-38, 1977.
[9] Y. Deng and B.S. Manjunath, “Unsupervised Segmentation of Color-Texture Regions in Images and Video,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 8, pp. 800-810, Aug. 2001.
[10] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification. John Wiley & Sons, 2001.
[11] E. Eisemann and F. Durand, “Flash Photography Enhancement via Intrinsic Relighting,” ACM Trans. Graphics, vol. 23, no. 3, pp.673-678, 2004.
[12] M. Galun, E. Sharon, R. Basri, and A. Brandt, “Texture Segmentation by Multiscale Aggregation of Filter Responses and Shape Elements,” Proc. Int'l Conf. Computer Vision, pp. 716-723, 2003.
[13] S. Gordon, H. Greenspan, and J. Goldberger, “Applying the Information Bottleneck Principle to Unsupervised Clustering of Discrete and Continuous Image Representations,” Proc. Int'l Conf. Computer Vision, pp. 370-377, 2003.
[14] R.M. Gray, Entropy and Information Theory. Springer, Nov. 2000.
[15] P. Grunwarld, Minimum Description Length and Maximum Probability. Kluwer Academic Publishers, 2002.
[16] J. Jia, J. Sun, C.K. Tang, and H.Y. Shum, “Bayesian Correction of Image Intensity with Spatial Consideration,” Proc. European Conf. Computer Vision, vol. III, pp. 342-354, 2004.
[17] V. Kolmogorov and R. Zabih, “Multi-Camera Scene Reconstruction via Graph Cuts,” Proc. European Conf. Computer Vision, p. III: 82 ff., 2002.
[18] M. Mirmehdi and M. Petrou, “Segmentation of Color Textures,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 2, pp. 142-159, Feb. 2000.
[19] R. Nock and F. Nielsen, “Statistical Region Merging,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 11, pp. 1452-1458, Nov. 2004.
[20] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
[21] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, and K. Toyama, “Digital Photography with Flash and No-Flash Image Pairs,” ACM Trans. Graphics, vol. 23, no. 3, pp. 664-672, 2004.
[22] M. Pietikainen, A. Rosenfeld, and I. Walter, “Split-and-Link Algorithms for Image Segmentation,” Pattern Recognition, vol. 15, no. 4, pp. 287-298, 1982.
[23] T. Porter and T. Duff, “Compositing Digital Images,” Proc. ACM Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '84), pp. 253-259, 1984.
[24] E. Reinhard, M. Ashikhmin, B. Gooch, and P. Shirley, “Color Transfer between Images,” IEEE Computer Graphics and Applications, vol. 21, pp. 34-41, 2001.
[25] C. Rother, V. Kolmogorov, and A. Blake, “Grabcut—Interactive Foreground Extraction Using Iterated Graph Cuts,” Proc. ACM Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '04), 2004.
[26] M.A. Ruzon and C. Tomasi, “Alpha Estimation in Natural Images,” Proc. Computer Vision and Pattern Recognition, pp. 18-25, 2000.
[27] E. Sharon, A. Brandt, and R. Basri, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements,” Proc. Computer Vision and Pattern Recognition, pp. I: 469-476, 2001.
[28] J. Shi and J. Malik, “Normalized Cuts and Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, Aug. 2000.
[29] A.R. Smith and J.F. Blinn, “Blue Screen Matting,” Proc. ACM Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '96), pp. 259-268, 1996.
[30] J. Sun, J. Jia, C.-K. Tang, and H.-Y. Shum, “Poisson Matting,” Proc. ACM Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '04), pp. 315-321, 2004.
[31] J. Sun, N.N. Zheng, and H.Y. Shum, “Stereo Matching Using Belief Propagation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, July 2003.
[32] R. Szeliski and P. Golland, “Stereo Matching with Transparency and Matting,” Int'l J. Computer Vision, vol. 32, no. 1, pp. 45-61, Aug. 1999.
[33] Y.W. Tai, J. Jia, and C.K. Tang, “Local Color Transfer via Probabilistic Segmentation by Expectation-Maximization,” Proc. Computer Vision and Pattern Recognition, pp. I: 747-754, 2005.
[34] K.L. Tang, C.K. Tang, and T.T. Wong, “Dense Photometric Stereo Using Tensorial Belief Propagation,” Proc. Computer Vision and Pattern Recognition, vol. 1, pp. 132-139, June 2005.
[35] Z. Tu and S.C. Zhu, “Image Segmentation by Data-Driven Markov Chain Monte Carlo,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 657-673, May 2002.
[36] J.K. Udupa, P.K. Saha, and R.A. Lotufo, “Relative Fuzzy Connectedness and Object Definition: Theory, Algorithms, and Applications in Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 11, pp. 1485-1500, Nov. 2002.
[37] L. Vincent and P. Soille, “Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 6, pp.583-598, June 1991.
[38] T. Welsh, M. Ashikhmin, and K. Mueller, “Transferring Color to Greyscale Images,” Proc. Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '02), pp. 277-280, 2002.
[39] A.P. Witkin, “Scale-Space Filtering,” Proc. Int'l Joint Conf. Artificial Intelligence, pp. 1019-1022, 1983.
[40] T.P. Wu, K.L. Tang, C.K. Tang, and T.T. Wong, “Dense Photometric Stereo: A Markov Random Fields Approach,” IEEE Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1830-1846, Nov. 2006.
Index Terms:
Color image segmentation, Image synthesis
Citation:
Yu-Wing Tai, Jiaya Jia, Chi-Keung Tang, "Soft Color Segmentation and Its Applications," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 9, pp. 1520-1537, June 2007, doi:10.1109/TPAMI.2007.1168
PURCHASE ARTICLE: $19
IEEE Xplore Subscribers
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb