RotoTexture: Automated Tools for Texturing Raw Video
|
Abstract—We propose a video editing system that allows a user to apply a time-coherent texture to a surface depicted in the raw video from a single uncalibrated camera, including the surface texture mapping of a texture image and the surface texture synthesis from a texture swatch. Our system avoids the construction of a 3D shape model and instead uses the recovered normal field to deform the texture so that it plausibly adheres to the undulations of the depicted surface. The texture mapping method uses the nonlinear least-squares optimization of a spring model to control the behavior of the texture image as it is deformed to match the evolving normal field through the video. The texture synthesis method uses a coarse optical flow to advect clusters of pixels corresponding to patches of similarly oriented surface points. These clusters are organized into a minimum advection tree to account for the dynamic visibility of clusters. We take a rather crude approach to normal recovering and optical flow estimation, yet the results are robust and plausible for nearly diffuse surfaces such as faces and t-shirts.
[1] 1580 H. Fang and J.C. Hart, “Textureshop: Texture Synthesis as a Photograph Editing Tool,” ACM Trans. Graphics, vol. 23, no. 3, Proc. SIGGRAPH, pp. 354-359, 2004.
[2] Y. Liu, W.-C. Lin, and J. Hays, “Near-Regular Texture Analysis and Manipulation,” ACM Trans. Graphics, vol. 23, no. 3, Proc. SIGGRAPH, pp. 368-376, 2004.
[3] M. Pollefeys, L.V. Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, and R. Koch, “Visual Modeling with a Hand-Held Camera,” Int'l J. Computer Vision, vol. 59, no. 3, pp. 207-232, 2004.
[4] G. Turk, “Texture Synthesis on Surfaces,” Proc. SIGGRAPH, pp.347-354, 2001.
[5] L.-Y. Wei and M. Levoy, “Texture Synthesis over Arbitrary Manifold Surfaces,” Proc. SIGGRAPH, pp. 347-354, 2001.
[6] B.K. Horn, “Height and Gradient from Shading,” Int'l J. Computer Vision, vol. 5, no. 1, pp. 37-75, 1990.
[7] R. Zhang, P.-S. Tsai, J.E. Cryer, and M. Shah, “Shape from Shading: A Survey,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 8, pp. 690-706, 1999.
[8] L. Zhang, G. Dugas-Phocion, J.-S. Samson, and S. Seitz, “Single View Modeling of Free-Form Scenes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, p. 990, 2001.
[9] L. Zhang, B. Curless, and S.M. Seitz, “Spacetime Stereo: Shape Recovery for Dynamic Scenes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 367-374, 2003.
[10] L. Zhang, N. Snavely, B. Curless, and S.M. Seitz, “Spacetime Faces: High Resolution Capture for Modeling and Animation,” Proc. SIGGRAPH, pp. 548-558, 2004.
[11] L. Torresani, D. Yang, G. Alexander, and C. Bregler, “Tracking and Modeling Non-Rigid Objects with Rank Constraints,” Proc. IEEE Computer Vision and Pattern Recognition Conf., p. 493, 2001.
[12] M. Brand, “Morphable 3D Models from Video,” Proc. IEEE Computer Vision and Pattern Recognition Conf., p. 456, 2001.
[13] C. Bregler, A. Hertzmann, and H. Biermann, “Recovering Non-Rigid 3D Shape from Image Streams,” Proc. IEEE Computer Vision and Pattern Recognition Conf., p. 2690, 2000.
[14] M.J. Black and P. Anandan, “The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields,” Computer Vision and Image Understanding (CVIU), vol. 63, no. 1, pp. 75-104, Jan. 1996.
[15] D. DeCarlo and D. Metaxes, “Optical Flow Constraints on Deformable Models with Applications to Face Tracking,” Int'l J.Computer Vision, vol. 38, no. 2, pp. 99-127, July 2000.
[16] D. DeCarlo and D. Metaxes, “Adjusting Shape Parameters Using Model-Based Optical Flow Residuals,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 814-823, June 2002.
[17] S.-Y. Lee, K.-Y. Chwa, S.Y. Shin, and G. Wolberg, “Image Metamorphosis Using Snakes and Free-Form Deformations,” Proc. SIGGRAPH, pp. 439-448, 1995.
[18] V. Kwatra, A. Schoedl, I. Essa, G. Turk, and A. Bobick, “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” ACM Trans. Graphics, vol. 22, no. 3, Proc. SIGGRAPH, pp. 277-286, 2003.
[19] A.A. Efros and W.T. Freeman, “Image Quilting for Texture Synthesis and Transfer,” Proc. SIGGRAPH, pp. 341-346, 2001.
[20] Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum, “Lazy Snapping,” ACM Trans. Graphics, vol. 23, no. 3, Proc. SIGGRAPH, pp. 303-308, 2004.
[21] Y.-Y. Chuang, A. Agarwala, B. Curless, D.H. Salesin, and R. Szeliski, “Video Matting of Complex Scenes,” ACM Trans. Graphics, vol. 21, no. 3, Proc. SIGGRAPH, pp. 243-248, 2002.
[22] Y. Li, J. Sun, and H.-Y. Shum, “Video Object Cut and Paste,” ACM Trans. Graphics, vol. 24, no. 3, Proc. SIGGRAPH, pp. 595-600, 2005.
Index Terms:
Video editing, shape from shading, texture synthesis.
Citation:
Hui Fang, John C. Hart, "RotoTexture: Automated Tools for Texturing Raw Video," IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 6, pp. 1580-1589, Nov./Dec. 2006, doi:10.1109/TVCG.2006.102
PURCHASE ARTICLE: $19
IEEE Xplore Subscribers
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb