Vision and Inertial Sensor Cooperation Using Gravity as a Vertical Reference
|
Abstract—This paper explores the combination of inertial sensor data with vision. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovery of 3D structure from images, increasing the capabilities of autonomous robots and enlarging the application potential of vision systems. In biological systems, the information provided by the vestibular system is fused at a very early processing stage with vision, playing a key role on the execution of visual movements such as gaze holding and tracking, and the visual cues aid the spatial orientation and body equilibrium. In this paper, we set a framework for using inertial sensor data in vision systems, and describe some results obtained. The unit sphere projection camera model is used, providing a simple model for inertial data integration. Using the vertical reference provided by the inertial sensors, the image horizon line can be determined. Using just one vanishing point and the vertical, we can recover the camera's focal distance and provide an external bearing for the system's navigation frame of reference. Knowing the geometry of a stereo rig and its pose from the inertial sensors, the collineation of level planes can be recovered, providing enough restrictions to segment and reconstruct vertical features and leveled planar patches.
[1] 1597 H. Carpenter, Movements of the Eyes. second ed. London Pion Limited, 1988.
[2] A. Berthoz, The Brain's Sense of Movement. Harvard Univ. Press, 2000.
[3] K.K. Gillingham and F.H. Previc, Spatial Orientation in Flight. second ed. chapter 11, Williams and Wilkins, 1996.
[4] R.O. Eason and R.C. Gonzalez, Least-Squares Fusion of Multisensory Data Data Fusion in Robotics and Machine Intelligence, M.A. Abidi and R.C. Gonzalez, eds., chapter 9, Academic Press, 1992.
[5] R.P.G. Collinson, Introduction to Avionics. Chapman&Hall, 1996.
[6] G.R. Pitman, Inertial Guidance. John Wiley&Sons, 1962.
[7] J.J. Allen, R.D. Kinney, J. Sarsfield, M.R. Daily, J.R. Ellis, J.H. Smith, S. Montague, R.T. Howe, B.E. Boser, R. Horowitz, A.P. Pisano, M.A. Lemkin, W.A. Clark, and T. Juneau, Integrated Micro-Electro-Mechanical Sensor Development for Inertial Applications Proc. Position Location and Navigation Symposium, Apr. 1998.
[8] A.C. Luther, Video Camera Technology. Artech House Publishers, 1998.
[9] T. Viéville and O.D. Faugeras, Computation of Inertial Information on a Robot Proc. Fifth Int'l Symp. Robotics Research, pp. 57-65, H. Miura and S. Arimoto, eds., MIT Press, 1989.
[10] T. Viéville and O.D. Faugeras, Cooperation of the Inertial and Visual Systems Traditional and NonTraditional Robotic Sensors, pp. 339-350. T.C. Henderson, ed., Springer Verlag, 1990.
[11] T. Viéville, F. Romann, B. Hotz, H. Mathieu, M. Buffa, L. Robert, P.E.D.S. Facao, O. Faugeras, and J.T. Audren, Autonomous Navigation of a Mobile Robot Using Inertial and Visual Cues Intelligent Robots and Systems, M. Kikode, T. Sato, and K. Tatsuno, eds., 1993.
[12] T. Viéville, E. Clergue, and P.E.D. Facao, Computation of Ego-Motion and Structure from Visual and Inertial Sensor Using the Vertical Cue Proc. Int'l Conf. Computer Vision, pp. 591-598, 1993.
[13] T. Viéville, A Few Steps Towards 3D Active Vision. Springer-Verlag, 1997.
[14] B. Bhanu, B. Roberts, and J. Ming, Inertial Navigation Sensor Integrated Motion Analysis for Obstacle Detection Proc. IEEE Int'l Conf. Robotics and Automation, pp. 954-959, 1990.
[15] F. Panerai and G. Sandini, Visual and Inertial Integration for Gaze Stabilization Proc. Int'l Symp. Intelligent Robotic Systems, 1997.
[16] F. Panerai and G. Sandini, Oculo-Motor Stabilization Reflexes: Integration of Inertial and Visual Information Neural Networks, vol. 11, nos. 7-8, pp. 1191-1204, 1998.
[17] F. Panerai, G. Metta, and G. Sandini, Visuo-Inertial Stabilization in Space-Variant Binocular Systems Robotics and Autonomous Systems, vol. 30, nos. 1-2, pp. 195-214, 2000.
[18] T. Mukai and N. Ohnishi, The Recovery of Object Shape and Camera Motion Using a Sensing System with a Video Camera and a Gyro Sensor Proc. Seventh Int'l Conf. Computer Vision, pp. 411-417, Sept. 1999.
[19] T. Mukai and N. Ohnishi, Object Shape and Camera Motion Recovery Using Sensor Fusion of a Video Camera and a Gyro Sensor Information Fusion, vol. 1, no. 1, pp. 45-53, 2000.
[20] R. Kurazume and S. Hirose, Development of Image Stabilization System for Remote Operation of Walking Robots Proc. IEEE Int'l Conf. Robotics and Automation, pp. 1856-1860, Apr. 2000.
[21] S.R. Coorg, Pose Imagery and Automated Three-Dimensional Modeling of Urban Environments PhD thesis, Massachusetts Inst. of Tech nology, Sept. 1998.
[22] S. You, U. Neumann, and R. Azuma, Hybrid Inertial and Vision Tracking for Augmented Reality Registration Proc. IEEE Virtual Realiy Conf., pp. 260-267, Mar. 1999.
[23] W.A. Hoff, K. Nguyen, and T. Lyon, Computer Vision-Based Registration Techniques for Augmented Reality Proc. Conf. Intelligent Robots and Computer Vision, pp. 538-548, Nov. 1996.
[24] E.D. Dickmanns, Vehicles Capable of Dynamic Vision: A New Breed of Technical Beings? Artificial Intelligence, vol. 103, pp. 49-76, 1998.
[25] J. Lobo and J. Dias, Integration of Inertial Information with Vision towards Robot Autonomy Proc. IEEE Int'l Symp. Industrial Electronics, pp. 825-830, July 1997.
[26] J. Lobo, L. Marques, J. Dias, U. Dias, and A.T. de Almeida, Sensors for Mobile Robot Navigation, pp. 50-81. Springer-Verlag, 1998.
[27] J. Lobo and J. Dias, Ground Plane Detection Using Visual and Inertial Data Fusion Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, pp. 912-917, Oct. 1998.
[28] J. Lobo, C. Queiroz, and J. Dias, Vertical World Feature Detection and Mapping Using Stereo Vision and Accelerometers Proc. Ninth Int'l Symp. Intelligent Robotic Systems, pp. 229-238, July 2001.
[29] J. Lobo and J. Dias, Fusing of Image and Inertial Sensing for Camera Calibration Proc. Int'l Conf. Multisensor Fusion and Integration for Intelligent Systems, pp. 103-108, Aug. 2001.
[30] J. Lobo, Inertial Sensor Data Integration in Computer Vision Systems MS thesis, Univ. of Coimbra, Apr. 2002.
[31] K. Kanatani, Geometric Computation for Machine Vision. New York, NY: Oxford Univ. Press, 1993.
[32] M.D. Shuster, The Kinematic Equation for the Rotation Vector IEEE Trans. Aerospace and Electronic Systems, vol. 29, no. 1, pp. 263-267, Jan. 1993.
[33] P.G. Savage, Strapdown System Algorithms. chapter 3, pp. 1-30, AGARD: Advisory Group for Aerospace Research and Development, 1984.
[34] M.J. Caruso, T. Bratland, C.H. Smith, and R. Schneider, A New Perspective on Magnetic Field Sensing technical report, Honeywell, Inc., 1998.
[35] Summit Instruments,http:/www.summitinstruments.com, 2003.
[36] Analog Devices, Mems Integrated Micro-Electromechanical Systems, Analog Devices, iMEMS @http:/www.analog.com/, 2003.
[37] B.K.P Horn, Closed-Form Solution of Absolute Orientation Using Unit Quaternions J. Optical Soc. Am., vol. 4, no. 4, pp. 629-462, Apr. 1987.
[38] J. Alves, J. Lobo, and J. Dias, Camera-Inertial Sensor Modelling and Alignment for Visual Navigation Proc. 11th Int'l Conf. Advanced Robotics, pp. 1693-1698, July 2003.
[39] J. Dias, C. Paredes, I. Fonseca, and A.T. de Almeida, Simulating Pursuit with Machines Proc. IEEE Conf. Robotics and Automation, pp. 472-477, 1995.
[40] J. Dias, C. Paredes, I. Fonseca, H. Araujo, J. Baptista, and A.T. de Almeida, Simulating Pursuit with Machine Experiments with Robots and Artificial Vision IEEE Trans. Robotics and Automation, vol. 3, no. 1, pp. 1-18, Feb. 1998.
[41] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000.
[42] L.-L. Wang and W.-H. Tsai, Camera Calibration by Vanishing Lines for 3-D Computer Vision IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 4, pp. 370-376, Apr. 1991.
[43] B. Caprile and V. Torre, “Using Vanishing Points for Camera Calibration,” Int'l J. Computer Vision, vol. 4, no. 2, pp. 127-140, Mar. 1990.
[44] B. Brillault and B. O'Mahony, New Method for Vanishing Point Detection Computer Vision, Graphics and Image Processing: Image Understanding, vol. 54, no. 2, pp. 289-300, 1991.
[45] M. Li, Camera Calibration of the Kth Head-Eye System Proc. European Conf. Computer Vision, pp. 543-554, 1994.
[46] R.G. Willson and S.A. Shafer, What is the Center of the Image J. Optical Soc. Am. A, vol. 11, no. 11, pp. 2946-2955, 1994.
[47] B. Jahne, Digital Image Processing. Springer-Verlag, 1997.
[48] Intel, Intel Open Source Computer Vision Library,http://www.intel.com/research/mrl/research opencv/, 2003.
[49] Z.Y. Zhang, A Flexible New Technique for Camera Calibration Proc. Int'l Conf. Computer Vision, vol. 1, pp. 666-673, Sept. 1999.
[50] S.M. Smith and J.M. Brady, “SUSAN—A New Approach to Low Level Image Processing,” Int'l J. Computer Vision, vol. 23, no. 1, pp. 45-78, 1997.
[51] J. O'Rourke, Computational Geometry in C. Cambridge Univ. Press, 1993.
[52] A.L. Ames, D.R. Nadeau, and J.L. Moreland, VRML 2.0 Sourcebook. second ed. John Wiley and Sons, 1997.
[53] J. Lobo, C. Queiroz, and J. Dias, World Feature Detection and Mapping Using Stereovision and Inertial Sensors Robotics and Autonomous Systems, vol. 44, no. 1, pp. 69-81, July 2003.
[54] J. Lobo, L. Almeida, and J. Dias, Segmentation of Dense Depth Maps Using Inertial Data: A Real-Time Implementation Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, pp. 92-97, Oct. 2002.
[55] J. Lobo and J. Dias, Inertial Sensed Ego-Motion for 3D Vision Proc. InerVis Workshop, 11th Int'l Conf. Advanced Robotics, pp. 1907-1914, July 2003.
Index Terms:
Image processing and computer vision, edge and feature detection, sensor fusion.
Citation:
Jorge Lobo, Jorge Dias, "Vision and Inertial Sensor Cooperation Using Gravity as a Vertical Reference," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 25, no. 12, pp. 1597-1608, Dec. 2003, doi:10.1109/TPAMI.2003.1251152
PURCHASE ARTICLE: $19
IEEE Xplore Subscribers
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb