Inspired by the movement of comet under gravitational interaction with planets and stars, we propose a novel wavelet-gravitation network (WGN) for endoscope optical tracking. We substitute the world gravitation law with interaction described by radial wavelet function. The WGN is trained based on the energy conservation law. The prediction of target image position is made with movement described by the second Newton?s law. The WGN is incorporated with Kalman filtering framework. Our experiments show that WGN performs especially well for nonlinear movement tracking. As the movement of surgical tools is handdriven, it is mostly nonlinear and affected by both tangential and normal accelerations. Our proposed methodology has been successfully applied to endoscope tracking for image guided surgery.