This paper proposes a multi-agent system-based grid middleware. There exist three types of agents in the middleware according to different functions. Each type of agent is further constructed to be a multi-agent system. The multi-agent system dispatches a set of individual agents to coordinate a user job over grid in a decentralized manner. In this paper, the middleware?s four layers architecture is detailed and multi-agent system work principle is analyzed. Also the individual agent?s implementation based on Java is illuminated. Finally the experiment result is presented via comparing the traditional grid with multi-agent system-based grid. The result indicates that our proposed approach can obtain about fifteen percent higher communication performance in the point-to-point communication.