The security of a system not only depends on the quality of the encryption algorithm it used, but also on the mechanism of key management. Traditional method layered key management is complex and inefficient for a database encryption system as well. In this paper, we apply Java Card to the database encryption system to solve the problem of key management and access control. By fully utilizing the cipher support of Java Card and its advantage of safe storage and portability, the complexity of layered key management is reduced. This paper also proposes a double-factor based indemnification, access control and data transfer scheme for the database encryption system based on Java Card. The most prominent advantage of our scheme is to enhance both efficiency and reliability of the system. Security analysis and simulation experiment is also given for supporting its validity.