General Test Result Checking with Log File Analysis
|
Abstract—We describe and apply a lightweight formal method for checking test results. The method assumes that the software under test writes a text log file; this log file is then analyzed by a program to see if it reveals failures. We suggest a state-machine-based formalism for specifying the log file analyzer programs and describe a language and implementation based on that formalism. We report on empirical studies of the application of log file analysis to random testing of units. We describe the results of experiments done to compare the performance and effectiveness of random unit testing with coverage checking and log file analysis to other unit testing procedures. The experiments suggest that writing a formal log file analyzer and using random testing is competitive with other formal and informal methods for unit testing.
[1] 634 J.H. Andrews, “Testing Using Log File Analysis: Tools, Methods and Issues,” Proc. 13th Ann. Int'l Conf. Automated Software Eng. (ASE '98), pp. 157–166, Oct. 1998.
[2] J.H. Andrews and Y. Zhang, Broad-Spectrum Studies of Log File Analysis Proc. 22nd Int'l Conf. Software Eng. (ICSE 2000), pp. 105-114, June 2000.
[3] P. Bates and J.C. Wileden, An Approach to High-Level Debugging of Distributed Systems Proc. ACM SIGPLAN Notices, vol. 18, no. 8, pp. 107-111, Aug. 1983.
[4] B. Beizer, Software Testing Techniques, second ed. Boston, Mass.: Int'l Thomson Computer Press, 1990.
[5] G.S. Boolos and R.C. Jeffrey, Computability and Logic, second ed. Cambridge, U.K.: Cambridge Univ. Press, 1980.
[6] D. Brown et al., "An Automated Oracle for Software Testing," IEEE Trans. Reliability, June 1992, pp. 272-279.
[7] Bullseye Tech nologies, C-Cover www.bullseye.comccover html, 2002.
[8] M. Chechik and J. Gannon, “Automatic Verification of Requirements Implementations,” Proc. 1994 Int'l Symp. Software Testing and Analysis (ISSTA), pp. 1–14, Aug. 1994.
[9] M.R. Donat, Automating Formal Specification-Based Testing TAPSOFT: Proc. Seventh Int'l Joint Conf. Theory and Practice of Software Eng., pp. 833-847, Apr. 1997.
[10] R.-K. Doong and P.G. Frankl, The ASTOOT Approach to Testing Object-Oriented Programs ACM Trans. Software Eng. and Methodology, vol. 3, no. 2, pp. 101-130, Apr. 1994.
[11] J.W. Duran and S.C. Ntafos, An Evaluation of Random Testing IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
[12] M.S. Feather, Rapid Application of Lightweight Formal Methods for Consistency Analyses IEEE Trans. Software Eng., vol. 24, no. 11, pp. 949-969, Nov. 1998.
[13] Free Software Foundation, gcov: A Test Coverage Program Free Software Foundation, 2001.
[14] D. Gueraichi and L. Logrippo, Derivation of Test Cases for LAP-B from a LOTOS Specification Proc. Second FORmal TEchniques Int'l Conf. (FORTE 1989), S. Vuong, ed., pp. 361-374, 1989.
[15] D. Harel, “Statecharts: A Visual Approach to Complex Systems,” Science of Computer Programming, Vol. 8, No. 3, pp. 231–274, 1987.
[16] D. Heimbold and D. Luckham, Debugging Ada Tasking Programs IEEE Software, vol. 2, no. 2, pp. 47-57, Mar. 1985.
[17] K. Heninger, Specifying Software Requirements for Complex Systems: New Techniques and Their Applications IEEE Trans. Software Eng., vol. 6, no. 1, pp. 2-12, Jan. 1980.
[18] D. Hoffman and P. Strooper, Classbench: A Framework for Automated Class Testing Software Practice and Experience, vol. 27, no. 5, pp. 573-597, May 1997.
[19] J. Joyce et al., "Monitoring Distributed Systems," ACM Trans. Computer Systems, Feb. 1987, pp. 121-150.
[20] S. Kim, J. A. Clark, and J. A. McDermid, Investigating the Effectiveness of Object-Oriented Testing Strategies with the Mutation Method Software Testing, Verification, and Reliability, vol. 11, no. 3, pp. 207-225, Sept. 2001.
[21] S. Kirani and W.T. Tsai, Method Sequence Specification and Verification of Classes J. Object-Oriented Programming, vol. 7, no. 6, pp. 28-38, Oct. 1994.
[22] J. Kundu and J.E. Cuny, A Scalable, Visual Interface for Debugging with Event-Based Behavioral Abstraction Proc. New Frontiers on Massively Parallel Processing, pp. 472-479, Feb. 1995.
[23] D. Kung, N. Suchak, J. Gao, P. Hsia, Y. Toyoshima, and C. Chen, “On Object State Testing,” Proc. COMPSAC '94 18th Int'l Computer Software and Applications Conf., pp. 222–227, Nov. 1994.
[24] T. Kunz, J. Black, D. Taylor, and T. Basten, Poet: Target-System-Independent Visualizations of Complex Distributed-Application Executions The Computer J., vol. 40, no. 8, pp. 499-512, Sept. 1997.
[25] R. Lencevicius, U. Hölzle, and A.K. Singh, “Query-Based Debugging of Object-Oriented Programs,” Proc. Conf. Object-Oriented Programming, Systems, Languages, and Applications, pp. 304–317, Oct. 1997.
[26] R. Lencevicius, A. Ran, and R. Yairi, Apache Web Server Execution Tracing Using Third Eye Proc. Fourth Int'l Workshop Automated Debugging (AADEBUG 2000), M. Ducassé, ed., Aug. 2000.
[27] N.G. Leveson, S.S. Cha, and T.J. Shimeall, "Safety Verification of Ada Programs Using Software Fault Trees," IEEE Software, pp. 48-59, July 1991.
[28] D. Luckham, S. Sankar, and S. Takahashi, "Two-Dimensional Pinpointing: Debugging with Formal Specifications," IEEE Software, Jan. 1991.
[29] D.C. Luckham and F.W. von Henke, An Overview of Anna, a Specification Language for Ada Proc. IEEE Conf. Ada Applications and Environments, pp. 116-127, Oct. 1984.
[30] P. Maker, Nana: Improved Support for Assertions and Logging in C and C++ Technical Report 12-95, School of Information Technology, Northern Territory Univ., Darwin, NT, Australia, Sept. 1995.
[31] J. McDonald, D. Hoffman, and P. Strooper, Programmatic Testing of the Standard Template Library Containers Proc. Int'l Conf. Automated Software Eng. (ASE'98), pp. 147-156, Oct. 1998.
[32] C.C. Michael, G. McGraw, and M.A. Schatz, Generating Software Test Data by Evolution IEEE Trans. Software Eng., vol. 27, no. 12, Dec. 2001.
[33] J. Offutt and R. Untch, Mutation 2000: Uniting the Orthogonal Mutation 2000: Mutation Testing in the Twentieth and the Twenty First Centuries, pp. 45-55, Oct. 2000.
[34] T.O. O'Malley, D.J. Richardson, and L.K. Dillon, Efficient Specification-Based Oracles for Critical Systems Proc. California Software Symp., pp. 50-59, Apr. 1996.
[35] D.K. Peters and D.L. Parnas, “Using Test Oracles Generated from Program Documentation,” IEEE Trans. Software Eng., vol. 24, no. 3, pp. 161–173, Mar. 1998.
[36] S. Qiao and H. Zhang, An Automatic Logfile Analyzer for Parallel Programs Proc. Int'l Conf. Parallel and Distributed Processing Techniques and Applications, H.R. Arabnia, ed., vol. III, pp. 1371-1376, 1999.
[37] D.J. Richardson, S.L. Aha, and T.O. O'Malley, “Specification-Based Test Oracles for Reactive Systems,” Proc. Int'l Conf. Software Eng. (ICSE), pp. 105–118, May 1992.
[38] D.S. Rosenblum, “A Practical Approach to Programming With Assertions,” IEEE Trans. Software Eng., vol. 21, no. 1, pp. 19–31, Jan. 1995.
[39] S. Sankar, A. Goyal, and P. Sikchi, Software Testing Using Algebraic Specification Based Test Oracles Technical Report CSL-TR-93-566, Computer Systems Laboratory, Stanford Univ., Apr. 1993.
[40] D. Sannella and A. Tarlecki, Toward Formal Development of Programs from Algebraic Specifications: Implementations Revisited Acta Informatica, vol. 25, no. 3, pp. 233-281, 1988.
[41] K.C. Tai, R.H. Carver, and E.E. Obaid, "Debugging Concurrent Ada Programs by Deterministic Execution," IEEE Trans. Soft. Eng., vol. 17, no. 1, pp. 45-63, Jan. 1991.
[42] J. Voas et al., "Predicting How Badly 'Good' Software Can Behave," IEEE Software, July/Aug. 1997, pp. 73-83.
[43] G. v. Bochmann, R. Dssouli, and J. Zhao, "Trace Analysis for Conformance and Arbitration Testing," IEEE Trans. Software. Eng., vol. 15, no. 11, pp. 1,347-1,356, Nov. 1989.
[44] E.J. Weyuker, On Testing Non-Testable Programs The Computer J., vol. 25, no. 4, pp. 465-470, Nov. 1982.
[45] Y. Zhang, Random Testing with Log File Analysis master's thesis, Dept. of Computer Science, Univ. of Western Ontario, London, Ontario, Canada, Sept. 2000.
Index Terms:
Testing, specification, safety verification, lightweight formal methods, test oracles, unit testing, log file analysis
Citation:
James H. Andrews, Yingjun Zhang, "General Test Result Checking with Log File Analysis," IEEE Transactions on Software Engineering, vol. 29, no. 7, pp. 634-648, July 2003, doi:10.1109/TSE.2003.1214327
PURCHASE ARTICLE: $19
IEEE Xplore Subscribers
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb