A Controlled Experiment to Assess the Benefits of Procedure Argument Type Checking
|
Abstract—Type checking is considered an important mechanism for detecting programming errors, especially interface errors. This report describes an experiment to assess the defect-detection capabilities of static, intermodule type checking. The experiment uses ANSI C and Kernighan&Ritchie (K&R) C. The relevant difference is that the ANSI C compiler checks module interfaces (i.e., the parameter lists calls to external functions), whereas K&R C does not. The experiment employs a counterbalanced design in which each of the 40 subjects, most of them CS PhD students, writes two nontrivial programs that interface with a complex library (Motif). Each subject writes one program in ANSI C and one in K&R C. The input to each compiler run is saved and manually analyzed for defects. Results indicate that delivered ANSI C programs contain significantly fewer interface defects than delivered K&R C programs. Furthermore, after subjects have gained some familiarity with the interface they are using, ANSI C programmers remove defects faster and are more productive (measured in both delivery time and functionality implemented).
[1] 302 V.R. Basili and B.T. Perricone,“Software errors and complexity: An empirical investigation,” Comm. ACM, vol. 27, no. 1, pp. 42-52, Jan. 1984.
[2] B. Beizer, Software Testing Techniques, second ed. Boston, Mass.: Int'l Thomson Computer Press, 1990.
[3] K. Bruce, "Typing in Object-Oriented Languages: Achieving Expressibility and Safety," ACM Computing Surveys? 1998, to appear. seehttp:/www.cs.williams.edu~kim/.
[4] L.B. Christensen, Experimental Methodology. Allyn and Bacon, Needham Heights, Mass., sixth edition, 1994.
[5] C.R. Cook, J.C. Scholtz, and J.C. Spohrer, eds.. Empirical Studies of Programmers: Fifth Workshop,Palo Alto, Calif., Ablex Publishing Corp., Dec. 1993
[6] A. Ebrahimi, "Novice Programmer Errors: Language Constructs and Plan Composition," Intl. J. Human-Computer Studies, vol. 41, pp. 457-480, 1994.
[7] M. Eisenstadt, "Tales of Debugging from the Front Lines," [5], pp. 86-112, 1993.
[8] P.G. Frankl and S.N Weiss,“An experimental comparison of the effectiveness of branch testing and data flow testing,” IEEE Transactions on Software Engineering, vol. 19, no. 8, Oct. 1993, pp. 774-787.
[9] J.D. Gannon, "An Experimental Evaluation of Data Type Conventions," Comm. ACM, 1977.
[10] R.B. Grady, "Practical Results from Measuring Software Quality," Comm. ACM, vol. 36, no. 11, pp. 62-68, Nov. 1993.
[11] P. Hudak and M.P. Jones, "Haskell vs. Ada vs. C++ vs. Awk vs. ... An Experiment in Software Prototyping Productivity," technical report, Dept. of Computer Science, Yale Univ., New Haven, Conn., July 1994.
[12] W.S. Humphrey, A Discipline for Software Engineering, Addison-Wesley, Reading, Mass., 1995.
[13] M. Nanja and C.R. Cook, "An Analysis of the On-Line Debugging Process," [14], pp. 172-184, 1987.
[14] G.M. Olson, S. Sheppard, and E. Soloway, eds., "Empirical Studies of Programmers: Second Workshop,"Washington, D.C., Ablex Publishing Corp., Dec. 1987.
[15] L. Prechelt and W.F. Tichy, "A Controlled Experiment Measuring the Impact of Procedure Argument Type Checking on Programmer Productivity," Technical Report CMU/SEI-96-TR-014, Software Engineering Inst., Carnegie Mellon Univ., Pittsburgh, Penn., June 1996.
[16] B.A. Sheil, "The Psychological Study of Programming," ACM Computing Surveys, 1981.
[17] E. Soloway, and S. Iyengar, eds., Empirical Studies of Programmers.Norwood, N.J.: Ablex Publishing Corp., June 1986. (The papers of the First Workshop on Empirical Studies of Programmers, Washington D.C.).
[18] J.G. Spohrer and E. Soloway, "Analyzing the High Frequency Bugs in Novice Programs," [17], pp. 230-251, 1986.
[19] W. Stacy and J. MacMillian, "Cognitive Bias in Software Engineering," Comm. ACM, vol. 38, no. 6, pp. 57-63, June 1995.
[20] B. Teasley, L.M. Leventhal, and D.S. Rohlman, "Positive Test Bias in Software Testing by Professionals: What's Right and What's Wrong," Empirical Studies of Programmers: Proc. Fifth Workshop, pp. 206-221.Palo Alto, Calif.: Ablex Publishing Corp., Dec. 1993.
[21] N. Wirth, "Gedanken zur Software-Explosion," Informatik Spektrum, vol. 17, no. 1, pp. 5-20, Feb. 1994.
[22] C. Wohlin and P. Runeson, "Certification of Software Components," IEEE Trans. Software Eng., vol. 20, no. 6, pp. 494-499, June 1994.
Index Terms:
Type checking, defects, quality, productivity, controlled experiment.
Citation:
Lutz Prechelt, Walter F. Tichy, "A Controlled Experiment to Assess the Benefits of Procedure Argument Type Checking," IEEE Transactions on Software Engineering, vol. 24, no. 4, pp. 302-312, Apr. 1998, doi:10.1109/32.677186
PURCHASE ARTICLE: $19
IEEE Xplore Subscribers
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb