Estimation of Defects Based on Defect Decay Model: ED^{3}M
|
An accurate prediction of the number of defects in a software product during system testing contributes not only to the management of the system testing process but also to the estimation of the product's required maintenance. Here, a new approach called ED^3M is presented that computes an estimate of the total number of defects in an ongoing testing process. ED^3M is based on estimation theory. Unlike many existing approaches the technique presented here does not depend on historical data from previous projects or any assumptions about the requirements and/or testers' productivity. It is a completely automated approach that relies only on the data collected during an ongoing testing process. This is a key advantage of the ED^3M approach, as it makes it widely applicable in different testing environments. Here, the ED^3M approach has been evaluated using five data sets from large industrial projects and two data sets from the literature. In addition, a performance analysis has been conducted using simulated data sets to explore its behavior using different models for the input data. The results are very promising; they indicate the ED^3M approach provides accurate estimates with as fast or better convergence time in comparison to well-known alternative techniques, while only using defect data as the input.
[1] 336 V. Basili and B. Perricone, “Software Errors and Complexity: An Empirical Investigation,” Comm. ACM, vol. 27, no. 1, pp. 42-52, 1984.
[2] B.T. Compton and C. Withrow, “Prediction and Control of ADA Software Defects,” J. Systems and Software, vol. 12, pp. 199-207, July 1990.
[3] A.E. Ferdinand, “A Theory of System Complexity,” Int'l J. General Systems, vol. 1, pp. 19-33, 1974.
[4] L.C. Briand, K.E. Emam, B.G. Freimut, and O. Laitenberger, “A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content,” IEEE Trans. Software Eng., vol. 26, pp. 518-540, June 2000.
[5] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, “Locating where Faults Will Be,” Richard Tapia Celebration of Diversity in Computing Conf., pp. 48-50, Oct. 2005.
[6] P. Zhang and A. Mockus, “On Measurement and Understanding of Software Development Processes,” Technical Report ALR-2002-048, Avaya Research Labs, Nov. 2002.
[7] N. Fenton, M. Neil, W. Marsh, P. Hearty, D. Marquez, P. Krause, and R. Mishra, “Predicting Software Defects in Varying Development Lifecycles Using Bayesian Nets,” Information and Software Technology, vol. 49, pp. 32-43, Jan. 2007.
[8] N. Fenton, P. Krause, and M. Neil, “Software Measurement: Uncertainty and Causal Modeling,” IEEE Software, vol. 19, no. 4, pp. 116-122, July-Aug. 2002.
[9] J.-J. Gras, R. Gupta, and E. Pérez-Miñana, “Generating a Test Strategy with Bayesian Networks and Common Sense,” Proc. Testing: Academia and Industry Conf.—Practice and Research Techniques, pp. 29-40, Aug. 2006.
[10] J. Musa, Software Reliability Engineering. McGraw-Hill, 1999.
[11] A.L. Goel, “Software Reliability Models: Assumptions, Limitations, and Applicability,” IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1411-1423, Dec. 1985.
[12] A. Leon-Garcia, Probability and Random Processes for Electrical Engineering, second ed. Addison-Wesley, May 1994.
[13] S.M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice Hall PTR, 1993.
[14] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, “A Formal Model for the Software Test Process,” IEEE Trans. Software Eng., vol. 28, no. 8, pp. 782-796, Aug. 2002.
[15] D.E. Knuth, “The Errors of $T_{E}X$ ,” Software—Practice and Experience, vol. 19, no. 7, pp. 607-685, 1989.
[16] S.R. Dalal and C.L. Mallows, “Some Graphical Aids for Deciding when to Stop Testing Software,” IEEE J. Selected Areas in Comm., vol. 8, pp. 169-175, Feb. 1990.
[17] R. Jacoby and Y. Tohma, “The Hyper-Geometric Distribution Software Reliability Growth Model (HGDM): Precise Formulation and Applicability,” 1990.
[18] W.K. Ehrlich, J.P. Stampfel, and J.R. Wu, “Application of Software Reliability Modeling to Product Quality and Test Process,” Proc. 12th Int'l Conf. Software Eng., pp. 108-116, Mar. 1990.
[19] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, “Using Sensitivity Analysis to Validate a State Variable Model of the Software Test Process,” IEEE Trans. Software Eng., vol. 29, no. 5, pp. 430-443, May 2003.
[20] J.W. Cangussu, R.M. Karcich, R.A. DeCarlo, and A.P. Mathur, “Software Release Control Using Defect Based Quality Estimation,” Proc. 15th Int'l Symp. Software Reliability Eng., Nov. 2004.
[21] K.O. John and D. Musa, “A Logarithmic Poisson Execution Time Model for Software Reliability Measurement,” Proc. Seventh Int'l Conf. Software Eng., pp. 230-238, Mar. 1984.
[22] J.D. Musa, A. Iannino, and K. Okumoto, Software Reliability Measurement, Prediction, Application. McGraw-Hill, 1987.
[23] A.P. Nikora, Computer Aided Software Reliability Estimation (CASRE) User's Guide, Open Channel Foundation, NASA, Jet Propulsion Laboratory, Mar. 2003.
[24] G. Seber and C. Wild, Nonlinear Regression. John Wiley & Sons, 2003.
[25] N.E. Fenton and M. Neil, “A Critique of Software Defect Prediction Models,” IEEE Trans. Software Eng., vol. 25, no. 5, pp.675-689, Sept.-Oct. 1999.
[26] V.Y. Shen, T. Yu, S.M. Thebaut, and L.R. Paulsen, “Identifying Error-Prone Software—An Empirical Study,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 317-323, Apr. 1985.
[27] K.H. Moeller and D. Paulish, “An Empirical Investigation of Software Fault Distribution,” Proc. First Int'l Software Metrics Symp., pp. 82-90, May 1993.
[28] L. Hatton, “Re-Examining the Fault Density-Component Size Connection,” IEEE Software, vol. 14, no. 2, pp. 89-98, Mar.-Apr. 1997.
[29] J.R. Gaffney, “Estimating the Number of Faults in Code,” IEEE Trans. Software Eng., vol. 10, pp. 459-464, July 1984.
[30] M.D. Neil, “Statistical Modelling of Software Metrics,” PhD thesis, South Bank Univ. and Strathclyde Univ., 1992.
[31] S. Henry and D. Kafura, “The Evaluation of Software System's Structure Using Quantitative Software Metrics,” Software—Practice and Experience, vol. 14, pp. 561-573, June 1984.
[32] N. Ohlsson and H. Alberg, “Predicting Error-Prone Software Modules in Telephone Switches,” IEEE Trans. Software Eng., vol. 22, no. 12, pp. 886-894, Dec. 1996.
[33] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, “Predicting the Location and Number of Faults in Large Software Systems,” IEEE Trans. Software Eng., vol. 31, no. 4, pp. 340-355, Apr. 2005.
[34] R.M. Bell, T.J. Ostrand, and E.J. Weyuker, “Looking for Bugs in All the Right Places,” Proc. Int'l Symp. Software Testing and Analysis, pp. 61-72, July 2006.
[35] F. Padberg, “A Fast Algorithm to Compute Maximum Likelihood Estimates for the Hypergeometric Software Reliability Model,” Proc. Second Asia-Pacific Conf. Quality Software, pp. 40-49, Dec. 2001.
[36] F. Padberg, “Maximum Likelihood Estimates for the Hypergeometric Software Reliability Model,” Int'l J. Reliability, Quality, and Safety Eng., July 2002.
[37] C. Bai, K.-Y. Cai, and T.Y. Chen, “An Efficient Defect Estimation Method for Software Defect Curves,” Proc. 27th Ann. Int'l Computer Software and Applications Conf., Nov. 2003.
[38] P. Zhang and A. Mockus, “On Measurement and Understanding of Software Development Processes,” Technical Report ALR-2002-048, Avaya Research Labs, Nov. 2002.
[39] P. Zhang, A. Mockus, and D. Weiss, “Understanding and Predicting Effort in Software Projects,” Proc. 25th Int'l Conf. Software Eng., pp. 274-284, 2003.
[40] D. Satoh and S. Yamada, “Discrete Equations and Software Reliability Growth Models,” Proc. 12th Int'l Symp. Software Reliability Eng., pp. 176-184, Nov. 2001.
[41] A. Onoma and T. Yamaura, “Practical Steps toward Quality Development,” IEEE Software, vol. 12, no. 4, pp. 68-77, July 1995.
[42] R.-H. Hou, S.-Y. Kuo, and Y.-P. Chang, “Applying Various Learning Curves to Hyper-Geometric Distribution Software Reliability Growth Model,” Proc. Fifth Int'l Symp. Software Reliability Eng., pp. 18-24, Nov. 1994.
[43] J.D. Musa, “Software Reliability Data,” technical report, Data and Analysis Center for Software, 1979.
[44] R. AL-Ekram, “Software Reliability Growth Modeling and Prediction,” Technical Report ECE 750, Dept. of Computer Science, Univ. of Waterloo, Mar. 2002.
[45] K.T.S.S. Gokhale and M.R. Lyu, “Model Validation Using Simulated Data,” Proc. First IEEE Workshop Application-Specific Software Eng. and Technology, pp. 22-27, Mar. 1998.
[46] I. Burnstein, T. Suwannasart, and C.R. Carlson, “Developing a Testing Maturity Model: Part 1,” CrossTalk, J. Defense Software Eng., vol. 9, pp. 21-24, Aug. 1996.
[47] I. Burnstein, T. Suwannasart, and C.R. Carlson, “Developing a Testing Maturity Model: Part 2,” CrossTalk, J. Defense Software Eng., vol. 9, pp. 19-26, Sept. 1996.
[48] I. Burnstein, A. Homyen, T. Suwanassart, G. Saxena, and R. Grom, “A Testing Maturity Model for Software Test Process Assessment and Improvement,” Software Quality Professional, vol. 1, Sept. 1999.
[49] D.G. Luenberger, Introduction to Dynamic Systems: Theory, Models and Applications. John Wiley & Sons, 1979.
Index Terms:
Statistical methods, Testing and Debugging, Metrics/Measurement, Defect prediction, system testing, estimation theory
Citation:
Syed W. Haider, João W. Cangussu, Kendra M.L. Cooper, Ram Dantu, Syed Haider, "Estimation of Defects Based on Defect Decay Model: ED^{3}M," IEEE Transactions on Software Engineering, vol. 34, no. 3, pp. 336-356, May/June 2008, doi:10.1109/TSE.2008.23
PURCHASE ARTICLE: $19
IEEE Xplore Subscribers
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb