Most performance analysis today uses either microbenchmarks or standard macrobenchmarks (e.g., SPEC, LADDIS, the Andrew benchmark). However, the results of such benchmarks provide little information to indicate how well a particular system will handle a particular application. Such results are, at best, useless and, at worst, misleading. In this paper, we argue for an application-directed approach to benchmarking, using performance metrics that reflect the expected behavior of a particular application across a range of hardware or software platforms. We present three different approaches to application-specific measurement, one using vectors that characterize both the underlying system and an application, one using trace-driven techniques, and a hybrid approach. We argue that such techniques should become the new standard.
Index Terms:
benchmark, performance, application-specific, measurement
Citation:
Margo Seltzer, David Krinsky, Keith Smith, Xiaolan Zhang, "The Case for Application-Specific Benchmarking," hotos, pp.102, The Seventh Workshop on Hot Topics in Operating Systems, 1999