Title: DoKnowMe: Towards a Domain Knowledge-driven Methodology for Performance Evaluation,
Abstract: Software engineering considers performance evaluation to be one of the key
portions of software quality assurance. Unfortunately, there seems to be a lack
of standard methodologies for performance evaluation even in the scope of
experimental computer science. Inspired by the concept of "instantiation" in
object-oriented programming, we distinguish the generic performance evaluation
logic from the distributed and ad-hoc relevant studies, and develop an abstract
evaluation methodology (by analogy of "class") we name Domain Knowledge-driven
Methodology (DoKnowMe). By replacing five predefined domain-specific knowledge
artefacts, DoKnowMe could be instantiated into specific methodologies (by
analogy of "object") to guide evaluators in performance evaluation of different
software and even computing systems. We also propose a generic validation
framework with four indicators (i.e.~usefulness, feasibility, effectiveness and
repeatability), and use it to validate DoKnowMe in the Cloud services
evaluation domain. Given the positive and promising validation result, we plan
to integrate more common evaluation strategies to improve DoKnowMe and further
focus on the performance evaluation of Cloud autoscaler systems.