Mutation testing is based upon seeding the implementation with a fault (mutating it), by applying a mutation operator, and determining whether testing identifies this fault. A mutated program is called a mutant and if a test case distinguishes between the mutant and the original program it is said to kill the mutant. The idea behind mutation testing is quite simple: given an appropriate set of mutation operators, if a test set kills all of the mutants generated by these operators then, since it is able to find these small differences, it is likely to be good at finding real faults.

Mutation testing may be used to judge the effectiveness of a test suite: it should kill all the mutants. Similarly, test generation may be based on mutation testing: tests are generated to kill the mutants. Interestingly, many test criteria may be represented using mutation testing by simply choosing appropriate mutation operators.

While mutation testing is a powerful and general technique, a number of associated problems have limited its practical impact. One of these problems is that the standard set of mutation operators may lead to a vast number of mutants. However, it has been noted that this problem can be reduced by an appropriate choice of mutation operators. The following are the other main problems that we have considered.

• Semantic mutation testing: rather than mutate the program or model, which is a syntactic entity, we can mutate the semantics of the language in which it is written. The aim here is to capture a different class of possible mistake: one that results from a misunderstanding of the semantics of the language used rather than from a syntactic slip.
• While a mutant is syntactically different from the initial program it may have the same behaviour as the program: a mutant may be equivalent to the original program. Clearly equivalent mutants can never be killed and thus it is important, but often difficult, to identify them. Some approaches to detecting equivalent mutants and reducing the number of equivalent mutants produced have been introduced.
• Some mutants, called stubborn mutants, may be non-equivalent but difficult to kill.
• Extending mutation testing to state-machines that have probabilities and/or stochastic time.

• J. Clark, H. Dan, and R. M. Hierons, Semantic Mutation Testing, Mutation 2010.
• R. M. Hierons and M. G. Merayo: Mutation Testing from Probabilistic and Stochastic Finite State Machines, Journal of Systems and Software, 82 11, pp. 1804-1818, 2009.
• R. M. Hierons and M. G. Merayo: Mutation Testing from Probabilistic Finite State Machines, Third Workshop on Mutation Analysis (Mutation 2007), September 10-11, 2007.
• K. Adamopoulos, M. Harman, R. M. Hierons: Mutation Testing Using Genetic Algorithms: A Co-evolution Approach, AAAI Genetic and Evolutionary Computation Conference 2004 (GECCO 2004), pp. 1338-1349, 2004.
• M. Harman, R. M. Hierons and S. Danicic: The relationship between program dependence and mutation testing, Mutation 2000, San Jose, California, USA, pp. 15-23, 2000.
• R.M. Hierons, M. Harman and S. Danicic: Using Program Slicing to Assist in the Detection of Equivalent Mutants, The Journal of Software Testing, Verification, and Reliability, 9 4, pp. 233-262, 1999.

Back to Rob Hierons' home page

Last updated: May 2010.