Combinatorial interaction testing tools for software product lines engineering: a comparative analysis

• Ano de defesa: 2018
• Autor(a) principal: Silva, Denivan do Carmo Campos da
• Orientador(a): Machado, Ivan do Carmo
• Banca de defesa: Assunção, Wesley Klewerton Guêz, Souza, Rodrigo Rocha Gomes e
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Universidade Federal da Bahia; Instituto de Matemática e Estatística; Departamento de Ciência da Computação
• Programa de Pós-Graduação: em Ciência da Computação
• País: brasil
• Palavras-chave em Português: Software Product Lines (SPL); Combinatorial Interaction Testing (CTI); Empirical Software Engineering; Software; Engenharia de software
• Área do conhecimento CNPq: Ciência exatas e da Terra; Ciência da Computação; Metodologia e Técnicas da Computação; Engenharia de Software
• Link de acesso: http://repositorio.ufba.br/ri/handle/ri/33544

Testing a system is a routine activity and plays an important role in the software quality assurance process. However, testing highly-configurable systems, such as Software Product Lines (SPL), is a complex activity due to the presence of variability in its engineering process, which increases the number of product configurations to test. In case a defect affects one (or a subset) of these functionalities, a range of products (and not just one, such as in traditional Software Engineering, in which each product is built from scratch) may be affected. Such complexity also implies a significant increase in the cost of testing. The use of tool-supported sampling testing techniques could bring significant contributions to achieve reductions in such a cost. Among the most effective techniques, Combinatorial Interaction Testing (CIT) has been used extensively to sample inputs to software, and to test highly-configurable software systems. CIT is based on the premise that many errors in software can only arise from the interaction of two or more parameters. CIT take as input a configuration model that defines the valid configuration space for the software under test. This model typically includes a set of configuration options, each of which takes a value from a small number of discrete settings, and a set of system-wide constraints among configuration options. Given the model, these methods compute a t-way covering array - a set of configurations, in which each valid combination of option settings for every combination of t options appears at least once. The system is then tested by running its test suite in all the configurations selected. In this investigation, we aimed to analyze the effectiveness of existing tool support for CIT. To accomplish ou goal, we introduced the MERCI - a Method to Evaluate Combinatory Interaction Testing techniques, aimed to establish a systematic means to evaluate the adequacy of existing CIT tools for highly-configurable systems testing. In this work, we performed an empirical evaluation to compare four CIT tools: ACTS, CATS, PICTMaster and VPTag. The analysis considered the metrics defect detection, test coverage and test execution length. The yielded results show that the method could be employed as a good mechanism to evaluate how CIT tools could behave in a practical SPL testing scenario.