Test amplification
DSpot
Publication date: February 7, 2016
Description: Automatically detect and generate missing assertions for Junit test cases (also known as test amplification).
Repository: https://github.com/STAMP-project/dspot
License: LGPL-3.0 License
Tags: java coverage test-automation junit test-amplification hacktoberfest inria h2020
Diff-test-selection
Publication date: February 7, 2016
Description: Diff-Test-Selection aims at selecting the subset of test classes and methods that execute the changed code between two versions of the same program. It relies on Clover and Maven to compute the coverage and returns this subset of tests. Diff-Test-Selection is implemented as a maven plugin that directly works from the command line, without modifying your pom.xml. See below for more details.
Repository: https://github.com/STAMP-project/dspot-diff-test-selection
License: LGPL-3.0 License
Tags: java coverage test-automation junit test-amplification hacktoberfest inria h2020
An approach and benchmark to detect behavioral changes of commits in continuous integration.
Authors: Benjamin Danglot, Martin Monperrus, Walter Rudametkin and Benoit Baudry.
Venue: Empirical Software Engineering 25, 2379–2415 (2020). https://doi.org/10.1007/s10664-019-09794-7
Publication date: March 5, 2020
Abstract: When a developer pushes a change to an application’s codebase, a good practice is to have a test case specifying this behavioral change. Thanks to continuous integration (CI), the test is run on subsequent commits to check that they do no introduce a regression for that behavior. In this paper, we propose an approach that detects behavioral changes in commits. As input, it takes a program, its test suite, and a commit. Its output is a set of test methods that capture the behavioral difference between the pre-commit and post-commit versions of the program. We call our approach DCI (Detecting behavioral changes in CI). It works by generating variations of the existing test cases through (i) assertion amplification and (ii) a search-based exploration of the input space. We evaluate our approach on a curated set of 60 commits from 6 open source Java projects. To our knowledge, this is the first ever curated dataset of real-world behavioral changes. Our evaluation shows that DCI is able to generate test methods that detect behavioral changes. Our approach is fully automated and can be integrated into current development processes. The main limitations are that it targets unit tests and works on a relatively small fraction of commits. More specifically, DCI works on commits that have a unit test that already executes the modified code. In practice, from our benchmark projects, we found 15.29% of commits to meet the conditions required by DCI.
Tags: Continuous Integration Test amplification Behavioral change detection
DCI Web page
Authors: Benjamin Danglot, Martin Monperrus, Walter Rudametkin and Benoit Baudry.
Venue: Empirical Software Engineering 25, 2379–2415 (2020). https://doi.org/10.1007/s10664-019-09794-7
Publication date: March 5, 2020
Abstract: When a developer pushes a change to an application’s codebase, a good practice is to have a test case specifying this behavioral change. Thanks to continuous integration (CI), the test is run on subsequent commits to check that they do no introduce a regression for that behavior. In this paper, we propose an approach that detects behavioral changes in commits. As input, it takes a program, its test suite, and a commit. Its output is a set of test methods that capture the behavioral difference between the pre-commit and post-commit versions of the program. We call our approach DCI (Detecting behavioral changes in CI). It works by generating variations of the existing test cases through (i) assertion amplification and (ii) a search-based exploration of the input space. We evaluate our approach on a curated set of 60 commits from 6 open source Java projects. To our knowledge, this is the first ever curated dataset of real-world behavioral changes. Our evaluation shows that DCI is able to generate test methods that detect behavioral changes. Our approach is fully automated and can be integrated into current development processes. The main limitations are that it targets unit tests and works on a relatively small fraction of commits. More specifically, DCI works on commits that have a unit test that already executes the modified code. In practice, from our benchmark projects, we found 15.29% of commits to meet the conditions required by DCI.
Description: When a developer change the behavior a program, he should provide a new test method (or modify an existing one) that specify the new behavior. However, developers do not always provide such test method because lack of time, expertise or discipline. In the context of the article “An Approach and Benchmark to Detect Behavioral Changes of Commits in Continuous Integration”, we devised a technique that produces automatically a test method that specify the new behavior. On this webpage, you can find: first, the seed test method that we amplified; second, the amplified test method that detect the behavioral change; third, the manual test method provided by the developer with the commit, used as a ground truth.
Tags: Continuous Integration Test amplification Behavioral change detection
A snowballing literature study on test amplification.
Authors: BenjaminDanglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit Baudry
Venue: Journal of Systems and Software Volume 157, November 2019, 110398, https://doi.org/10.1016/j.jss.2019.110398
Publication date: November 1, 2019
Abstract: The adoption of agile approaches has put an increased emphasis on testing, resulting in extensive test suites. These suites include a large number of tests, in which developers embed knowledge about meaningful input data and expected properties as oracles. This article surveys works that exploit this knowledge to enhance manually written tests with respect to an engineering goal (e.g., improve coverage or refine fault localization). While these works rely on various techniques and address various goals, we believe they form an emerging and coherent field of research, which we coin “test amplification”. We devised a first set of papers from DBLP, searching for all papers containing “test” and “amplification” in their title. We reviewed the 70 papers in this set and selected the 4 papers that fit the definition of test amplification. We use them as the seeds for our snowballing study, and systematically followed the citation graph. This study is the first that draws a comprehensive picture of the different engineering goals proposed in the literature for test amplification. We believe that this survey will help researchers and practitioners entering this new field to understand more quickly and more deeply the intuitions, concepts and techniques used for test amplification.
Tags: Test amplification Test augmentation Test optimization Test regeneration Automatic testing