Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming José Ribeiro1 Mário Zenha-Rela2 Francisco Vega3 1 Polytechnic 2 University 3 University

Institute of Leiria, Portugal

of Coimbra, Portugal (supervisor )

of Extremadura, Spain (supervisor )

GECCO’08 Graduate Student Workshop

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming

Outline Introduction Software Testing Evolutionary Algorithms Evolutionary Testing Technical Approach Overview Input Domain Reduction Test Case Evaluation Experimental Studies Test Case Evaluation Case Study Input Domain Reduction Case Study Conclusions and Future Work

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction

Outline Introduction Software Testing Evolutionary Algorithms Evolutionary Testing Technical Approach Overview Input Domain Reduction Test Case Evaluation Experimental Studies Test Case Evaluation Case Study Input Domain Reduction Case Study Conclusions and Future Work

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction Software Testing

Software Testing I

Software testing is expensive, typically consuming roughly half of the total costs involved in software development – while adding nothing to the raw functionality of the final product.

I

Yet, it remains the primary method through which confidence in software is achieved.

Test Data Generation I

Automating the test data generation process is vital to advance the state-of-the-art in software testing.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction Software Testing

Unit-Testing

I

When performing unit-testing, the goal is to warrant the robustness of the smallest units – the test objects – by testing them in an isolated environment.

I

Unit-testing is performed by executing the test objects in different scenarios using relevant test cases.

I

A test set is said to be adequate with respect to a given criterion if the entirety of test cases in this set satisfies this criterion.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction Evolutionary Algorithms

Evolutionary Algorithms I

Evolutionary Algorithms use simulated evolution as a search strategy to evolve candidate solutions, using operators inspired by genetics and natural selection.

Genetic Programming I

Genetic Programming is a machine-learning approach usually associated with the evolution of tree structures.

I

It focuses on automatically creating computer programs by means of evolution.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction Evolutionary Algorithms

Strongly Typed Genetic Programming

Strongly Typed Genetic Programming (STGP) I

Was proposed with the intention of addressing the “closure” limitation of the Genetic Programming technique.

I

Is particularly suited for representing method call sequences in strongly-typed languages such as Java, as it enables the reduction of the search space to the set of compilable sequences.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction Evolutionary Testing

Evolutionary Testing ET and SBTCG I

The application of evolutionary algorithms to test data generation is often referred to as Evolutionary Testing or Search-Based Test Case Generation.

I

The problem is finding a set of input data (test cases) that satisfies a certain test criterion.

I

The search-space is the input domain of the test object.

I

Evolutionary Testing is an emerging technology for automatically generating high quality test data.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Introduction Evolutionary Testing

Evolutionary Testing Example Test Case A a = new A(); B b = new B(); b.f(2); a.m(5, b);

I

Method Under Test: m

I

Test Cluster: A, B

I

Invocation of f on b aims at changing the state of b before passing it to m.

The State Problem I

Occurs with objects that exhibit state-like qualities by storing information in fields that are protected from external manipulation – encapsulation.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach

Outline Introduction Software Testing Evolutionary Algorithms Evolutionary Testing Technical Approach Overview Input Domain Reduction Test Case Evaluation Experimental Studies Test Case Evaluation Case Study Input Domain Reduction Case Study Conclusions and Future Work

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Overview

Our Approach

I

The focus of our on-going work is that of employing evolutionary algorithms for generating and evolving test cases for the structural unit-testing of third-party object-oriented Java programs.

I

The main goal is to develop an automated test case generation tool – eCrash.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Overview

Our Approach

I

Test cases are evolved using the Strongly-Typed Genetic Programming technique.

I

Test data quality evaluation includes instrumenting the test object, executing it with the generated test cases, and tracing the structures traversed in order to derive coverage metrics.

I

Static analysis and instrumentation is performed solely with basis on the information extracted from the test objects’ Java Bytecode.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Overview

Our Approach

I

The strategy for efficiently guiding the search process towards achieving full structural coverage involves favouring test cases that exercise problematic structures and control-flow paths.

I

Purity Analysis is employed as a means to automatically reduce the search space.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Overview

Test Case Representation

Figure: STGP tree and the corresponding Method Call Sequence.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Overview

Test Object Representation

Figure: Java Bytecode and corresponding Control-Flow Graph.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Overview

Methodology Overview foreach class under test do instrument for structural tracing; generate control-flow graphs; identify test cluster; generate EMCDGs and function sets; foreach method under test do repeat reevaluate weight of CFG nodes; generate individuals; foreach individual do generate method call sequence; generate test case; compile and execute test case; trace CFG nodes hit; evaluate test case; remove hits from remaining nodes list; recombine and mutate individuals; until stopping criteria is met ;

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Input Domain Reduction

Function Set Definition

I

Potential solutions (i.e. test cases) are encoded as STGP individuals and method call sequences as STGP trees.

I

Each tree subscribes to a function set, which defines the STGP nodes legally permitted in the tree.

I

For modelling call dependences an Extended Method Call Dependence Graph (EMCDG) is employed.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Input Domain Reduction

Purified EMCDG I

Our Input Domain Reduction strategy involves the removal of irrelevant edges from the EMCDG.

I

This is performed by analysing the purity of parameters with the aid of the Soot Framework.

I

Purity information is used to build the purified EMCDG.

Figure: EMCDG and purified EMCDG for the Stack class.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Input Domain Reduction

Purified Function Set

I

I

The purified Function Set is computed with basis on the purified EMCDG. Only those entries that are relevant to the search are included.

Member

Return Type

Stack() Object pop() Object pop() Object push(Object)

Stack [IP] Object [RE] Stack [IP] Object [RE]

Object push(Object)

Stack [IP]

Object peek() Object()

Object [RE] Object [RE]

Child Types Stack [IP] Stack [IP] Stack [IP] Object [P0] Stack [IP] Object [P0] Stack [IP]

Entries Excluded Object push(Object)

Object [P0]

Object peek() boolean empty() int search(Object)

Object [IP] Stack [IP] Stack [IP]

int search(Object)

Stack [P0]

Table: Function Set for the Stack class.

Stack [IP] Object [P0] Stack [IP] Stack [IP] Stack [IP] Object [P0] Stack [IP] Object [P0]

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Test Case Evaluation

Fitness Function Search Goal I

The primary goal is finding a set of test cases that achieves full structural coverage of the test object.

I

The quality of test cases is related to the structural entities of the method under test which are the current targets of the evolutionary search.

I

However, the execution of test cases may abort prematurely if a runtime exception is thrown during execution. When this happens, it is not possible to trace the structures traversed because the final instruction of the method call sequence is not reached.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Test Case Evaluation

Feasible and Unfeasible Test Cases I

Feasible Test Cases are effectively executed and terminate with a call to the method under test.

I

Unfeasible Test Cases abort prematurely because a runtime exception is thrown.

Example Unfeasible Test Case Stack stack0 = new Stack(); String string1 = “HelloWorld!”; int int2 = stack0.search(string1); Object object3 = stack0.pop(); ⇒ EmptyStackException Object object4 = stack0.pop(); Object object5 = stack0.peek();

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Test Case Evaluation

Weight Reevaluation

I

If unfeasible test cases are blindly penalised, the definition of elaborate state scenarios test cases will be disencouraged.

I

The issue of steering the search towards the traversal of interesting CFG nodes and paths was address by assigning weights to the CFG nodes.

I

The higher the weight of a given node the higher the cost of exercising it, and hence the higher the cost of transversing the corresponding control-flow path.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Test Case Evaluation

Weight Reevaluation Weight Reevaluation Wni = (αWni ) I



hitCni |T |

+1

 P

Wx x∈Nsni W Nsni × 2init

|



|

i.e., at the beginning of each generation the weight of a given node is multiplied by: I

I

I

the weight decrease constant value α, so as to decrease the weight of all CFG nodes indiscriminately; the hit count factor, which worsens the weight of recurrently hit CFG nodes; the path factor, which improves the weight of nodes that lead to interesting nodes and belong to interesting paths.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Technical Approach Test Case Evaluation

Test Case Evaluation Feasible Test Case Evaluation P

Fitnessfeasible (t) = I

h∈Ht

Wh

|Ht |

i.e., fitness := the average weight of the cfg nodes traversed.

Unfeasible Test Case Evaluation Fitnessunfeasible (t) = β + I

(seqLent −exIndt )×100 seqLent

i.e., fitness := percentage of instructions executed plus the unfeasible penalty constant β.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Experimental Studies

Outline Introduction Software Testing Evolutionary Algorithms Evolutionary Testing Technical Approach Overview Input Domain Reduction Test Case Evaluation Experimental Studies Test Case Evaluation Case Study Input Domain Reduction Case Study Conclusions and Future Work

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Experimental Studies Test Case Evaluation Case Study

Test Case Evaluation

Performed in order to: I

I

verify the adequateness of our test case evaluation strategy; experiment with different configurations for the I

I

probabilities of evolutionary operators – mutation, reproduction and crossover; values of the test case evaluation parameters – the weight decrease constant α and the unfeasible penalty constant β.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Experimental Studies Test Case Evaluation Case Study

Test Case Evaluation Results: I

Our strategy allows unfeasible test cases to be considered at certain points of the evolutionary search – once the feasible test cases that are being bred cease to be interesting because they only traverse recurrently hit CFG nodes.

I

A good compromise between the intensification and diversification of the search can be achieved, so as to favour the diversity and complexity of method call sequences – which are often needed to define elaborate state scenarios and enable the test set to achieve full structural coverage.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Experimental Studies Input Domain Reduction Case Study

Input Domain Reduction

Performed in order to: I

verify the impact of the Input Domain Reduction Strategy proposed in terms of: I I

the size of the input domain; the performance of our the eCrash tool;

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Experimental Studies Input Domain Reduction Case Study

Input Domain Reduction Results: I

The statistics show a clear reduction in the size of the input domain. I

I

The number of function set entries when purity analysis is used is, on average, 31.6% lower than that obtained when no purity analysis is employed.

The impact of this reduction on the test case generation process was also visible on the case studies performed on the using the eCrash tool. I

I

For the Stack class, the number of generations required to attain full coverage decreased 66%. For the BitSet class, the percentage of test cases that accomplished full coverage increased approximately 6%.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Conclusions and Future Work

Outline Introduction Software Testing Evolutionary Algorithms Evolutionary Testing Technical Approach Overview Input Domain Reduction Test Case Evaluation Experimental Studies Test Case Evaluation Case Study Input Domain Reduction Case Study Conclusions and Future Work

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Conclusions and Future Work

Conclusions I

Search-Based Test Case Generation is an emerging methodology for automatically generating quality test data.

I

With our approach, test cases are evolved using the STGP technique.

I

Purity Analysis is particularly useful in this context, as it provides a means to automatically discard Function Set entries that do not contribute to the definition of interesting state scenarios.

I

The set of method calls from which the algorithm can choose is trimmed to those that are relevant to the test case generation process.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Conclusions and Future Work

Conclusions

I

The state problem of OO programs requires the definition of carefully fine-tuned methodologies that promote the transversal of problematic structures must be promoted.

I

Complex method call sequences are often needed for traversing difficult control-flow paths.

I

If unfeasible test cases are blindly penalised, the definition of elaborate state scenarios test cases will be disencouraged.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Conclusions and Future Work

Conclusions

I

Our Test Case Evaluation strategy causes the fitness of feasible test cases that exercise recurrently traversed structures to fluctuate throughout the search process.

I

It allows unfeasible test cases to be considered at certain points of the evolutionary search – once the feasible test cases being bred cease to be interesting.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Conclusions and Future Work

Relevant Contributions I

I

We have outlined our evolutionary approach to the automated generation of test cases for the structural unit-testing of object-oriented Java programs. Our our-going work led to relevant contributions in the area, which include I

I

I

the introduction of novel methodologies for automation, search guidance and input domain reduction; the presentation of the eCrash tool.

The strategies proposed have been empirically evaluated with encouraging results.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Conclusions and Future Work

Future Work

I

The most promising research-related challenges that lie ahead of us seem to be the following: I

I

Input Domain Reduction - deals with removing irrelevant variables from a given test data generation problem, thereby reducing the size of the search space. Search Space Sampling - deals with the inclusion of all the relevant variables to a given test object into test data generation problem, so as to enable the coverage of the entire search space.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Appendix For Further Reading

For Further Reading I A. Kinneer, M. B. Dwyer, and G. Rothermel. Sofya: Supporting Rapid Development of Dynamic Program Analyses for Java. In ICSE COMPANION ’07: Companion to the Proceedings of the 29th International Conference on Software Engineering, pages 51–52, Washington, DC, USA, 2007. IEEE Computer Society. S. Luke. ECJ 16: A Java Evolutionary Computation Library. http://cs.gmu.edu/∼eclab/projects/ecj/, 2007. D. J. Montana. Strongly Typed Genetic Programming. Evolutionary Computation, 3(2):199–230, 1995. Raja Vallée-Rai, Phong Co, Etienne Gagnon, Laurie Hendren, Patrick Lam, and Vijay Sundaresan. Soot - a java bytecode optimization framework. In CASCON ’99: Proceedings of the 1999 conference of the Centre for Advanced Studies on Collaborative research, page 13. IBM Press, 1999. Alexandru Salcianu and Martin C. Rinard. Purity and side effect analysis for java programs. In VMCAI’05: Proceedings of the 6th International Conference on Verification, Model Checking, and Abstract Interpretation, volume 3385 of Lecture Notes in Computer Science, pages 199–215. Springer, 2005. S. Wappler and J. Wegener. Evolutionary Unit Testing of Object-Oriented Software Using Strongly-Typed gGnetic Programming. In GECCO ’06: Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pages 1925–1932, New York, NY, USA, 2006. ACM Press.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming Appendix For Further Reading

For Further Reading II

José Carlos Bregieiro Ribeiro, Mário Zenha-Rela, and Francisco Fernández de Vega. An evolutionary approach for performing structural unit-testing on third-party object-oriented java software. In Nature Inspired Cooperative Strategies for Optimization (NICSO 2007), volume 129/2008 of Studies in Computational Intelligence, pages 379–388. Springer Berlin / Heidelberg, 11 2007. José Carlos Bregieiro Ribeiro, Mário Zenha Rela, and Francisco Fernandéz de Vega. A strategy for evaluating feasible and unfeasible test cases for the evolutionary testing of object-oriented software. In AST ’08: Proceedings of the 3rd International Workshop on Automation of Software Test, pages 85–92, New York, NY, USA, 3 2008. ACM. José Carlos Bregieiro Ribeiro, Mário Zenha-Rela, and Francisco Fernández de Vega. Strongly-Typed Genetic Programming and Purity Analysis: Input Domain Reduction for Evolutionary Testing Problems. In GECCO’08: Proceedings of the Genetic and Evolutionary Computation Conference (to appear), 7 2008.

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming The End

Thank You!

José Carlos Bregieiro Ribeiro Polytechnic Institute of Leiria, Portugal

Phone: (+351) 965522037 Email: [email protected] Homepage: http://jcbribeiro.googlepages.com/

Search-Based Test Case Generation for Object ...

Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed ... half of the total costs involved in software development –.

732KB Sizes 0 Downloads 164 Views

Recommend Documents

Search-Based Test Case Generation for Object-Oriented Java ...
Jul 16, 2008 - Java Software Using Strongly-Typed Genetic Programming. José Carlos ..... ObjectVector of JDK 1.4.2), the statistics show a clear re- duction in ...

Search-Based Test Case Generation for Object-Oriented Java ...
tal principle of OO programming known as data encapsulation. The goal of the evolutionary search is to find MCS that define interesting state scenarios for the variables which will be passed, as arguments, in the call to the MUT. The input domain thu

Search-Based Test Case Generation for Object ...
Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic .... Test data quality evaluation includes instrumenting the.

Automated and Scalable T-wise Test Case Generation ...
of features F ⊆ N representing a valid product according to ... implemented mostly in JAVA (approximatively 2.3 KLOC) for. T-wise generation and metrics ...

Guided Test Generation for Web Applications
as business rules. A business rule is typically related to business calculations or access control policies. For example, banking applications have rules for ...

Model generation for robust object tracking based on ...
scription of the databases of the PASCAL object recogni- tion challenge). We try to overcome these drawbacks by proposing a novel, completely unsupervised ...

Metamodel-based Test Generation for Model ...
makes existing test techniques hard to use, especially for test data generation ... there are no precise stopping criteria to formally ensure that the transformation ...

Automatic, evolutionary test data generation for dynamic ... - CiteSeerX
Jan 18, 2008 - University of Cyprus, Department of Computer Science, 75 Kallipoleos .... their system (DifferentialGA) does not support efficiently ...... 365–381.

Test Case Evaluation and Input Domain Reduction Strategies for the ...
Mar 4, 2009 - Test Case Evaluation and Input Domain Reduction Strategies ... The application of Evolutionary Algorithms to test data gen- ...... clear(int,int).

Moral intuitions: A test case for evolutionary theories of human ...
irrational aspects of moral intuitions and opens a way to build a unitary theory of morality. Words: 9047 a Address correspondence: ... From an evolutionary point of view, there are two kinds of theories of cooperation: altruistic theories, for which

Test Case Prioritization and Test Suite Optimization ...
Abstract: Software Testing is an important activity in Software Development Life Cycle. (SDLC). Software testing is a process of executing a program or application with the intent of finding the bugs. Testing is expensive and prone to mistakes and ov

Case-Based Story Generation through Story ...
This paper explores a CBR approach to story generation based on the idea of story amalgamation, i.e. ... where L is a language, and ⊑ is a subsumption between the terms of the lan- guage L. We say that a term ψ1 .... Finally, an asymmetric amalgam

Natural Language Generation through Case-based ...
Overview of the CeBeTA architecture for case-based text modification. .... where SED(s, s ) is defined as a custom Sentence Edit Distance (SED)[12] func- tion.

Automatic Test Data Generation from Embedded C Code.pdf ...
There was a problem loading this page. Automatic Test Data Generation from Embedded C Code.pdf. Automatic Test Data Generation from Embedded C Code.

Utilizing S-TaLiRo as an Automatic Test Generation ... - public.asu.edu
more are being used in high-end modern automotive systems. Prototype autonomous ... C.E. Tuncali, T.P. Pavlic, and G. Fainekos are with School of Computing,.

Automatic Test Data Generation using Constraint Programming and ...
GOA. Goal Oriented Approach. IG-PR-IOOCC Instance Generator and Problem Representation to Improve Object. Oriented Code Coverage. IT. Information Technology. JPF. Java PathFinder. OOP. Object-Oriented Programming. POA. Path Oriented Approach. SB-STDG

Dynamic Generation of Test Cases with Metaheuristics.pdf ...
test data set in a suitable timeframe and with a. greater coverage than conventional methods. such as ... ultimately communicating, the best solution. that it has ...

Automatic Generation of Test-Cases Using Model ...
The methods and tools addressing model based test generation can be classified based on .... test generation meeting various structural and behavioral criteria.

Utilizing S-TaLiRo as an Automatic Test Generation ... - public.asu.edu
(e-mail:{etuncali, tpavlic, fainekos}@asu.edu) autonomous vehicle testing. Such frameworks would produce large number of tests generated in an intelligent way ...

Next Generation ACCUPLACER Study Guide: Test ...
money! - in introductory or remedial college courses. You want to accelerate your education, not ... are ready for the ACCUPLACER, but realize too late when they get their score back that they were not ready at all. They weren't incapable, and they c

ATGen: Automatic Test Data Generation using Constraint Logic ...
Page 1 of 11. ATGen: Automatic Test Data Generation using Constraint Logic. Programming and Symbolic Execution. Christophe Meudec. Computing, Physics & Mathematics Department. Institute of Technology, Carlow. Kilkenny Road. Carlow, Ireland. +353 (0)5