Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
m Crash: a Framework for the Evaluation of Mobile Devices' Trustworthiness Properties José Carlos Ribeiro1 1 Polytechnic 2 University
2
Mário Zenha-Rela
Institute of Leiria, Portugal
of Coimbra, Portugal (supervisor )
Master in Informatics and Systems
Departament of Informatics Engineering School of Sciences and Technology University of Coimbra, Portugal October 2008
1 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Outline 1
Introduction Mobile Devices and Dependability The State and Notications Broker
2
Framework and Methodology The
3
m Crash Automated Software Testing Tool
Experimental Study Targets and Methodology Results and Observations
4
Conclusions
5
On-Going Work Evolutionary Unit-Testing of Object-Oriented Software
2 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
Outline 1
Introduction Mobile Devices and Dependability The State and Notications Broker
2
Framework and Methodology The
3
m Crash Automated Software Testing Tool
Experimental Study Targets and Methodology Results and Observations
4
Conclusions
5
On-Going Work Evolutionary Unit-Testing of Object-Oriented Software
3 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
Mobile Devices The philosophy for mobile devices has been evolving towards the wallet paradigm: they contain important personal information, and virtually every adult carries one.
What's more, they function in a very uid environment, subject to frequent changes. However, not all mobile devices were designed with enterprise class security in mind.
Retrotting trust in any technology is considerably harder than building it in from the start especially when users have already perceived it as invasive, intrusive, or dangerous.
4 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
Dependability Dependability can be dened as the trustworthiness of a computing system. It is related not only to correct behaviour under normal circumstances, but also and most importantly to
reliability in the presence of errors. One of the major problems in dependability evaluation is the diculty of observing what happens inside the system that is submitted to stress.
Monitoring and proling of system behaviour under abnormal circumstances and, in particular, error propagation, can be greatly improved by built-in system support.
5 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
Black-Box Testing and Robustness Testing The key idea behind Black-Box Testing is that tests are
based on the values of parameters, and not on the implementation details. It is the preferred approach whenever the source code is
unavailable. Robustness Testing Robustness testing intends to detect opportunities for
obtaining faulty responses within the software being tested. Black-box testing is adequate to ascertain the robustness of the software.
6 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
The State and Notications Broker The Windows Mobile 5.0 operating system has centralized its state information into a single entity the State and
Notications Broker (SNB). The SNB is an API aimed, essentially, to the development of
context-aware applications. Context-Awareness
Context-awareness focuses on creating information services that deliver the right information at the right time, in
the right place, in the right way, to the right person . The SNB makes the system transparent enough to allow for a semantically-oriented monitoring of relevant state-variables. 7 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
The State and Notications Broker The SNB provides a standard architecture for monitoring
state values for changes and for distributing change notications to the interested parties using a publish-subscribe model. It renders it unnecessary to hunt down a separate function or API for each individual state value. Base State and Notication Properties The base State and Notication Properties encompass information on the system state, phone, user, tasks and
appointments, connections, messages, media player and time. 8 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Mobile Devices and Dependability The State and Notications Broker
The State and Notications Broker Our initial idea was that of employing the SNB a toolkit primarily aimed at context-aware applications to take a
step back , and monitor the system as a whole during the fault injection campaign. In fact, the experiments described in the following sections
show that the SNB built-in services provide a means to for keeping an eye on undesirable state value modications. Also and, perhaps, most importantly it allowed
uncovering errors before they were unveiled by the runtime environment, and detecting the contamination of objects unrelated with those directly tempered with. 9 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
The
m Crash Automated Software Testing Tool
Outline 1
Introduction Mobile Devices and Dependability The State and Notications Broker
2
Framework and Methodology The
3
m Crash Automated Software Testing Tool
Experimental Study Targets and Methodology Results and Observations
4
Conclusions
5
On-Going Work Evolutionary Unit-Testing of Object-Oriented Software
10 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
The
m Crash Automated Software Testing Tool
Overview m Crash In order to access the usefulness of the SNB, we have developed a prototype automated software testing tool
m Crash.
Four fundamental modules m Crash tool: embody the
the Faultload Database; the Input Generation and Fault Injection Module; the Postcondition Checker; and the Execution Manager. 11 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
The
m Crash Automated Software Testing Tool
Faultload Database All the MUT's information is
including public properties, their data types and error codes. cataloged
The rationale for
focusing on the
public properties
is related with the
extended insight that the SNB allows
.
A domain analysis is performed for each individual data type in order to establish the faultload. Valid, boundary and invalid test values are dened for each of the data types.
12 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
The
m Crash Automated Software Testing Tool
Input Generation and Fault Injection Module The
Input Generation
component
dynamically generates test cases
a given set of properties.
for
The methodology followed is that of performing fault-injection by changing the target public
.
properties' values
is employed to and to set the minimum amount of state needed for each individual test case. Manual stubbing
instantiate objects
The Fault Injection component automatically executes the test cases, and collects the information returned by a particular test case.
13 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
The
m Crash Automated Software Testing Tool
Postcondition Checker The Postcondition Checker
monitors
the environment for unacceptable
. Assertions are put in two main places: at the system level and at the output. events
At the system level,
global
environmental events are tracked using the SNB
.
At the output level,
the tool checks
the properties' values and the exceptions thrown
.
14 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
The
m Crash Automated Software Testing Tool
Execution Manager Responsible managing the remaining modules and for presenting the results to the user. The logs generated by the Postcondition Checker are automatically compared to the previously recorded gold run. The
exceptions
annotated
.
thrown are
The property values assumed after the fault injection process are compared to those expected. 15 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Outline 1
Introduction Mobile Devices and Dependability The State and Notications Broker
2
Framework and Methodology The
3
m Crash Automated Software Testing Tool
Experimental Study Targets and Methodology Results and Observations
4
Conclusions
5
On-Going Work Evolutionary Unit-Testing of Object-Oriented Software
16 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Targets
The
m Crash tool was employed to conduct a software testing
campaign. The targets of this experiment were the public properties of the
Microsoft.WindowsMobile.PocketOutlook
namespace.
During the Faultload Database building process, 9 distinct
classes, including 96 distinct public properties, were identied and catalogued.
17 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Exception Latency The exceptions thrown during the software testing campaign were categorized according to their latency. Immediate Exceptions and Late Exceptions
Immediate Exceptions the exception is thrown during the process of assigning an erroneous value to a property (i.e., the assertion is located in the property's setter method);
Late Exceptions the exception is thrown by the method that receives the object containing the faulty property as an input parameter (i.e., the assertion is located in the method called).
18 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Exception Latency Late exceptions are more problematic, due to the high probability of error propagation.
Objects containing faulty properties could linger in the system until they are used as an input parameter and the exception is triggered. What's more, the analysis of the exceptions' data does not
allow the typication the data types according to the exception category. There is no coherent behaviour or pattern. Similar invalid test values generate both immediate and late exceptions, which can only be explained by the API's
internal structure. 19 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Latent Error Detection
We decided to devote special attention to the detection
of latent errors, by continuously monitoring property values for changes by means of the SNB. This methodology allowed: identifying incoherent error handling situations; detecting fault propagation; pinpointing faults before the runtime system.
20 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Incoherent Error Handling We observed that when the
Appointment.Start
property was
set to a value below the allowed range, an immediate exception was thrown. Nevertheless, the Postcondition Checker received a notication of the property being set to its lower bound i.e., the property value was changed even though an exception was thrown. What's more, in a similar situation when the
Appointment.Start
property was set to a value above its
upper bound an immediate exception of the was thrown, and the property kept its previous value.
This irregular behaviour requires distinct handling of similar situations. 21 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Fault Propagation Other anomalous behaviour observed using the SNB included receiving notications of changes to properties other than those directly disturbed:
when the Appointment.Start property was set to an invalid value, the Appointment.End property was set to its default value; when the Appointment.End property was set to an invalid value the Appointment.Start property was set to its default value. This example provides a clear sign that the postcondition checking procedure's eectiveness is increased if the system is monitored as a whole. 22 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Targets and Methodology Results and Observations
Early Fault Detection In some circumstances, it was possible to detect the contamination of objects before the errors were unveiled by the runtime environment. In the
Appointment.Subject
property, the String with 4096
characters boundary test case generated a late exception when the object was used as an argument in a method call. Nevertheless, by means of the SNB, it was possible to observe that this property assumed a
null
value immediately after the
erroneous value was assigned to the property.
It must be stressed that this anomalous behaviour was unveiled by the SNB (by publishing a notication of the property change) before the runtime environment threw an exception. 23 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Outline 1
Introduction Mobile Devices and Dependability The State and Notications Broker
2
Framework and Methodology The
3
m Crash Automated Software Testing Tool
Experimental Study Targets and Methodology Results and Observations
4
Conclusions
5
On-Going Work Evolutionary Unit-Testing of Object-Oriented Software
24 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Conclusions
We proposed employing the State and Notications Broker API for error monitoring and propagation proling. The SNB centralizes system state information in documented locations, and distributes change notications to interested parties using a publish-subscribe model.
It provides built-in monitoring services to internal system variables, which constitutes a means for keeping an eye on undesirable state value modications.
25 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Conclusions The experiments performed showed:
that system built-in assertions are sparsely distributed and less than thoroughly documented; that errors can remain dormant in the system until they are detected and dealt with e.g., with an exception.
Interesting observations made possible by the SNB receiving notications of changes to properties other than
those disturbed; receiving notication of a property being changed, even
though an exception was immediately thrown after an invalid value was assigned to it; receiving notication of invalid values being assigned to a
property; an exception was only triggered when the faulty property's instance was used as an argument in a method call.
26 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Conclusions These results indicate that the SNB is particularly useful for detecting erroneous internal states. By continuously monitoring the system as a whole, it allows the detection of faults before the runtime system, detecting the propagation of errors, and pinpointing irregular error handling behaviors. Future Work Our work so far was limited to the base State and Notication Properties dened by default; nevertheless, these are clearly insucient to cover the system as a whole. Future work includes extending the set of properties exposed, with the purpose of broadening the range of relevant system variables being monitored.
27 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
Outline 1
Introduction Mobile Devices and Dependability The State and Notications Broker
2
Framework and Methodology The
3
m Crash Automated Software Testing Tool
Experimental Study Targets and Methodology Results and Observations
4
Conclusions
5
On-Going Work Evolutionary Unit-Testing of Object-Oriented Software
28 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
The Focus of Our On-Going Work Automating the test data generation process is vital to advance the state-of-the-art in software testing.
Evolutionary Unit-Testing of Object-Oriented Software The focus of our on-going work is that of employing evolutionary
algorithms for generating and evolving test cases for the structural unit-testing of third-party object-oriented Java programs.
29 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
Evolutionary Algorithms and Software Testing Evolutionary Algorithms use simulated evolution as a search strategy to evolve candidate solutions, using operators inspired by genetics and natural selection.
Genetic Programming is a machine-learning approach usually associated with the evolution of tree structures. It focuses on automatically creating computer programs by means of evolution. Evolutionary Testing The application of evolutionary algorithms to test data generation is often referred to as Evolutionary Testing. The problem is nding a set of input data (test cases) that satises a certain test criterion. The search-space is the input domain of the test object.
30 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
Approach and Contributions Approach Test cases are evolved using the Strongly-Typed Genetic
Programming technique. Test data quality evaluation includes instrumenting the test object, executing it with the test cases, and tracing the structures traversed so as to derive coverage metrics. The strategy for guiding the search process towards achieving
full structural coverage involves favouring test cases that exercise problematic structures and control-ow paths. Relevant Contributions The introduction of novel methodologies for automation, search guidance and input domain reduction; the presentation of the
eCrash
tool.
31 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
Test Case Representation
Figure: STGP tree and the corresponding Method Call Sequence.
32 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
Test Object Representation
Figure: Java Bytecode and corresponding Control-Flow Graph.
33 / 34
Introduction Framework and Methodology Experimental Study Conclusions On-Going Work
Evolutionary Unit-Testing of Object-Oriented Software
Methodology Overview foreach class under test do instrument for structural tracing; generate control-ow graphs; identify test cluster; generate EMCDGs and function sets; foreach method under test do repeat reevaluate weight of CFG nodes; generate individuals; foreach individual do generate method call sequence; generate test case; compile and execute test case; trace CFG nodes hit; evaluate test case; remove hits from remaining nodes list; recombine and mutate individuals; until stopping criteria is met ; 34 / 34
Appendix The End
Peer-Reviewed Publications
Peer-Reviewed Publications I Ribeiro, J. and Luis, B.M. and Rela, M. Z. Error propagation monitoring on windows mobile-based devices. In Proc. of the LADC 2007, pp. 111-122, Vol. 4746/2007, Lecture
Notes in Computer Science,
Latin-American Symposium on Dependable Computing, Morelia, Mexico, September 2007.
Ribeiro, J. and Rela, M. Z. mCrash: a Framework for the Evaluation of Mobile Devices Trustworthiness Properties. In Proc. of the CSMU 2006, pp. 163-166, Conference on Mobile and Ubiquitous System, Guimarães, Portugal, June 2006. ISBN: 972-8692-29-3.
Ribeiro, J. Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming. In Proc. of the GECCO '08, pp. 1819-1822, Graduate Student Workshop, Atlanta, Georgia, USA, July 2008.
Ribeiro, J. and Rela, M. Z. and Vega, F.F. Strongly-Typed Genetic Programming and Purity Analysis: Input Domain Reduction for Evolutionary Testing Problems. In Proc. of the GECCO '08, pp. 1783-1784, 10th Annual Conference on Genetic and Evolutionary Computation, Atlanta, Georgia, USA, July 2008.
35 / 34
Appendix The End
Peer-Reviewed Publications
Peer-Reviewed Publications II Ribeiro, J. and Rela, M. Z. and Vega, F.F. A strategy for evaluating feasible and unfeasible test cases for the evolutionary testing of object-oriented software. In Proc. of the 3rd International Workshop on Automation of Software Test (AST'08), pp.
85-92,
30th International Conference on Software Engineering (ICSE'08), Leipzig, Germany, May 2008.
Ribeiro, J. and Rela, M. Z. and Vega, F.F. An Evolutionary Approach For Performing Structural Unit-Testing On Third-Party Object-Oriented Java Software. In Proc. of the NICSO 2007, pp. 379-388, Vol. 129/2008, Studies in Computational
Intelligence,
International Workshop on Nature Inspired Cooperative Strategies for Optimization, Acireale, Italy, November 2007.
Ribeiro, J. and Rela, M. Z. and Vega, F.F. eCrash: a Framework for Performing Evolutionary Testing on Third-Party Java Components. In Proc. of the I Jornadas sobre Algoritmos Evolutivos y Metaheuristicas (JAEM), pp. 137-144,
II
Congreso Español de Informática (CEDI), Zaragoza, Spain, September 2007. ISBN: 978-84-9732-593-6.
Ribeiro, J. and Vega, F.F. and Rela, M. Z. Using Dynamic Analysis of Java Bytecode for Evolutionary Object-Oriented Unit Testing. In Proc. of the 8th Workshop on Testing and Fault Tolerance (WTF), pp. 143-156, 25th Brazilian Symposium on Computer Networks and Distributed Systems (SBRC), Belém, Brazil, May 2007. ISBN: 85-766-0119-1.
36 / 34
Appendix The End
Thank You!
José Carlos Bregieiro Ribeiro Phone: Email:
(+351) 965522037
[email protected]
Homepage:
http://jcbribeiro.googlepages.com/
37 / 34