Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

m Crash: a Framework for the Evaluation of Mobile Devices' Trustworthiness Properties José Carlos Ribeiro1 1 Polytechnic 2 University

2

Mário Zenha-Rela

Institute of Leiria, Portugal

of Coimbra, Portugal (supervisor )

Master in Informatics and Systems

Departament of Informatics Engineering School of Sciences and Technology University of Coimbra, Portugal October 2008

1 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Outline 1

Introduction Mobile Devices and Dependability The State and Notications Broker

2

Framework and Methodology The

3

m Crash Automated Software Testing Tool

Experimental Study Targets and Methodology Results and Observations

4

Conclusions

5

On-Going Work Evolutionary Unit-Testing of Object-Oriented Software

2 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

Outline 1

Introduction Mobile Devices and Dependability The State and Notications Broker

2

Framework and Methodology The

3

m Crash Automated Software Testing Tool

Experimental Study Targets and Methodology Results and Observations

4

Conclusions

5

On-Going Work Evolutionary Unit-Testing of Object-Oriented Software

3 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

Mobile Devices The philosophy for mobile devices has been evolving towards the wallet paradigm: they contain important personal information, and virtually every adult carries one.

What's more, they function in a very uid environment, subject to frequent changes. However, not all mobile devices were designed with enterprise class security in mind.

Retrotting trust in any technology is considerably harder than building it in from the start  especially when users have already perceived it as invasive, intrusive, or dangerous.

4 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

Dependability Dependability can be dened as the trustworthiness of a computing system. It is related not only to correct behaviour under normal circumstances, but also  and most importantly  to

reliability in the presence of errors. One of the major problems in dependability evaluation is the diculty of observing what happens inside the system that is submitted to stress.

Monitoring and proling of system behaviour under abnormal circumstances and, in particular, error propagation, can be greatly improved by built-in system support.

5 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

Black-Box Testing and Robustness Testing The key idea behind Black-Box Testing is that tests are

based on the values of parameters, and not on the implementation details. It is the preferred approach whenever the source code is

unavailable. Robustness Testing Robustness testing intends to detect opportunities for

obtaining faulty responses within the software being tested. Black-box testing is adequate to ascertain the robustness of the software.

6 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

The State and Notications Broker The Windows Mobile 5.0 operating system has centralized its state information into a single entity  the State and

Notications Broker (SNB). The SNB is an API aimed, essentially, to the development of

context-aware applications. Context-Awareness

Context-awareness focuses on creating information services that deliver the right information at the right time, in

the right place, in the right way, to the right person . The SNB makes the system transparent enough to allow for a semantically-oriented monitoring of relevant state-variables. 7 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

The State and Notications Broker The SNB provides a standard architecture for monitoring

state values for changes and for distributing change notications to the interested parties using a publish-subscribe model. It renders it unnecessary to hunt down a separate function or API for each individual state value. Base State and Notication Properties The base State and Notication Properties encompass information on the system state, phone, user, tasks and

appointments, connections, messages, media player and time. 8 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Mobile Devices and Dependability The State and Notications Broker

The State and Notications Broker Our initial idea was that of employing the SNB  a toolkit primarily aimed at context-aware applications  to take a

step back , and monitor the system as a whole during the fault injection campaign. In fact, the experiments described in the following sections

show that the SNB built-in services provide a means to for keeping an eye on undesirable state value modications. Also  and, perhaps, most importantly  it allowed

uncovering errors before they were unveiled by the runtime environment, and detecting the contamination of objects unrelated with those directly tempered with. 9 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

The

m Crash Automated Software Testing Tool

Outline 1

Introduction Mobile Devices and Dependability The State and Notications Broker

2

Framework and Methodology The

3

m Crash Automated Software Testing Tool

Experimental Study Targets and Methodology Results and Observations

4

Conclusions

5

On-Going Work Evolutionary Unit-Testing of Object-Oriented Software

10 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

The

m Crash Automated Software Testing Tool

Overview m Crash In order to access the usefulness of the SNB, we have developed a prototype automated software testing tool 

m Crash.

Four fundamental modules m Crash tool: embody the

the Faultload Database; the Input Generation and Fault Injection Module; the Postcondition Checker; and the Execution Manager. 11 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

The

m Crash Automated Software Testing Tool

Faultload Database All the MUT's information is

 including public properties, their data types and error codes. cataloged

The rationale for

focusing on the

public properties

is related with the

extended insight that the SNB allows

.

A domain analysis is performed for each individual data type in order to establish the faultload. Valid, boundary and invalid test values are dened for each of the data types.

12 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

The

m Crash Automated Software Testing Tool

Input Generation and Fault Injection Module The

Input Generation

component

dynamically generates test cases

a given set of properties.

for

The methodology followed is that of performing fault-injection by changing the target public

.

properties' values

is employed to and to set the minimum amount of state needed for each individual test case. Manual stubbing

instantiate objects

The Fault Injection component automatically executes the test cases, and collects the information returned by a particular test case.

13 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

The

m Crash Automated Software Testing Tool

Postcondition Checker The Postcondition Checker

monitors

the environment for unacceptable

. Assertions are put in two main places: at the system level and at the output. events

At the system level,

global

environmental events are tracked using the SNB

.

At the output level,

the tool checks

the properties' values and the exceptions thrown

.

14 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

The

m Crash Automated Software Testing Tool

Execution Manager Responsible managing the remaining modules and for presenting the results to the user. The logs generated by the Postcondition Checker are automatically compared to the previously recorded gold run. The

exceptions

annotated

.

thrown are

The property values assumed after the fault injection process are compared to those expected. 15 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Outline 1

Introduction Mobile Devices and Dependability The State and Notications Broker

2

Framework and Methodology The

3

m Crash Automated Software Testing Tool

Experimental Study Targets and Methodology Results and Observations

4

Conclusions

5

On-Going Work Evolutionary Unit-Testing of Object-Oriented Software

16 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Targets

The

m Crash tool was employed to conduct a software testing

campaign. The targets of this experiment were the public properties of the

Microsoft.WindowsMobile.PocketOutlook

namespace.

During the Faultload Database building process, 9 distinct

classes, including 96 distinct public properties, were identied and catalogued.

17 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Exception Latency The exceptions thrown during the software testing campaign were categorized according to their latency. Immediate Exceptions and Late Exceptions

Immediate Exceptions  the exception is thrown during the process of assigning an erroneous value to a property (i.e., the assertion is located in the property's setter method);

Late Exceptions  the exception is thrown by the method that receives the object containing the faulty property as an input parameter (i.e., the assertion is located in the method called).

18 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Exception Latency Late exceptions are more problematic, due to the high probability of error propagation.

Objects containing faulty properties could linger in the system  until they are used as an input parameter and the exception is triggered. What's more, the analysis of the exceptions' data does not

allow the typication the data types according to the exception category. There is no coherent behaviour or pattern. Similar invalid test values generate both immediate and late exceptions, which can only be explained by the API's

internal structure. 19 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Latent Error Detection

We decided to devote special attention to the detection

of latent errors, by continuously monitoring property values for changes by means of the SNB. This methodology allowed: identifying incoherent error handling situations; detecting fault propagation; pinpointing faults before the runtime system.

20 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Incoherent Error Handling We observed that when the

Appointment.Start

property was

set to a value below the allowed range, an immediate exception was thrown. Nevertheless, the Postcondition Checker received a notication of the property being set to its lower bound  i.e., the property value was changed even though an exception was thrown. What's more, in a similar situation  when the

Appointment.Start

property was set to a value above its

upper bound  an immediate exception of the was thrown, and the property kept its previous value.

This irregular behaviour requires distinct handling of similar situations. 21 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Fault Propagation Other anomalous behaviour observed using the SNB included receiving notications of changes to properties other than those directly disturbed:

when the Appointment.Start property was set to an invalid value, the Appointment.End property was set to its default value; when the Appointment.End property was set to an invalid value the Appointment.Start property was set to its default value. This example provides a clear sign that the postcondition checking procedure's eectiveness is increased if the system is monitored as a whole. 22 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Targets and Methodology Results and Observations

Early Fault Detection In some circumstances, it was possible to detect the contamination of objects before the errors were unveiled by the runtime environment. In the

Appointment.Subject

property, the String with 4096

characters boundary test case generated a late exception when the object was used as an argument in a method call. Nevertheless, by means of the SNB, it was possible to observe that this property assumed a

null

value immediately after the

erroneous value was assigned to the property.

It must be stressed that this anomalous behaviour was unveiled by the SNB (by publishing a notication of the property change) before the runtime environment threw an exception. 23 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Outline 1

Introduction Mobile Devices and Dependability The State and Notications Broker

2

Framework and Methodology The

3

m Crash Automated Software Testing Tool

Experimental Study Targets and Methodology Results and Observations

4

Conclusions

5

On-Going Work Evolutionary Unit-Testing of Object-Oriented Software

24 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Conclusions

We proposed employing the State and Notications Broker API for error monitoring and propagation proling. The SNB centralizes system state information in documented locations, and distributes change notications to interested parties using a publish-subscribe model.

It provides built-in monitoring services to internal system variables, which constitutes a means for keeping an eye on undesirable state value modications.

25 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Conclusions The experiments performed showed:

that system built-in assertions are sparsely distributed and less than thoroughly documented; that errors can remain dormant in the system until they are detected and dealt with  e.g., with an exception.

Interesting observations made possible by the SNB receiving notications of changes to properties other than

those disturbed; receiving notication of a property being changed, even

though an exception was immediately thrown after an invalid value was assigned to it; receiving notication of invalid values being assigned to a

property; an exception was only triggered when the faulty property's instance was used as an argument in a method call.

26 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Conclusions These results indicate that the SNB is particularly useful for detecting erroneous internal states. By continuously monitoring the system as a whole, it allows the detection of faults before the runtime system, detecting the propagation of errors, and pinpointing irregular error handling behaviors. Future Work Our work so far was limited to the base State and Notication Properties dened by default; nevertheless, these are clearly insucient to cover the system as a whole. Future work includes extending the set of properties exposed, with the purpose of broadening the range of relevant system variables being monitored.

27 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

Outline 1

Introduction Mobile Devices and Dependability The State and Notications Broker

2

Framework and Methodology The

3

m Crash Automated Software Testing Tool

Experimental Study Targets and Methodology Results and Observations

4

Conclusions

5

On-Going Work Evolutionary Unit-Testing of Object-Oriented Software

28 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

The Focus of Our On-Going Work Automating the test data generation process is vital to advance the state-of-the-art in software testing.

Evolutionary Unit-Testing of Object-Oriented Software The focus of our on-going work is that of employing evolutionary

algorithms for generating and evolving test cases for the structural unit-testing of third-party object-oriented Java programs.

29 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

Evolutionary Algorithms and Software Testing Evolutionary Algorithms use simulated evolution as a search strategy to evolve candidate solutions, using operators inspired by genetics and natural selection.

Genetic Programming is a machine-learning approach usually associated with the evolution of tree structures. It focuses on automatically creating computer programs by means of evolution. Evolutionary Testing The application of evolutionary algorithms to test data generation is often referred to as Evolutionary Testing. The problem is nding a set of input data (test cases) that satises a certain test criterion. The search-space is the input domain of the test object.

30 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

Approach and Contributions Approach Test cases are evolved using the Strongly-Typed Genetic

Programming technique. Test data quality evaluation includes instrumenting the test object, executing it with the test cases, and tracing the structures traversed so as to derive coverage metrics. The strategy for guiding the search process towards achieving

full structural coverage involves favouring test cases that exercise problematic structures and control-ow paths. Relevant Contributions The introduction of novel methodologies for automation, search guidance and input domain reduction; the presentation of the

eCrash

tool.

31 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

Test Case Representation

Figure: STGP tree and the corresponding Method Call Sequence.

32 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

Test Object Representation

Figure: Java Bytecode and corresponding Control-Flow Graph.

33 / 34

Introduction Framework and Methodology Experimental Study Conclusions On-Going Work

Evolutionary Unit-Testing of Object-Oriented Software

Methodology Overview foreach class under test do instrument for structural tracing; generate control-ow graphs; identify test cluster; generate EMCDGs and function sets; foreach method under test do repeat reevaluate weight of CFG nodes; generate individuals; foreach individual do generate method call sequence; generate test case; compile and execute test case; trace CFG nodes hit; evaluate test case; remove hits from remaining nodes list; recombine and mutate individuals; until stopping criteria is met ; 34 / 34

Appendix The End

Peer-Reviewed Publications

Peer-Reviewed Publications I Ribeiro, J. and Luis, B.M. and Rela, M. Z. Error propagation monitoring on windows mobile-based devices. In Proc. of the LADC 2007, pp. 111-122, Vol. 4746/2007, Lecture

Notes in Computer Science,

Latin-American Symposium on Dependable Computing, Morelia, Mexico, September 2007.

Ribeiro, J. and Rela, M. Z. mCrash: a Framework for the Evaluation of Mobile Devices Trustworthiness Properties. In Proc. of the CSMU 2006, pp. 163-166, Conference on Mobile and Ubiquitous System, Guimarães, Portugal, June 2006. ISBN: 972-8692-29-3.

Ribeiro, J. Search-Based Test Case Generation for Object-Oriented Java Software Using Strongly-Typed Genetic Programming. In Proc. of the GECCO '08, pp. 1819-1822, Graduate Student Workshop, Atlanta, Georgia, USA, July 2008.

Ribeiro, J. and Rela, M. Z. and Vega, F.F. Strongly-Typed Genetic Programming and Purity Analysis: Input Domain Reduction for Evolutionary Testing Problems. In Proc. of the GECCO '08, pp. 1783-1784, 10th Annual Conference on Genetic and Evolutionary Computation, Atlanta, Georgia, USA, July 2008.

35 / 34

Appendix The End

Peer-Reviewed Publications

Peer-Reviewed Publications II Ribeiro, J. and Rela, M. Z. and Vega, F.F. A strategy for evaluating feasible and unfeasible test cases for the evolutionary testing of object-oriented software. In Proc. of the 3rd International Workshop on Automation of Software Test (AST'08), pp.

85-92,

30th International Conference on Software Engineering (ICSE'08), Leipzig, Germany, May 2008.

Ribeiro, J. and Rela, M. Z. and Vega, F.F. An Evolutionary Approach For Performing Structural Unit-Testing On Third-Party Object-Oriented Java Software. In Proc. of the NICSO 2007, pp. 379-388, Vol. 129/2008, Studies in Computational

Intelligence,

International Workshop on Nature Inspired Cooperative Strategies for Optimization, Acireale, Italy, November 2007.

Ribeiro, J. and Rela, M. Z. and Vega, F.F. eCrash: a Framework for Performing Evolutionary Testing on Third-Party Java Components. In Proc. of the I Jornadas sobre Algoritmos Evolutivos y Metaheuristicas (JAEM), pp. 137-144,

II

Congreso Español de Informática (CEDI), Zaragoza, Spain, September 2007. ISBN: 978-84-9732-593-6.

Ribeiro, J. and Vega, F.F. and Rela, M. Z. Using Dynamic Analysis of Java Bytecode for Evolutionary Object-Oriented Unit Testing. In Proc. of the 8th Workshop on Testing and Fault Tolerance (WTF), pp. 143-156, 25th Brazilian Symposium on Computer Networks and Distributed Systems (SBRC), Belém, Brazil, May 2007. ISBN: 85-766-0119-1.

36 / 34

Appendix The End

Thank You!

José Carlos Bregieiro Ribeiro Phone: Email:

(+351) 965522037 [email protected]

Homepage:

http://jcbribeiro.googlepages.com/

37 / 34

mCrash: a Framework for the Evaluation of Mobile ...

The SNB provides a standard architecture for monitoring ..... eCrash: a Framework for Performing Evolutionary Testing on Third-Party Java Components. In Proc.

1MB Sizes 1 Downloads 250 Views

Recommend Documents

mCrash: a Framework for the Evaluation of Mobile ...
Mobile Devices. The philosophy for mobile devices has been evolving towards ..... when the object was used as an argument in a method call. Nevertheless .... 1783-1784, 10th Annual Conference on Genetic and Evolutionary. Computation ...

Performance Evaluation of a QoS-Aware Framework for ...
good response times, by selecting different replicas to service different clients concurrently ... and responsiveness of the replicas by monitoring them at run- time.

A Java Framework for Mobile Data Synchronization
file systems, availability is more important than serializability. .... accumulate a list of newly inserted objects, and listen for completion of the receiving phase to ...

ePub The Framework for Teaching Evaluation ...
Standards, with clarity of language for ease of use and scoring Read Full Online. Books detail. Title : ePub The Framework for Teaching q. Evaluation Instrument ...

evaluation of future mobile services based on the ... - Semantic Scholar
However, most of the mobile data services have fallen short of the expectation and have ..... Journal of the Academy of Marketing Science,. 33 (3), pp. 330-346.

LowMIPv6: A Framework for Mobile IPv6 for the Low ...
Abstract. Low capable device such as Wireless Sensor ... IP connectivity to WSNs has enabled ubiquity of .... Internet connectivity to all mobile network nodes.

evaluation of future mobile services based on the technology ...
Keywords: Mobile Broadcast Services, Technology Acceptance Model ... The proposed TAM model was widely accepted by the information system research ..... Antecedents and Cross-Service Comparisons, Journal of the Academy of ...

A Proposed Framework for Proposed Framework for ...
approach helps to predict QoS ranking of a set of cloud services. ...... Guarantee in Cloud Systems” International Journal of Grid and Distributed Computing Vol.3 ...

A Framework for Identifying the Sources of Local ...
with an Empirical Application* .... Such factors may include the small costs of re-pricing (the so- .... wholesale level, accounting for 10.5% on average for the incomplete pass-through. .... School of Business and include aggregate retail volume mar

A Framework for Developing the Structure of Public Health Economic ...
placed on these approaches for health care decision making [4], methods for the .... the methods described in the articles were identified using a data extraction ...

A conceptual framework for the integration of learning ...
Test LT in situ. • Students using the LT. Monitor and adapt the integration. • Continuous “integrative evaluation”. • Adapt the LT and the REST of the course “system”. Evaluation of implementation ..... operates, but whether it does so

A framework for the study of zoonotic disease ...
ever to use our research to develop long-term program- mes and strategies that reduce the ...... diversity (e.g. IUCN, IPBES, CBD, CMS, TEEB) and bats (e.g. Bat ...

Optimized Lightweight Thread Framework for Mobile Devices ...
are two software approaches to adjust the stack size: • Changing system ..... understanding the detailed interactions among threads to ana- lyze the purpose of ...

A model-based framework for the detection of ...
algorithm are presented in the form of free-response receiver operating characteristic .... The Hough domain of each ROI was thresholded to detect ... of-Gaussian filters to identify and create a likelihood map of ..... the authors' website.44. II.

A Framework for Developing the Structure of Public Health Economic ...
presented. Detailed process suggestions and an example to illustrate ... the diabetes example are drawn upon throughout the article. ... heterogeneous changes.

Adv for Bid - Mobile - City of Mobile
Sep 2, 2015 - All bidders bidding in amounts exceeding that established by the State Licensing Board for. General Contractors must be properly licensed ...

Performance Evaluation of Mobile IPv6 Wireless Test ...
agement inside the domain is handled by a Mobility Anchor Point (MAP), which acts ... wills forward the multicast packets in a similar way to unicast packets.

A Framework for Visual Characterization of Number ...
by developers to create applications based on number theoretic problems. .... control which has been developed as a component in the windows control library.

NetTopo: A Framework of Simulation and Visualization for ... - CiteSeerX
Oct 30, 2008 - networks (WSNs) applications drive the fast development of research in various .... e.g., energy consumption, bandwidth management. NetTopo.

A variational framework for spatio-temporal smoothing of fluid ... - Irisa
discontinuities. Vorticity-velocity scheme To deal with the advective term, we use the fol- lowing semidiscrete central scheme [13, 14]:. ∂tξi,j = −. Hx i+ 1. 2 ,j (t) − Hx i− 1. 2 ,j (t). ∆x. −. Hy i,j+ 1. 2(t) − Hy i,j− 1. 2. (t).