Early Automated Verification of Tool Chain Design Matthias Biehl Embedded Control Systems Royal Institute of Technology Stockholm, Sweden [email protected]

Abstract. Tool chains are expected to increase the productivity of product development by providing automation and integration. If, however, the tool chain does not have the features required to support the product development process, it falls short of this expectation. Tool chains could reach their full potential if it could be ensured that the features of a tool chain are aligned with the product development process. As part of a systematic development approach for tool chains, we propose a verification method that measures the extent to which a tool chain design conforms to the product development process and identifies misalignments. The verification method can be used early in tool chain development, when it is relatively easy and cheap to perform the necessary corrections. Our verification method is automated, which allows for quick feedback and enables iterative design. We apply the proposed method on an industrial tool chain, where it is able to identify improvements to the design of the tool chain. Key words: Tool Integration; Process Modeling; Verification; Modeldriven Development

1

Introduction

Tool chains are becoming increasingly important in the development of complex systems, since tool chains have the potential to increase development productivity. In order to reach the goal of higher productivity, tool chains need to support the product development process. The development process of each product is different, specific to the company, its traditions, the preferences of the engineers and the set of development tools used. To provide support in development, a tool chain cannot be one-size-fits-all solution, but it needs to be customized to the development context it is used in [7]. It is especially important that the tool chain is customized and aligned with the product development process, otherwise practitioners do not accept a new tool chain in their work [5]. It is thus important to verify that a tool chain is suitable and fits in a specific development context. We verify by checking the degree to which a tool chain is aligned with a given product development process.

This work is part of a vision to support the development of a tool chain using a systematic methodology and tools that manage the complexity of tool chain development [3]. As part of the systematic development methodology, the design of a customized tool chain needs to be analyzed and verified early in the tool chain development process1 . When verifying the design of a tool chain, we are interested in checking how well the design realizes the requirements. A verification of the design is a complement - not a replacement - to a verification of the implementation. Early verification of tool chain design can detect possible misalignments between the product development process and the tool chain, when corrections are still relatively simple and cheap. In addition, when we automate the early verification of tool chain design, it can be performed repeatedly with little effort, allowing for iterative development of tool chains. This leads to the central research question addressed in this paper: How can early verification of the alignment between tool chain design and product development process be performed and automated? The remainder of this paper is structured as follows. In section 2 we describe the approach for automated verification. In section 3 we investigate modeling the product development process, the tool chain and the relation between these models. In section 4 we devise a method to identify misalignments between the development process and the design of the tool chain and we measure the degree of alignment in section 5. In section 6 we apply the verification method on an industrial tool chain and discover possible improvements. In sections 7 and 8, we relate our approach to other work in the field, list future work and consider the implications of this work.

2

Approach

Our goal is to support the development of tool chains by verifying that the tool chain design is aligned to the product development process. The verification produces a list of misalignments and a measurement indicating the degree of alignment between the tool chain and the product development process. In figure 1 we give an overview of the verification approach. As input we need a description of the tool chain design (section 3.2) and a description of the process including the set of tools and their capabilities (section 3.1). These descriptions need to be in an appropriate language for describing the tool chain and its needs. An initial verification graph is created based on the two input models. We automatically add mapping links to the verification graph, which are constructed according to mapping rules (section 3.3). We then apply alignment rules (section 4) on the verification graph to find out in how far the tool chain supports the development process. We also apply metrics (section 5) to determine the degree of alignment between the tool chain and the process. The metrics and 1

Note: In this paper we deal with two different development processes: the development process of the tool chain, called tool chain development process, and the development process of some product, where the development process will be supported by the tool chain (see figure 4), called product development process.

the list of misalignments provide feedback for an iterative design and guide the tool chain designer towards building an improved, more useful tool chain that is aligned with the process.

Fig. 1. Approach for early automated verification of tool chains

3

Building up the Verification Graph

In this section, we describe how we build up the data structure used for verification. The data structure combines the description of the development process (section 3.1) and the tool chain design (section 3.2) by adding mapping links between both descriptions (section 3.3). 3.1

Description of the Product Development Process

We apply the Software & Systems Process Engineering Metamodel (SPEM) [10] to describe both the product development process and the set of tools used. A SPEM model can thus be used to describe the process requirements of a tool chain. A number of concepts are defined in SPEM, we focus here on the core concepts relevant in this context. A Process is composed of several Activities. An

Activity is described by a set of linked Tasks, WorkProducts and Roles. A Role can perform a Task and a WorkProduct can be marked as the input or output of a Task. A WorkProduct can be managed by a Tool and a Task can use a Tool. An example model is provided in figure 4. SPEM can be used at several levels of abstraction: A high-level description may consist only of a process and activities and a low level description may break down each task into steps and detailed guidance. In the following we assume a process description on an appropriate level of abstraction that contains at least instances of all of the above metaclasses. 3.2

Description of the Design of the Tool Chain

We would like to create an early design model that describes all important design decisions of a tool chain. We chose to apply the Tool Integration Language (TIL) [3], a domain specific modeling language for tool chains. TIL allows us not only to model a tool chain, but also to analyze it and generate code from it. Here we can only give a short overview: In TIL we describe the tool chain in terms of a number of ToolAdapters and the relation between then. A ToolAdapter exposes data and functionality of a tool. The relation between the ToolAdapters is realized as any of the following Channels: a ControlChannel describes a service call, a DataChannel describes data exchange and a TraceChannel describes the creation of trace links. An example for using the graphical language of TIL is provided in figure 5. 3.3

Mapping Rules and the Verification Graph

For the purpose of analysis we create an initial verification graph, which contains all the elements and internal links from both the SPEM model and the TIL model. We assume that the descriptions of the process and of the tool chain are internally verified and consistent, i.e., that all used variables have been defined. This graph is the basis for the construction of mapping links that connect model elements from the SPEM and TIL model. The mappings are constructed according to the mapping rules defined in table 1. Table 1. Mapping rules for SPEM and TIL metaclasses SPEM Metaclass TIL Metaclass Role User Tool ToolAdapter Task Channel

The mapping rules introduce mapping links between the model elements of the given types that also have similar instance names. We measure the similarity of names by using the Levensthein distance [9]. This ensures that corresponding

elements will be mapped, even if they do not have the exact same name: e.g. a Tool in SPEM with name ’UML Tool’ will be mapped to a ToolAdapter in TIL with name ’Papyrus UML TOOL’. We add the mapping links to the initial verification graph, which results in a complete verification graph exemplified in figure 2.

Fig. 2. Example of a complete verification graph

4

Verification by Alignment Checking

In order to verify a given tool chain design, we check for alignment of the requirements provided by a SPEM model with the design provided by a TIL model. We base the alignment check on the verification graph described in the previous section and a number of alignment rules. 4.1

Alignment Rules

The basic assumption of our alignment rules is that each SPEM Task that has two different SPEM Tools associated with it, is a candidate task for support by the tool chain; we call it integration-related task. Arbitrary SPEM Tasks connected to only one tool, are of no interest for tool integration; for these tasks, users will work directly with the GUI of a single tool. We distinguish two alignment rules in figure 3: (A) for Tasks and (B) for WorkProducts. Apart from rules (A) and (B), additional rules may be added if additional correspondences between TIL and SPEM are identified. (A) For each SPEM Task with two SPEM Tool associated with it, this rule expects corresponding TIL ToolAdapters and a TIL Channel between them. A pair of TIL ToolAdapter and SPEM Tool are corresponding if they have mapping links between them (see section 3.3). This rule requires two such pairs, one pair with name X and one pair with name Y, where X 6= Y. Optionally, for each

SPEM Role that is connected to the SPEM Task, we expect a corresponding TIL User that is connected via a TIL ControlChannel with the TIL Channel. (B) For each SPEM Task that has associated SPEM WorkProducts as input and output, where the input SPEM WorkProduct has a different associated SPEM Tool than the output SPEM WorkProduct, the rule expects to find TIL ToolAdapters corresponding to the SPEM Tool and a TIL Channel between them.

Fig. 3. Alignment rules and examples: (A) for tasks and (B) for workproducts, using the notation of figure 2

The rules contain optional parts, visualized by dashed elements. For both rules A and B we need to check the (1) realized and (2) necessary alignments, thus the alignment rules can be interpreted in two directions: – (1) If the lower part of the rule can be found in the SPEM model, the upper part needs to be found in the TIL model as well. Each integration-related task in the process model should also be represented in the tool chain, we call it a realized alignment, because the requirements provided by the process model are realized by the tool chain. This direction of rule interpretation checks if the requirements given by the product development process are actually realized by the tool chain. – (2) If the upper part of the rule can be found in the TIL model, the lower part needs to be found in the SPEM model. Each Channel in the tool chain

design should have a corresponding requirement in the process model, we call it a necessary alignment, because this part of the tool chain is necessary for fulfilling the requirements of the process This direction of rule interpretation checks if the tool chain contains superfluous features that are not used by the process. 4.2

Alignment Checking

We perform pattern matching of the alignment rules on the verification graph. As a result of the pattern matching we get a list of alignments and a list of misalignments. An alignment is a match of an alignment rule, a misalignment is a mismatches of the alignment rules, where only the upper part (TIL part) or lower part (SPEM part) of the rule can be found in the verification graph. Assuming that the product development process is described correctly, we can add a recommendation for resolving the misalignment by adapting the tool chain design in TIL. If a Channel is missing, we can suggest to add it to the TIL model because it would improve the functionality of the tool chain. If the Task is missing in the SPEM model, we can suggest to remove the Channel from the TIL model, because it is not required.

5

Quantification by Alignment Metrics

Tool chains provide increased automation of integration-related tasks. Moreover, the automation needs to be appropriate to the needs of the tool chain. Appropriate automation depends on the alignment of tool chain and product development process, the set of product development tools and their capabilities. We measure usefulness based on two degrees of alignment. The degree of alignment can be determined based on the results of the alignment checks explained in the previous section. We make the following definitions: – #matchesSP EM are the number of matches of the SPEM part (lower part) of rules A and B. – #matchesT IL are the number of matches of the TIL part (upper part) of rules A and B. – #matches are the number of matches of the combined SPEM and TIL part of rules A and B. The recall alignment in formula 1 measures the ratio of integration related tasks in the process model that are realized by a channel in the tool chain. The precision alignment in formula 2 measures the ratio of channels in the tool chain that are required by a corresponding integration-related task in the process model. alignmentrecall = alignmentprecision =

#matches #matchesSP EM #matches #matchesT IL

(1) (2)

A well-aligned tool chain has both a high degree of precision alignment and a high degree of recall alignment, typically expressed by F-measure. A high degree of precision alignment shows that the tool chain has little or no superfluous features. A high degree of recall alignment shows that the tool chain has all or almost all the features needed. A low degree of precision alignment can be fixed by reducing unused features from the tool chain. These features will not be used according to the process description and only increase the development cost of the tool chain. A low degree of recall alignment might be even more critical. It needs to be fixed by introducing new features into the tool chain design, so the integration-related tasks are supported by the tool chain.

6

Case Study

This case-study shows the development process of an embedded system that is characterized by tightly coupled hardware and software components: – The requirements of the system are captured in a requirements management tool. The system architect designs a UML component diagram and creates trace links between UML components and the requirements. – The UML model is refined and a fault tree analysis is performed in the safety analysis tool. When the results are satisfactory, the control engineer creates a Simulink model for simulation and partitions the functionality for realization in software and hardware. – The application engineer uses Simulink to generate C code, which is refined in the WindRiver tool. The data from UML and Simulink is input to the IEC-61131-3 conform ControlBuilder tool. The data from ControlBuilder, Simulink and WindRiver is integrated in the Freescale development tool for compiling and linking to a binary for the target hardware. – A hardware engineer generates VHDL code from Simulink and refines it in the Xilinx ISE tool. A SPEM model of this development process for hardware-software co-design used by the company is modeled as shown in figure 4. The tool chain designer creates the initial tool chain design shown in figure 5. Both models are input to our algorithm, which identifies the misalignments shown in table 2. As it turns out, the tool chain designer built the tool chain according to the verbatim description of the process given above. This description, however, does not cover the intricate details of the relationships between tools that are present in the SPEM model. The tool chain design only contains forward links between tools, as they would be used in a waterfall process model, but neglects the cross links and dependencies typical for modern engineering processes. In this case study we have a cross link between Simulink and ControlBuilder and one between WindRiver and ControlBuilder. In addition, the process model specifies a pair of links between ControlBuilder and FreeScale and another pair between Simulink and FreeScale, but the tool chain design only realizes one link out of each pair.

Fig. 4. Product development process of the case study as a SPEM model

Fig. 5. Tool chain of the case study as a TIL model

For the case study the degree of recall is 69%, the degree of precision is 100%. This tells us that the tool chain has no superfluous features, but does not realize all necessary ones. The tool chain could be improved by adding the four missing Channels to the TIL model, as listed in table 2 Table 2. The misalignments and suggested improvements for the tool chain design TaskDefinition Transfer HWD Transfer CIT2 Transfer 1131 Transfer CIT

7

Input ToolDefinition Simulink WindRiver WorkBench ControlBuilder Tool Simulink Tool

Output ToolDefinition ControlBuilder Tool ControlBuilder Tool FreeScale FreeScale

Suggested Improvement Add TIL Channel Add TIL Channel Add TIL Channel Add TIL Channel

Related Work

There are a number of approaches for tool integration, as documented in the annotated bibliographies [4, 12], however, most tool integration approaches do not explicitly take the development process into account. In this section we focus on the few approaches that acknowledge a relationship between process and tool chain. We classify this related work on tool integration according to two dimensions: Executing the process using (1a) interpretation vs. (1b) compilation and applying a process modeling formalisms that is (2a) proprietary vs. (2b) standardized. (1a) Interpretation-based approaches use the process definition directly to integrate tools; this techniques is also known as enactment of process models. Since the description of the process is the same as that of the tool chain, the two have the advantage to always be aligned. There are two preconditions for the interpretation approach: the process model needs to be executable and the access to data and functionality of the development tools needs to be possible. (2a) The use of a proprietary process model in tool chains is introduced in [6] as the process-flow pattern. In this approach a workflow realizes control integration and is executed by interpretation. (2b) A well known standardized process model is SPEM. SPEM itself [10] is not executable but there are extensions to SPEM that make the process model executable [8, 2]. The orchestration of tools by a process model is shown in [11]. However, the interpretation of integration related tasks is often not possible, since the interfaces to the development tools are too different. Thus the use of process enactment for building tool chains is limited. (1b) Compilation based approaches transform the process model into another format, where the process model serves as a set of requirements. (2a) Proprietary process models provide great flexibility to adapt them to the specific needs of tool integration. An integration process model is developed in [1], where each process step can be linked to a dedicated activity in a tool. For execution it is compiled into a low-level process model. The proprietary process model needs

to be created specifically for constructing a tool chain. (2b) In this work we assume a compilation-based approach, where a process model is compiled or manually transformed into a tool chain design. We use the standardized process metamodel SPEM [10], which allows us to reuse existing process models as a starting point for creating a tool chain design. The described approach verifies that a tool chain design fulfills the requirements described in SPEM.

8

Future Work and Conclusion

In this work we propose a verification method for early design that checks the alignment of a tool chain with a product development process. We formalize this relationship, which allows us to reason about the relationship in an automated way and verify the tool chain against the needs of the product development process using consistency checks and metrics. There are various ways in which tool integration can be modeled in SPEM. In this work we have identified two patterns, in future work, we would like to identify additional patterns for modeling tool integration in SPEM. The verification rules presented in section 4 may not only be used analytically for verification, but also constructively for automatically building a design model for a tool chain from a given process description. In the future we would like to explore this idea and integrate it into a comprehensive methodology for developing tool chains. We would also like to apply this approach on several bigger case studies. The case study should contain a thorough investigation of the false positives and false negatives found with the approach. The approach contributes to the vision of a systematic development methodology for tool chains with a potential for cost savings and an increased time to market. An important prerequisite for reaching this potential is a clear description of the product development process that the tool chain is intended to support. The approach guides the tool chain designer by analyzing different integration options and eventually building a tailored tool chain that is suitable and useful to the tool chain user in the context of a given product development process.

References 1. A. Balogh, G. Bergmann, G. Csert´ an, L. G¨ onczy, Horv´ ath, I. Majzik, A. Pataricza, B. Polg´ ar, I. R´ ath, D. Varr´ o, and G. Varr´ o. Workflow-driven tool integration using model transformations. In G. Engels, C. Lewerentz, W. Sch¨ afer, A. Sch¨ urr, and B. Westfechtel, editors, Graph transformations and model-driven engineering, chapter pages 224–248. 2010. 2. R. Bendraou, B. Combemale, X. Cregut, and M. P. Gervais. Definition of an Executable SPEM 2.0. In Software Engineering Conference, 2007. APSEC 2007. 14th Asia-Pacific, pages 390–397. IEEE, December 2007. 3. Matthias Biehl, Jad El-Khoury, Fr´ed´eric Loiret, and Martin T¨ orngren. A Domain Specific Language for Generating Tool Integration Solutions. In 4th Workshop on Model-Driven Tool & Process Integration (MDTPI2011), June 2011.

4. Alan W. Brown and Maria H. Penedo. An annotated bibliography on integration in software engineering environments. SIGSOFT Notes, 17(3):47–55, 1992. 5. Alan Christie and et al. Software Process Automation: Interviews, Survey, and Workshop Results. Technical report, SEI, 1997. 6. Gabor Karsai, Andras Lang, and Sandeep Neema. Design patterns for open tool integration. Software and Systems Modeling, 4(2):157–170, May 2005. 7. James D. Kiper. A framework for characterization of the degree of integration of software tools. Journal of Systems Integration, 4(1):5–32, February 1994. 8. Ali Koudri and Joel Champeau. MODAL: A SPEM Extension to Improve Codesign Process Models New Modeling Concepts for Today’s Software Processes. volume 6195 of Lecture Notes in Computer Science, chapter 22, pages 248–259. Springer Berlin / Heidelberg, Berlin, Heidelberg, 2010. 9. Vladimir I. Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Doklady Akademii Nauk SSSR, 163(4):845–848, 1965. 10. OMG. Software & Systems Process Engineering Metamodel Specification (SPEM). Technical report, ”OMG”, April 2008. 11. B. Polgar, I. Rath, Z. Szatmari, A. Horvath, and I. Majzik. Model-based Integration, Execution and Certification of Development Tool-chains. In Workshop on model driven tool and process integration, June 2009. 12. M. N. Wicks. Tool Integration within Software Engineering Environments: An Annotated Bibliography. Technical report, Heriot-Watt University, 2006.