What is Trust in a Recommender for Software Development? Position Paper Gail C. Murphy and Emerson Murphy-Hill Dept. of Computer Science University of British Columbia Vancouver, Canada

{murphy,emhill}@cs.ubc.ca ABSTRACT Many recommendation systems have been built to aid software developers. Few have been evaluated using human-based evaluation strategies. In studying situations where recommendation systems have been used or might be used, we have observed that issues related to trust are important to a developer’s acceptance of recommendations. In this position paper, we outline the trust issues that we have identified and suggest some mechanisms for promoting trust in recommendation systems aimed at software developers.

Categories and Subject Descriptors D.2.2 [Software Engineering]: Design Tools and Techniques— user interfaces

General Terms Human Factors

Keywords trust, recommendation, software development

1.

INTRODUCTION

A wide variety of recommenders to aid software developers in the performance of their tasks have been built. Some of these recommenders suggest code elements either from within a software system’s code base that a developer should likely visit as part of an evolution task (e.g., ROSE [5]) or code examples from other systems to aid a developer in using an existing framework or library (e.g., Strathcona [2]). Other recommenders suggest other development artifacts, such as bugs or mail messages related to a given task (e.g., Hipikat [1]). Still other recommenders suggest tools that might help a developer work more effectively (e.g., [4]). Most of these recommenders have been evaluated analytically by determining the goodnesss of the recommendations in terms of precision and recall. Very few of the recommenders have been evaluated in realistic usage situations by developers. We have recently

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. RSSE ’10, May 4 2010, Cape Town, South Africa Copyright 2010 ACM 978-1-60558-974-9/10/05 ...$10.00.

conducted two human-based investigations related to recommendations in software development. In the first investigation, we evaluated a recommender for software tools [4]. This recommender watches a developer’s actions as she moves between code elements and then makes a tool recommendation. For example, upon noticing that a developer makes repeated use of the Eclipse IDE’s “Find References” command on a series of methods in a call chain, the recommender suggests the Call Hierarchy as a means for the developer to navigate more efficiently between the code elements in the future. The developers in this study were eighteen third-year undergraduate students working on a multi-week software development project. Part of the data we collected involved interviewing these students at the end of their projects. We refer to this investigation as Study1 in this paper. In the second investigation, we interviewed eighteen professional software developers to explore how they learned tools they used to complete programming tasks. These interviews were conducted semi-structured, one-on-one, phone or chat interviews lasting about an hour each. We refer to this investigation as Study2 in this paper. Through these investigations, we have observed several important issues related to the context of providing recommendations. In several cases, these observations echo situations we have seen arise in other human-based evaluations of recommender systems that we have conducted. In this position paper, we focus on issues related to the trust a developer has in recommendations provided.

Timing of Recommendation Effects Trust. When a developer considers a recommendation, he or she expects the recommendation to be directly related to the task the developer would identify he or she is currently performing. In Study1 , the recommendations being provided to developers were about the process a developer was using to navigate the code and not about the specific navigations. In designing a user interface for these recommendations, we wanted to both make a developer aware that a recommendation was available and yet not interrupt the flow of the developer’s performance of the task. The user interface for the recommender used in Study1 was iteratively designed to help ensure these characteristics were met; although alerted to the presence of recommendations, developers chose the point in time at which they investigated the recommendations. Even though the developers chose when to consider a recommendation, they still often tried to relate the recommendation to their current task, immediately invoking a recommended tool rather than reflecting upon their process. When they found that the recommended tool did not produce a meaningful result at the moment for the current point in their task, their trust in the recommendations immediately dropped. In discussions with colleagues creating recommenders for other engi-

neering domains, similar problems have arisen where the recommender makes a suggestion that the user then misinterpret. The result is a lose of trust of the user in the recommender’s ability to function properly.

Social Trust Eases Uptake of Recommendations. From Study2 , we gained evidence that recommendations of tools from one peer to another is more likely to be effective when the peers know each other. As the following quotes from participants in Study2 indicate, this effectiveness is in part because the peer who is receiving the recommendation from another peer is more easily able to assess the relevance of the recommendation to their own working style. "If you can get a feel for people that you know, either because you know them online or online discussions or forums, or a person that you know in person, people your work with or user groups or whatever, you can get a handle on how their thinking style works, relative to your own, what their skill level is, etc, and based on their reaction to tools, you have a pretty good estimate of what your reaction is going to be." "I’ve known them for a while, and how they get things done, so I can translate that to how I get things done"

Trust Trumps Precision. A participant in Study2 articulated the undercurrent of trust issues we saw in both investigations. "Honestly; I bet the [recommender system] would have better success rate [than a peer] at recommending things that I would like, but that doesn’t mean that I would trust the [recommender system] more." This comment suggests the community needs to place more emphasis on issues of trust to make usable recommender systems given that the precision of many of these systems is argued to be sufficiently high from analytic investigations.

2.

ESTABLISHING TRUST

How do developers establish trust with one another? It appears from our interviews in Study2 that for trust to exist, the recommender must have had some prior interaction with the developer that enables the developer to estimate the recommender’s knowledge and skills. How can prior interaction be provided when a recommender system is installed into a developer’s work environment? One way might be to borrow trust from a trusted peer of the developer. Instead of just presenting a recommendation, the recommender system could say "when John checks in code, he always looks at the following code elements", assuming John is a trusted peer of the developer receiving the recommendation. This approach was taken in Toolbox, where the system looked for expert tool users, requested that these experts write down descriptions of the tools, and published the descriptions as newsletters in the expert’s organization [3]. Another way might be to avoid the trust dilemma by sidestepping it altogether. One participant in Study2 suggested recommendations from a recommender system might be more acceptable if the system made it feel like the developer discovered the information themself. Achieving this feeling for a developer is challenging but might be possible through user interface mechanisms that subtly reorganize code that might be of interest to a developer or that somehow make a tool that could be helpful to a developer more obvious in the interface.

3.

SUMMARY

Investigations we have conducted involving both novice and professional programmers suggest that trust is a critical factor in the acceptance of recommenders aimed to improve software development. This result is not surprising. Our investigations also suggest some strategies for potentially dealing with the trust issue in software development recommenders. One suggestion is to borrow trust from a trusted peer, which might be possible through rephrasings of presented recommendations. Another suggestion is to sidestep trust, creating interfaces that instead of recommending enable the user to discover the desired (recommended) information.

4.

REFERENCES

[1] D. Cubranic and G. C. Murphy. Hipikat: Recommending pertinent software development artifacts. In Int’l Conf. on Soft. Eng., pages 408–418, 2003. [2] R. Holmes and G. C. Murphy. Using structural context to recommend source code examples. In Int’l Conf. on Soft. Eng., pages 117–125, 2005. [3] C. Maltzahn. Community help: discovering tools and locating experts in a dynamic environment. In Conf. Comp. on Human Factors in Comp. Sys., pages 260–261, 1995. [4] P. Viriyakattiyaporn. An active help system to improve program navigation. Master’s thesis, Dept. of Comp. Sci., Univ. of British Columbia, 2009. [5] T. Zimmermann, P. Weißgerber, S. Diehl, and A. Zeller. Mining version histories to guide software changes. In Int’l Conf. on Soft. Eng., pages 563–572, 2004.

What is Trust in a Recommender for Software Development?

trust, recommendation, software development. 1. INTRODUCTION. A wide variety of recommenders to aid software developers in the performance of their tasks ...

78KB Sizes 0 Downloads 187 Views

Recommend Documents

What is Trust in a Recommender for Software ...
University of British Columbia. Vancouver, Canada. {murphy,emhill}@cs.ubc.ca. ABSTRACT. Many recommendation systems have been built to aid software de-.

In Response to: What Is a Grid?
Correspondence: Ken Hall, BearingPoint, South Terraces Building, ... the dynamic creation of multiple virtual organizations and ... Int J. Supercomp App. 2001 ...

In Response to: What Is a Grid?
uted and cost-effective way to boost computational power to ... Correspondence: Ken Hall, BearingPoint, South Terraces Building, .... Int J. Supercomp App. 2001 ...

Trust Is Risk: A Decentralized Financial Trust Platform - GitHub
cannot allocate much direct trust to her friends, no matter how trust- worthy they are. On the .... [4] that the trust Alice shows to Bob in real-world social networks cor- responds to the extent of ..... decentralizedcurrency.pdf (2004). 28. Schartz

A Recommender-System for Telecommunications ...
telecoms network management system using a. Recommender System. This Recommender System approach has the advantage that the network management system will learn the human expert's approaches, preferences and knowledge over time and evolve the recomme

what is open source software pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. what is open ...

what is software project management pdf
what is software project management pdf. what is software project management pdf. Open. Extract. Open with. Sign In. Main menu. Displaying what is software ...

What Is Conveyancing In Pakenham.pdf
If you don't know what conveyancing means, let me tell you, it is a legal procedure for the transfer of the. property rights between the buyer and the seller.

What Is Conveyancing In Pakenham.pdf
If you don't know what conveyancing means, let me tell you, it is a legal procedure for the transfer of the. property rights between the buyer and the seller.

Open Position: Software Engineer MAQ Software is a ... -
Our solutions use advanced Business Intelligence features of SQL. Server 2012, the latest ... Display web site traffic data for ten million users per day. In order to ...

Utilizing Recommender Systems to Support Software ...
requirements gathering phase in which business analysts work with project ... technology have led to a growing trend towards using online forums and wikis to facilitate ... The recommender system can then suggest ideas for new discussion ... Internet

what is millennium development goals pdf
Loading… Page 1. Whoops! There was a problem loading more pages. what is millennium development goals pdf. what is millennium development goals pdf.