Personality-based selection 1 Running Head: Personality-based selection

Commentary on “Reconsidering the Use of Personality Tests in Personnel Selection Contexts”: Thoughts from a Practitioner Perspective

Steven T. Hunt

October 23, 2007

Author Contact Information: Steven T. Hunt 17145 SW Stellar Drive Sherwood OR, 97140 Phone: 503-709-6572 E-mail: [email protected]

Personality-based selection 2 Abstract A practitioner with experience utilizing personality tests to support personnel selection decisions provides commentary on several aspects of the article “Reconsidering the Use of Personality Tests in Personnel Selection Contexts”. Several areas of agreement are noted between views expressed in the article and conclusions drawn from the author’s experience using personality tests. Comments are made regarding the implications that faking has on the applied use of personality tests, the risks posed by questionable metaanalytic research findings regarding the validity of personality-based assessments, and the financial value provided by using personality tests that have relatively low levels of validity. Concerns are raised regarding criticisms made in the article that question the value of existing self-report personality tests.

Personality-based selection 3

Commentary on “Reconsidering the Use of Personality Tests in Personnel Selection Contexts”: Thoughts from a Practitioner Perspective As a research oriented practitioner with experience using personality assessments for personnel selection, I took great interest in the recent article by Morgeson, Campion, Dipboye, Hollenbeck, Murphy, and Schmitt (2007) entitled “Reconsidering the use of personality tests in personnel selection contexts.” The article provides observations from five journal editors with extensive experience reviewing research exploring the applied use of personality tests. Because of their unique role in the scientific community, these editors are presented as having a perspective on personality research that is both expansive yet “relatively impartial” (p.687). I agree that the editors have a perspective that is both insightful and fairly unbiased. But it would have been useful to also know how much experience the editors have had implementing personality measures in an applied setting. The article does not indicate if the editors have helped organizations actually use personality tests for selection, or if they have any current or past associations with personality test vendors or other staffing assessment companies. Such full disclosure seems relevant and appropriate for this sort of article. It also seems that the article might have benefited from including commentary drawn from people with first-hand experience implementing personalitybased selection measures in applied settings. The purpose of this commentary is to provide thoughts on the Morgeson et al. (2007) article from the perspective of a person with experience designing, validating, and

Personality-based selection 4 implementing personality based assessment instruments. A brief summary of the author’s professional experience is provided to justify why I might be able to provide some useful thoughts into the dialogue on personality testing. I then give my general impression of the article, followed by a more in-depth discussion of issues related to faking on personality measures, the impact of meta-analysis on the field of applied personality testing, the value of personality tests with relatively low levels of validity, and summary recommendations for future directions in this field.

My Experience I have been involved in the field of personality-based selection since 1992. This time period roughly overlaps with the time the authors of the Morgeson et al. (2007) paper served as journal editors. During this time I have worked for three different personality test vendors1, worked for a Fortune 500 company to develop large scale staffing assessment processes, developed and validated approximately 15 new personality measures, conducted roughly twenty validation studies using various “off-the-shelf” personality tests, and read hundreds of published and unpublished papers on personality assessment from peer-review journals, scientific conferences, client specific research studies, and internal assessment vendor technical analyses. I have also written several peer-review and practitioner articles discussing topics relevant to personality based assessment and authored two books that deal extensively with issues related to the applied use of personality tests for personnel selection (Handler & Hunt, 2003; Hunt, 2007). While this experience does not make me the “world’s authority” on applied use of

1

Personnel Decisions Inc., SHL Inc., and Kronos Inc. (formerly Unicru Inc.)

Personality-based selection 5 personality tests, it does give me a fair bit of insight into issues affecting how these measures are used in organizations.

General Impression. What struck me most when reading the Morgeson et al. (2007) was the overlap between the views expressed in the article and views I have developed through my own experience. I tend to agree with most of the points made in the article, with a few important exceptions that will be discussed later. Whether this is “great minds thinking alike” or just a happy coincidence is for others to decide. But it does seem like some general truths may be emerging in the field of personality-based selection.

The Prevalence of Faking and Its Implications. Morgeson et al. (2007) point out that although faking is one of the more common criticisms made toward using personality tests for employee selection, it does not seem to be nearly as problematic as is often believed. This aligns closely with conclusions I have drawn from applied experience. I recently published a book on the use of staffing assessments that summarized the following points regarding faking (Hunt, 2007): Applicant faking is not the same as lying Many applicants do not fake Most applicants who fake are not very good at it Some faking is not faking but a lack of self-awareness Faking can significantly affect the scores of some assessments, but does not affect all assessments equally

Personality-based selection 6 These points can be mapped almost directly to comments made in Morgeson et al. (2007). Morgeson et al. (2007) are less clear on whether faking is a significant threat to the use of personality measures. From my experience, I would say that it depends on the nature of the assessment and how it is used. If an assessment seems like it is easy to fake then some people, but not all will probably fake it. And if an assessment seems very easy to fake, then it is likely that high scores on the assessment will reflect candidates’ willingness to respond in a socially desirable manner as much or more than it reflects other, potentially more enduring and influential attributes. Companies should also remember that faking is not limited to personality tests. Faking is a potential source of error whenever candidates are asked to describe themselves or indicate how they would act on the job. It can and does affect all forms of self-report assessments including interviews, job applications, pre-screening questionaires, biodata, job fit measures, interest inventories, simulations, and resumes. This still leaves the problem of what to do when faking appears to be heavily influencing assessment results. Morgeson et al. (2007) provide some insightful observations regarding what it means when candidates respond candidly to personality items that other people tend to fake (i.e., endorsing a response that most people would view as socially inappropriate). If someone fails to pass a personality test that anyone with a modicum of impression management skills should be able to pass, then it clearly says something about that person. Usually what it says is “don’t hire them”. One does not need extensive validation data to justify not hiring someone who responds affirmatively to a question like “Is it acceptable to steal from your employer?”

Personality-based selection 7 But what does it mean if someone passes a personality test that is considered relatively easy to fake? This poses a particularly significant problem for personality based selection measures that classify candidates based on normative data. Companies often want assessments that rank order candidates from best to least. As a result, many vendors norm their assessments so each candidate is given a percentile score from 1 to 100. But, as Morgeson et al. (2007) suggest, the validity of many personality tests may be driven primarily by predictive variance associated with applicants who score poorly on the assessment. Percentile scores for such tests are likely to be misleading because there may be no substantive differences in the quality of candidates scoring at the 50th percentile and those scoring at the 90th percentile. In fact, I have seen data where people scoring above the 90th percentile tend to be less successful candidates, probably because their score is more a reflection of their willingness to fake than their actual personality. Using norming and percentiles with personality tests that are highly susceptible to faking can give the impression that the test results are more meaningful than is warranted. It may be more appropriate to interpret these sorts of personality tests in a manner similar to how companies interpret drug test results. Companies do not ask for percentile scores from drug test vendors. Drug tests provide a simple pass-fail based on objective cut off values. These cut-offs remain constant over time and do not vary depending on the characteristics of a company’s applicant population. Companies also recognize that while knowing a candidate failed a drug test may be a good indication of potential performance problems, knowing a candidate passed a drug test provides little value for predicting superior performance. One could argue that a similar approach should be used with personality tests that are notoriously easy to fake (e.g., integrity tests). Rather than

Personality-based selection 8 norming these tests, vendors should establish set cut-points based on data indicating that candidates who score below certain levels are statistically more likely to exhibit performance problems. These cut-points should remain fixed and should not change based on normative comparisons to other applicants. This is likely to result in extremely high pass rates for some personality-based selection tests currently used in the market. But it will reduce the risk of evaluating candidates based on normative comparisons that may depend more on a person’s willingness to fake test responses than substantive differences in job relevant personality traits. Meta-analysis and Meta-assumptions As a person who has spent of much of his career analyzing personality data, I strongly applaud the comments made by Morgeson et al. (2007) regarding questionable assumptions made in meta-analytic studies of personality test validity (readers are encouraged to see Schmitt (2007) for more on this topic). Meta-analysis has been useful for gaining broad acceptance toward the value of personality measures in general. But meta-analytic results are also used to justify deploying personality tests without conducting adequate research to verify their relevance for specific hiring situations. For example, sweeping statements in meta-analytics studies implying that conscientiousness has universal validity across all jobs serves to encourage sloppy implementation of personality tests. I have been in conversations where questions about test validity have been addressed with the statement “this is a measure of conscientiousness”. This statement is interpreted as a guarantee that the test always predicts job performance regardless of the position, organization, or applicant population. Rarely is discussion raised around the specific job behaviors the test has been shown to predict or which

Personality-based selection 9 facets of conscientiousness it actually measures (Dudley, Orvis, Lebiecki, & Cortina, 2006). These are important questions that should be addressed before deploying any selection instrument. Unfortunately, claims made by some meta-analytic studies seem to promote a broad brush approach toward assessment validity that encourages such simplistic thinking. Given the low levels of validity frequently associated with personality measures, the field should be emphasizing more rigorous validation efforts, not less. Yet some assessment vendors use the results of meta-analytic studies along with other content validity generalization arguments as an excuse for deploying personality tests without conducting thorough job analysis and validation studies. It is refreshing to see limitations of meta-analysis noted in a high profile journal like Personnel Psychology, particularly by such well respected academicians. Hopefully this will lead to more emphasis on conducting primary research into the precise nature of relationships between specific facets of personality and different dimensions of job performance.

The Value of Personality Tests With Low Levels of Validity The only area where I tended to explicitly disagree with statements in Morgeson et al. (2007) was in the section discussing the value of personality measures with low levels of validity. I agree that the validities of many existing personality tests are depressingly low. This represents an enduring problem and I concur with the call to look for alternative methods to measure personality. Although, as Dipboye notes, “the time, effort, and cost of developing such measures…are likely to remain as major impediments” to their use (p. 692). I also agree with Schmitt and Campion’s

Personality-based selection 10 recommendation that companies should strive to use personality measures designed and validated for specific jobs rather than relying on generalized measures. But this approach remains cost prohibitive for many organizations due to the resources currently required to create such tailored measures. Until we develop better and more affordable methods for measuring personality, the option to use generalized personality tests with relatively low validity levels remains one of the better selection methods available to many organizations. In certain hiring situations, the use of these tests can provide considerable financial gains to a company. Consider the following example, which is a realistic representation of many hiring situations I have encountered: A company employs 100,000 hourly retail clerks who work an average of 30 hours per week with an average tenure of 6 months. These clerks are hired by store managers working in remote store locations. Applicants apply by walking into the store and filling out an application or applying online. The company receives an average of 6 qualified applications for every hire they make. But store managers only interview an average of 3 applicants for each hiring decision. In other words, there is a limit to how many interviews Store Managers will conduct before making a hiring decision. After interviewing a certain number of candidates they simply hire the best candidate they have interviewed so far. Because only 3 out of 6 qualified applicants are interviewed, considerable value can be provided by influencing how Store Managers choose which applicants to interview. Historically, applicants’ chances of being interviewed were based primarily on when they applied. If you applied when the manager was in the store then you were

Personality-based selection 11 more likely to be interviewed. There is no evidence that this method had any validity for predicting job success. For this reason, the company decided to explore other means to prioritize which applicants to interview first. But for operational, legal, and cultural reasons, the company is unwilling to spend more than 15 minutes assessing candidates, will not support the use of proctored assessments in a controlled setting, does not want to use assessments that violate the EEOC 4/5th guideline, and is unwilling to invest the resources needed to develop a tailored assessment. The company decided to use an off-the-shelf personality test designed to measure traits associated with sales performance. This test takes 15 minutes to complete and shows no adverse impact. This assessment categorizes applicants as “green” or “red” based on whether they score above or below the 50th percentile relative to other candidates2. Hiring managers are instructed to interview available green applicants before they interview red applicants. For simplification, imagine this resulted in hiring managers only hiring green applicants (in actuality, I have found that companies using this type of selection process tend to hire around 90% green applicants). Suppose the predictive validity of the personality test is .10 when compared with employee sales per hour. This is reflective of actual predictive validation results for this sort of assessment based on comparing applicant data collected in an un-proctored setting with objective job performance criteria collected after the applicants are hired (ThissenRoe, Scarborough, & Chambless, 2006). Imagine the SDy for clerks was $5 per hour, which seems a reasonable estimate given the typical salary paid to hourly retail clerks (Becker & Huselid, 1992). If we assume the scores on the personality test follow a

2

Assume this measure is relatively resistant to influences associated with applicant faking.

Personality-based selection 12 standard normal distribution, then using the Brogden-Cronbach-Gleser utility model suggests the following annual financial value from using this assessment (Cascio, 1991): Annual Value = (N)(T)(Annual SDy)(rxy)(Zx) N

= 50,000 hires per year

T

= 6 months average tenure (.5 years)

Annual SDy

= ($5/hour)(30 hours/week)(50 weeks/year) = $7500

Rxy

= .10

Zx

= Average standard predictor score of the selected group = .48

Annual Value = (50,000) (.5 years) ($7500) (.10) (.48) = $9,000,000 As with all utility estimates, one can find limitations with this model. For example, because retail sales people tend to “poach” accounts from one another it is unlikely there would be a uniform increase in sales across the entire workforce as is assumed in this model. But such limitations aside, the model does make the following point pretty clearly: assessments with small levels of validity provide significant financial value when applied to large numbers of hiring decisions. Hollenbeck mentions this in Morgeson et. al. (2007) but refers to the value as merely “non trivial” (p. 708). I know of several staffing technology companies that could implement a fully automated version of the assessment solution described in this example for under $2 million per year. This solution would provide a return on investment (ROI) of more than 300%. Few strategic human resource interventions can claim such high levels of financial return. Moreover,

Personality-based selection 13 this value is achieved with an intervention that requires little ongoing time or effort on the part of internal organizational members. In sum, while we should not stop working to build better personality tests, vendors need not apologize for offering clients access to currently available personality tests with uncorrected, predictive validities around .10, assuming these solutions are applied in the appropriate context. Nor should the academic community be overly critical of these assessments given the financial value they provide. It is true that these measures are not as predictive as we would like them to be, but in many hiring situations such personality tests are clearly better than other forms of assessment used by many organizations (e.g., unstructured interviews, credit checks).

Summary Thoughts Morgeson et al. (2007) provides a valuable contribution to the literature discussing the applied use of personality tests. The article includes some refreshingly direct commentary on the limitations of personality measures, as well as some excellent suggestions for further research. But as a practitioner, the article struck me as overly focused on the absolute level of validity provided by personality tests, without adequately acknowledging the relative value these measures provide to organizations over other methods frequently used for employee selection. The article is correct in pointing out that personality measures are not as accurate as many vendors and researchers claim. But despite their limitations, personality tests still provide companies with considerable financial value by helping hire people into jobs where they are likely to succeed.

Personality-based selection 14 Morgeson et al. (2007) also contains what in my view are overly strong criticisms of self-report personality measures. Schmitt urges readers to “avoid published personality measures in almost all instances” (p. 715), Campion urges us to “abandon self-report [personality] measurement” (p.719), and Murphy refers to the current methods of self-report personality test as “not salvageable” (p. 719). Such statements could hinder research progress in the field of personality tests by discouraging businesses from using these tools at all. As pointed out in the article, it was not too long ago that the general perception of personality tests was that they do not work and should not be used to guide employment decisions. Many people outside the field of industrial-organizational psychology continue to hold this view and would probably be quite happy to see legislation passed that severely restricts use of these measures (Murphy-Paul, 2006). It has taken years to convince the business community of the potential value offered by scientifically based personality assessments, with many organizations still remaining reluctant to explore use of these complex yet often powerful tools. Yet comments in Morgeson et al. (2007) seem to imply that these efforts have been devoted to measures that never should have been used in the first place. If the authors wish to see continued interest in the business community toward supporting research on personality based selection methods then they might do well to temper some of their more damning criticisms regarding the value provided by the gradual resurgence in personality based assessment over the last 20 years. I am not implying that we should be satisfied with using personality tests with low levels of validity. Creating a personality test with a validity of coefficient of .10 is

Personality-based selection 15 nothing to get excited about, even if such assessments do provide considerable value in certain hiring situations. Research has clearly shown it is possible to consistently achieve validities in the .2 to .3 range by developing personality tests that are designed to predict specific job outcomes, utilize multiple types of items, and incorporate features to reduce the influence of faking (Salgado, Viswesvaran, & Ones, 2002). Because we know how to build these tests, we should increasingly expect to achieve these higher levels of validity. I once heard someone refer to personality assessments as being “like a chainsaw”. If used correctly they make your hiring processes much more efficient and effective, but if used incorrectly they can quickly do some serious damage to your organization. The key to using these powerful but potentially risky tools is to approach them with a clear sense of their strengths and limitations. On one hand, researchers and practitioners should avoid making sweeping statements about the generalized validity of personality assessments. Such statements encourage companies to deploy poorly designed personality tools without adequately testing them to ensure they are actually predicting what they are supposed to predict. On the other hand, we need to be careful to not overly focus on the limitations of personality tests and lose sight of their advantages. Self-report personality tests often provide the best solution available to companies that need to sort through large numbers of candidates in an efficient manner but that do not want to create adverse impact against certain applicant populations. Remember, the question is not whether candidates are going to be evaluated, but how. If companies do not use personality measures for personnel selection, then it is quietly likely they may turn to assessment methods that have even less predictive validity (e.g., un-validated prescreening questionnaires).

Personality-based selection 16 The current state of personality-based selection tests might be summarized as “we know they work and we know they could work better”. Because we know they could work better, as scientists it is incumbent upon us to develop more effective methods of personality assessment. But because we know they work, as practitioners it is incumbent upon us to support and encourage their appropriate use as long as they continue to be the best method available for evaluating candidates based on job relevant personality traits.

Personality-based selection 17 References Cascio, W.F. (1991). Applied psychology in personnel management (4th ed.). New Jersey: Prentice Hall. Dudley, N. M., Orvis, K. A., Lebiecki, J. E., & Cortina, J. M. (2006).A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91, 40-57. Handler, C. and Hunt, S.T. (2003). Rocket-Hire Buyer’s Guide to Online Screening & Staffing Assessment Systems. PubSync: Saint John, WA. Hunt, S.T. (2007). Hiring success: the art and science of staffing assessment and employee selection. Pfeiffer: San Francisco, CA. Morgeson, F.P., Campion, M.A., Dipboye, R.L., Hollenbeck, J.R., Murphy, K., & Schmitt, N. (2007). Reconsidering the use of personality tests in personnel selection contexts. Personnel Psychology, 60, 683-729. Paul, A.M. (2004). The cult of personality. New York, NY: The Free Press. Salgado, J.F., Viswesvaran, C., & Ones, D.S. (2002). Predictors used for personnel selection: an overview of constructs, methods and techniques. In N. Anderson, D.S. Ones, H.K. Sinangil, & C. Viswesvaran (Eds.), Handbook of Industrial, Work, and Organizational Psychology (pp. 165-199). Thousand Oaks, CA: Sage Publications. Schmitt, N. (2007). The value of personnel selection: reflections on some remarkable claims. Academy of Management Perspectives, 21, 19-23. Thissen-Roe, A., Scarborough, D.J., & Chambless, B. (2006). Inadvertent honesty: occurrence and meaning of applicant faking in unproctored personality tests. Paper presented at the 21st Annual Conference of the Society for Industrial and Organizational Psychology. Dallas, TX.

Personality-based selection Commentary on

“Reconsidering the Use of Personality Tests in Personnel Selection Contexts”: ... analytic research findings regarding the validity of personality-based assessments, and ... worked for a Fortune 500 company to develop large scale ... be driven primarily by predictive variance associated with applicants who score poorly on.

39KB Sizes 1 Downloads 255 Views

Recommend Documents

ARTICLES COMMENTARY ON KELLY AND ...
cal model, custodial transfer, gender bias, DSM-/C: empirical studies, and the misapplication of PAS. In their reformulation of the parental alienation syndrome ...

On Efficient Graph Substructure Selection
Abstract. Graphs have a wide range of applications in many domains. The graph substructure selection problem is to find all subgraph isomor- phic mappings of ...

pdf-0961\the-way-of-awakening-a-commentary-on-shantidevas ...
... she met the late Geshe Yeshe. Page 3 of 11. pdf-0961\the-way-of-awakening-a-commentary-on-shantidevas-bodhicharyavatara-by-geshe-yeshe-tobden.pdf.

298922249-Hayman-Market-Commentary-on-China-February-2016.pdf
Feb 10, 2016 - 298922249-Hayman-Market-Commentary-on-China-February-2016.pdf. 298922249-Hayman-Market-Commentary-on-China-February-2016.

Commentary on Synofzik, Vosgerau and Newen 2008
I offer an alternative reading of some of the data they use to argue against the ..... actually performed and those movements that were visually displayed to them .

PDF Lions' Commentary on UNIX with Source Code ...
The Design and Implementation of the FreeBSD Operating System · The UNIX Programming Environment ... UNIX and Linux System Administration Handbook.

pdf-1830\a-commentary-on-the-executive-summary-of-mission ...
... the apps below to open or edit this item. pdf-1830\a-commentary-on-the-executive-summary-of-mi ... ciation-of-academic-health-centers-by-geoffrey-v.pdf.

Military Social Influence Commentary on the article by ...
The author presents two critiques of King's (2010) examination of U.S. military social influence activities. The first concerns her discussion of five competing conceptions of the relationship between social influence and war. It is proposed that the

Commentary on 'Synaesthesia' by Ramachandran and ...
phenomenon that many of my students have asked me to fill. Up to now I have ... in the brain or brain stem 'representations' must be able to get together. The evi- ... The dilemma would not have occurred if the human clinical data had been reli- ...

pdf-1413\commentary-on-revelation-by-ew-bullinger ...
There was a problem loading this page. pdf-1413\commentary-on-revelation-by-e-w-bullinger.pdf. pdf-1413\commentary-on-revelation-by-e-w-bullinger.pdf.

PDF A Critical and Exegetical Commentary on the ...
PDF download Shoeless Joe & Me - Dan Gutman - Book,The Great Naturalists: From Aristotle to Darwin - Robert Huxley - Book,EPUB The Enchanted Apples of Oz - Eric Shanower - Book,PDF Minerva's Voyage - Lynne Kositsky - Book,PDF The Twilight Zone: Walki

Jameson - Representing Capital - A Commentary on Volume One.pdf ...
Page 3 of 15. Jameson - Representing Capital - A Commentary on Volume One.pdf. Jameson - Representing Capital - A Commentary on Volume One.pdf. Open.

(Brazos Theological Commentary on the Bible) By ...
May 1, 2012 - --John Nolland, Trinity College Bristol. "A work of such literary beauty ... creedally for the twenty-first century, just as the church fathers, the Reformers, and other orthodox Christians did for their times and places. Luke, like eac

Commentary on Goldin-Meadow and Brentari
components into a single compositional system. Goldin-Meadow and Brentari highlight the relationship between imagistic properties and categorical properties in communication, focusing on how they emerge from and interact in speech, sign, and gesture.

Commentary on 'Synaesthesia' by Ramachandran and ...
The dilemma would not have occurred if the human clinical data had been reli- ... K.H. (1971b), Languages of the Brain (Englewood Cliffs, NJ: Prentice-Hall).

Selection on Productivity or Profitability
2 While the models cited above and their literature counterparts do actually construct ... 3 Syverson (2004), which uses physical output data as we do in this study, is an ... The output equivalence in this example is meaningful not due ..... 11 As a