Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

Towards a Methodology for Literature Reviews in Social Sciences Irina Saur-Amaral

[email protected]; [email protected] IPAM – The Marketing School; GOVCOPP, Universidade de Aveiro

Abstract The emergence of the Internet, the generalized access to online databases and top journal presence in these databases generated a shift in how literature reviews are to be performed. While fifteen-twenty years ago, researchers in social sciences would focus their efforts in finding journal articles to analyze and they would read and use some dozens of articles in their research, nowadays efforts fall upon selecting the right set of articles, reading and synthesizing in such ways that helicopter and detailed views are combined. There is a generational gap between senior and junior researchers, which so far has turned difficult the development of a methodological perspective of literature reviews in social sciences that bring together both perspectives. Our paper develops a methodology for literature reviews in social sciences, combining the systematic literature review approach, with the traditional literature review approach. We present compare perspectives, analyzing them from a critical perspective and we propose a combined approach to be tested and used for research in social sciences, indicating key validity concerns to be taken into account. Results are useful for senior and junior researchers in social sciences, which undertake literature reviews for their own or for group research. Keywords:

Systematic literature review, Social sciences, methodology

Introduction Literature review in social sciences1 is frequently performed in unsystematic ways. Researchers tend to use informal and unstructured processes to review literatures and decide upon the key areas to research (Tranfield, Denyer, & Smart, 2003, see Table 1, page 213, for a very detailed perspetive over the process), which necessarily leads to an incomplete process of identification of scientific school of thought and to low efficiency in the research process (Kofinas & SaurAmaral, 2008; Saur-Amaral, 2010, 2011). The emergence of Internet and ubiquous access to online databases, in a short period of time, created a gap between senior and junior scientists, as the paradigm shifted from obtaining literature to finding and selecting the relevant literature from a wide number of publications (Hart, 2006; Saur-Amaral, 2011).

1

Chris Hart (2006, p. 13) defines literature review as “the selection of available documents […] and the effective evaluation of these documents in relation to the research being proposed”.

1

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

In our paper, we historically analyze the method of systematic literature review and its application in social sciences research, we compare it with traditional (before-internet) literature review and in the end we propose a methodological framework to literature reviews in social sciences based on those two approaches. Historical overview of Systematic literature reviews Systematic literature review has been extensively used in medical sciences, in order to identify and compile all relevant knowledge on how human body reacts to medicines and medical practices (e.g. BestBETs; Greenhalgh, 1997; Mulrow, 1994). In this field, systematic literature reviews and meta-analyses are common practice. Manuals of good practices of such reviews have been developed, one of the most famous being Cochrane (Amato et al., 2011; Clarkson & Bonetti, 2008; Coster & Norman, 2009; Ferri, Davoli, & Perucci, 2006; Francis, 2006a, 2006b; Grimes, Lopez, Manion, & Schulz, 2007; Neilson, 1998; Riley, Gates, Neilson, & Alfirevic, 2011), and focuses, as before mentioned, on the identification of all relevant knowledge in a given medical field, using a variety of information sources, including unpublished sources (Tranfield, et al., 2003; Tranfield & Mouchel, 2002). With the emergence of online databases and development of Google Scholar, new steps were given to facilitate systematic searches, yet a new problem aroused: how to search and compile information from those sources? In social sciences, the first adaptation of systematic literature reviews was done in 2002 and 2003 (Tranfield, et al., 2003; Tranfield & Mouchel, 2002). Tranfield and his colleagues analyzed the approach used in medical sciences and suggested its adaptation to social sciences, emphasizing however that social sciences had a different way of building science and there were more heterogeneous methodological approaches that turned difficult their combination and made compulsory a close verification of results, via triangulation of scientific papers. They proposed that systematic reviews could be used to develop the decision-making evidence database for managers, to overcome the typical unsystematic, informal and unconditioned process of literature review and identification of key areas to research, which they considered low-effectiveness. They suggested the methodological approach presented in Table 1.

2

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

Later on, other authors (Kofinas & Saur-Amaral, 2008; Saur-Amaral & Amaral, 2010; SaurAmaral & Borges Gouveia, 2007) started to apply the method and to adapt it to include the specificities on certain online databases. Table 1 – How to perform a systematic literature review (synthesis of approach suggested by Tranfield, et al., 2003)

Steps

Methodological concerns

Planning the review:

May require previous studies to better understand the field and identify alternative ways on how the topic has been previously addressed The review protocol should contain a conceptual discussion of research problem. Keywords and search terms should be identified.

  

Why do a review? Prepare review proposal Develop a review protocol

Conducting the review:     

Identify research Select studies Assess their quality Extract data Synthesize data

Reporting and dissemination:  

Developing the report Dissemination into practice

Should be an comprehensive, unbiased search, rigorously applying the review protocol and the inclusion/exclusion criteria. Search should be reported in sufficient detail to ensure replicability. Disagreement between reviewers should be explained and consensus should be reached. The output of the search should be the full list of relevant results. Should be clear and effective. Two types of reports can be produced: descriptive analysis of all results (most relevant authors, journals etc.) and thematic analysis (emergent themes and research questions)

Kofinas and Saur-Amaral (2008), for example, refer how a systematic literature review can reveal different results in different online databases and show that same scientific journal may appear in several databases, thus the usage of various information sources online needs to be carefully planned. Saur-Amaral and colleagues (Saur-Amaral & Amaral, 2010; Saur-Amaral & Borges Gouveia, 2007), on another line of reasoning, show how can a systematic literature review can be performed based on abstracts of recognized scientific papers and help mapping the scientific field under analysis. Other studies simply apply the method to perform a very effective literature review and map the relevant literature (Kofinas & Saur-Amaral, 2008; Macpherson & Holt, 2007; Saur-Amaral & Amaral, 2010; Saur-Amaral & Borges Gouveia, 2007; Thorpe, Holt, Macpherson, & Pittaway, 2005) Further on, books and specific tools have been developed to help scientists in social sciences to plan a systematic literature review and perform it in an effective and rigorous way (in SaurAmaral, 2010; 2011, you can find, among other relevant tools, a comprehensive model for a review protocol).

3

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

Comparative analysis between Traditional literature review and Systematic literature review The importance of literature review is unquestionable in any science, allowing to develop scientific knowledge based on the current knowledge base and taking into account the schools of thought and scientific paradigms to build and report new knowledge (Bohman, 1991; Hollis, 1994; Hughes & Sharrock, 1997; Ladyman, 2002; Outhwaite, 2000). Unsystematic literature reviews have similar objectives to systematic literature reviews (Hart, 2006; Saur-Amaral, 2011; Sekaran & Bougie, 2000; Tranfield, et al., 2003). We compare both approaches in Table 2. Table 2 – Unsystematic versus Systematic literature review (author’s compilation)

Planning - Define the topic

Unsystematic literature review

Systematic literature review





 

Read general material, books, encyclopedia Take notes on the subject Prepare a list of terms and phrases to search (search vocabulary)

Planning - Plan the search



Plan where to search – create a list of possible sources (libraries, librarians to contact)

Conducting - Searching the sources

 

Start searching on all sources Follow relevant references in sources to discover more information

  

Fill in the review protocol, share with peers Include a conceptual discussion of research problem



Comprehensive, unbiased search, rigorously applying the review protocol and the inclusion/exclusion criteria Use several reviewers to perform the same search Never “trust” a database, always doublecheck thoroughly The output of the search should be the full list of relevant results

  

Conducting - Ensuring quality of sources



Reporting

 

Take notes and make sure you search everything

Should be clear Thematic analysis

Do previous studies to better understand the field and identify alternative ways on how the topic has been previously addressed Identify keywords and search terms

  

Report in sufficient detail to ensure replicability Document and explain filtering decisions Disagreement between reviewers should be explained and consensus should be reached

  

Should be clear and effective Descriptive analysis of all results Thematic analysis

4

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

As it can be seen from the comparative analysis presented in Table 2, both methods to perform literature review involve five steps: 1. Define the topic 2. Plan the search 3. Search the sources 4. Ensure quality of the sources 5. Report The approaches on topic definition seem to be more developed in the unsystematic literature review and they are complementary. Planning the search is less oriented in the unsystematic literature review, which may result in lack of effectiveness and focus in the search process (3rd step), while in the systematic literature review, the approach is very focused and effective, allowing to foresee difficulties and to implement validity measures to control the search process. The search process is clearly better oriented in the systematic literature review, ensuring objectivity, transparency and reasoning for the decisions taken along the process, while in the unsystematic literature review there is very limited way to verify whether all relevant sources have been analyzed and decisions to follow new relevant sources are not justified. In the systematic literature review, there are ways to ensure quality of the sources, as the inclusion/exclusion criteria have been previously defined and applied in the search process. The possibility to perform the same review by different reviewers, independently, and comparing the final results, ensure transparency to this scientific method. The unsystematic approach focuses on taking notes, yet gives limited indications on how those notes should be taken to ensure all the process is transparent. When something cannot be verified by peers, it is not scientific (Hollis, 1994; Hughes & Sharrock, 1997; Ladyman, 2002; Outhwaite, 2000). Reporting is better addressed in the systematic literature review process, focusing on the clarity of the report, on the effectiveness of its communication, and undertakes two types of reports: descriptive and thematic.

5

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

On the overall, systematic literature review is more precise and more effective, however should be improved in the topic definition phase, complemented with elements from the unsystematic literature review to ensure better comprehension of the topic under review. Towards a Methodology for Literature Reviews in Social Sciences: A Proposal When looking at a literature review from a methodological perspective, both systematic and unsystematic methods refer four general phases, which are to be developed sequentially (see Figure 1).

PLAN DO CHECK REPORT

• Define the topic • Plan the effective search

• Search for relevant information sources • Apply filters to ensure results are according to plan

• Check coherence of results with topics (thematic check) • Check search coherence (methodological check)

• Descriptive (process and overall statistical data on final results) • Thematic (themes that emerged from the review, linked with the topic, and their importance/potential) Figure 1 – Literature Review Methodology for Social Sciences: Four-fold Approach

At a methodological level, researchers should take decisions on how to systematically review the field, applying all criteria specified in the previous section for the systematic literature review. They should decide, based on the goals of their research and their final public, whether they wish to: 

Analyze published and/or unpublished information, and why;



Use digital and/or paper information, and why;



Search impact factor journals (databases like ISI Web of Knowledge) and/or lower-rank scientific journals, and why;



Analyze abstracts/introductions only or full-text information sources, and why;

6

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

Then, they should look deeper in the topic and understand how it can be searched effectively: 

What expressions are used in field to refer to the topic;



What keywords are associated to previous studies on the topic;



What databases contain the eligible information sources on the topic and which have a more practical export interface;



What search equations can be developed for the selected databases to perform the search effectively;



How it will be ensured the comparability of the results (if several databases are to be used).

Further, researchers should fill in the review protocol (see Appendix 1 for a model), justifying their decisions and developing instruments to measure and control the efficacy of the search. Then, search should be performed, applying the review protocol, registering steps and justifying decisions. Data should be extracted to an offline solution to be subject, after filtering and application of inclusion and exclusion criteria, to statistical analysis and content analysis. Finally, the descriptive and thematical reports should be produced and the methodological process of performing the review should all be documented and written down. Conclusions There is a clear need to approach the literature review process in a methodological way, so as to ensure the quality of the conceptual model and to allow researchers to effectively map the field they study and link their research to the relevant schools of thought. Insofar, only rarely do researchers speak of the methodology they use to perform their literature reviews. While lack of knowledge on how to use systematic literature reviews or meta-analyses may be one explanation for this fact, this can be no motive to avoid rigor and transparency in their research, which includes rigor and transparency in their literature review phase. No researcher should lack focus, clarity and reasoning, nor when defining the problem and coining the question for his/her research, nor when performing the literature review. Science is not a relative, subjective quest for “relevant” scientific papers, based on feelings and intuition. Science is a rigorous process of critical thinking, implying planning (method), doing (applying

7

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

method), checking (applying validity criteria) and reporting (presenting the research process, data and results in a transparent, clear manner). The quality of science draws strongly on the quality of the literature review, which needs to be planned (defining the method), performed rationally (according with plan and registering all steps/decisions), verified objectively (applying validity criteria) and reported effectively (transparent communication, descriptive and thematic reports).

Bibliography Amato, L., Davoli, M., Vecchi, S., Ali, R., Farrell, M., Faggiano, F., . . . Chengzheng, Z. (2011). Cochrane systematic reviews in the field of addiction: What's there and what should be. Drug and Alcohol Dependence, 113(2-3), 96-103. doi: 10.1016/j.drugalcdep.2010.08.003 BestBETs. BETs Search Strategies strategies.php

Retrieved October 23, 2011, from http://www.bestbets.org/links/search-

Bohman, J. (1991). New philosophy of social science: Problems of indeterminacy. Clarkson, J. E., & Bonetti, D. (2008). Dissemination of Cochrane Resources Beyond the Library. Journal of Evidence Based Dental Practice, 8(3), 195-202. doi: 10.1016/j.jebdp.2008.06.002 Coster, S., & Norman, I. (2009). Cochrane reviews of educational and self-management interventions to guide nursing practice: A review. International Journal of Nursing Studies, 46(4), 508-528. doi: 10.1016/j.ijnurstu.2008.09.009 Ferri, M., Davoli, M., & Perucci, C. A. (2006). Heroin maintenance treatment for chronic heroin-dependent individuals: A Cochrane systematic review of effectiveness. Journal of Substance Abuse Treatment, 30(1), 63-72. doi: 10.1016/j.jsat.2005.09.003 Francis, G. (2006a). The Cochrane Collaboration is in its fourteenth year. Joint Bone Spine, 73(3), 236-238. doi: 10.1016/j.jbspin.2006.01.002 Francis, G. (2006b). La démarche Cochrane a plus de 13 ans. Revue du Rhumatisme, 73(5), 415-417. doi: 10.1016/j.rhum.2005.10.016 Greenhalgh, T. (1997). How to read a paper: Papers that summarise other papers (systematic reviews and metaanalyses). BMJ, 315(672). Grimes, D. A., Lopez, L. M., Manion, C., & Schulz, K. F. (2007). Cochrane systematic reviews of IUD trials: lessons learned. Contraception, 75(6, Supplement), S55-S59. doi: 10.1016/j.contraception.2006.12.004 Hart, C. (2006). Doing a literature review: Releasing the social science research imagination: Sage. Hollis, M. (1994). Philosophy of social science. The Blackwell Companion to Philosophy, 375-402. Hughes, J. A., & Sharrock, W. W. (1997). The philosophy of social research. Kofinas, A., & Saur-Amaral, I. (2008). 25 years of knowledge creation processes in pharmaceutical industry: contemporary trends. Comportamento Organizacional e Gestão, 14(2), 257-280. Ladyman, J. (2002). Understanding philosophy of science: Psychology Press. Macpherson, A., & Holt, R. (2007). Knowledge, learning and small firm growth: A systematic review of the evidence. Research Policy, 36(2), 172-192. doi: 10.1016/j.respol.2006.10.001

8

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

Mulrow, C. D. (1994). Systematic reviews: rationale for systematic reviews. Bmj, 309(6954), 597. Neilson, J. P. (1998). Evidence-based intrapartum care: evidence from the Cochrane library. International Journal of Gynecology & Obstetrics, 63, Supplement 1(0), S97-S102. doi: 10.1016/s0020-7292(98)00190-8 Outhwaite, W. (2000). The philosophy of social science. The Blackwell companion to social theory, 1, 47. Riley, R. D., Gates, S., Neilson, J., & Alfirevic, Z. (2011). Statistical methods can be improved within Cochrane pregnancy and childbirth reviews. Journal of Clinical Epidemiology, 64(6), 608-618. doi: 10.1016/j.jclinepi.2010.08.002 Saur-Amaral, I. (2010). Revisão sistemática da literatura. Lisboa: Bubok. Saur-Amaral, I. (2011). Revisão sistemática da literatura com Endnote X4. Lisboa: Bubok. Saur-Amaral, I., & Amaral, P. (2010). Contract innovation organisations in action: doing collaborative new product development outside the firm. International Journal of Technology Intelligence and Planning, 6(1), 42-62. Saur-Amaral, I., & Borges Gouveia, J. J. (2007). Uncertainty in drug development: insights from a Portuguese firm. International Journal of Technology Intelligence and Planning, 3(4), 355–375. Sekaran, U., & Bougie, R. (2000). Research methods for business (5th ed.). Sussex: Wiley. Thorpe, R., Holt, R., Macpherson, A., & Pittaway, L. (2005). Using knowledge within small and medium sized firms: A systematic review of the evidence. International Journal of Management Reviews, 7(4), 257-281. Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14(3), 207-222. Tranfield, D., & Mouchel, D. D. (2002). Developing an evidence-based approach to management knowledge using systematic review.

9

Investigação e Intervenção em Recursos Humanos 2011 – gestão para a cidadania Escola Superior de Estudos Industriais e de Gestão do Instituto Politécnico do Porto 27 e 28 de Outubro de 2011

Appendix 1 – Example of a review protocol for systematic literature review Components

Content and rationale

Research goal

Why is the review being performed? To what public it will be presented?

Research topic

What is the topic under analysis? How is it addressed in the literature? What alternative expressions are used to refer to the topic?

Keywords

What keywords are used to refer to the topic?

Research scope

Where it will be done the search? What databases and why? What search characteristics need to be taken into account in each selected database?

Search equations

Search equations for each selected database (online). Include the field where the equation is to be applied (e.g. Title, Abstract etc.)

Technical concerns

How will be filtered the results in each database? How will be exported the results for descriptive and thematical analysis and reporting?

Inclusion/exclusion criteria

How will be identified and selected the results that correspond to the topic under analysis? Specify filters to apply (e.g. publication years, language, disciplinary area) and minimum criteria to fulfill (e.g. need to have an abstract and be published in scientific journals, no conferences or book will be taken into account)

Quality and validity criteria

How will be ensured that all results are relevant and no relevant result has been eliminated during the search process? How will be applied the inclusion/exclusion criteria and why? How many researchers will do the search independently and how will results be compared and differences treated to ensure internal validity? How will be registered all steps during the search process?

Data extraction

How will results be extracted and how it will be ensured that no result is affected in the transfer process from the online database to an offline solution?

10