University of Alberta

Course Guide for SPH 631 Winter 2017 (Campus)

Health Program Evaluation Thursdays 0900-1150 hr ECHA- LEC 1-163

Instructor:

Dr. Zubia Mumtaz Phone: 780 492 7709 e-mail: [email protected] Office: 3309 ECHA

Course description

This course introduces the concept of program evaluation, its importance and use as a management, planning and policy formulation tool. It provides students with a theoretical and practical understanding of the different types of evaluations needs assessment, program theory, process evaluation, monitoring of outputs and outcomes, impact assessment, and cost analysis. A wide range of quantitative and qualitative methods including experimental and quasi-experimental study designs will be presented. Discussions will centre on the social and political dimensions of program evaluation with examples from industrialized and low- and middle-income countries (LMIC’s).

Objectives

Upon completion of this course, students will be able to: 1. Demonstrate an understanding of the importance and use of health program evaluations. 2. Describe the key aspects and types of program evaluation. 3. Identify and interpret the principles underlying the selection of appropriate strategies, methods and data sources necessary for evaluating a health program. 4. Be able to carry out a program evaluation designed to guide management, planning and policy decisions.

Competencies

SPH 631 2017 Course Syllabus

• Describe key concepts in monitoring and evaluation • Discuss and differentiate program evaluation, policy evaluation, evaluation research • Explain/create a program evaluation plan • Initiate and guide evaluation activities, in response to specific community, health program and management needs • Demonstrate ability to match evaluation questions with appropriate methodologies and approaches • Describe/explain the range of program evaluation methods that generate knowledge to inform program policy development, along with strengths and limitations and appropriate use of each. • Identify major sources of existing health data (e.g., local, provincial/state, national and international sources of health data). • Assess relevance and quality of evidence from a range of sources and apply to a specific public health, or management issue University of Alberta, School of Public Health

2

• Identify and communicate potential and /or common pitfalls in using various research methods in program evaluation

This course employs an active learning approach with learning strategies that include whole class lectures, group discussions, individual and small group exercises, student presentations, and written assignments. This course is team-taught. The instructors have different academic and professional backgrounds and they bring their unique perspectives to the course. Read their biographical sketches in the School’s website. Readings are assigned for each session. The first one in the list, enclosed in a box, is a “must-read”. While you are not “tested” on the readings, those who have done them will benefit the most from the lectures and be able to participate fully in the class discussions.

Library

The University of Alberta library system’s website www.library.ualberta.ca details the range of services offered to students on and off campus. If you need further information or assistance, contact the Library's Electronic Reference Desk at www.library.ualberta.ca/ereference/index.cfm or call 1-800-2070172.

SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

3

Course evaluation Mid-Point Course Evaluation:

Final Course Evaluation:

As is the case with all SPH instructors, we are interested in improving the course. We need to hear from you in order to do that well, so will gather feedback and recommendations from you at the mid-point of the course, by asking a student to administer and deliver an anonymous course evaluation.

Following completion of the course, you will receive a standardized summative evaluation. The standard University of Alberta course evaluation procedure will be followed. This will involve a standard questionnaire with anonymous responses returned to the Student Services Coordinator.

Student Evaluation Assignment Health Program presentation End of term examination A Program Evaluation Proposal Classroom Participation

Percentage 5%

Date 19th Jan, 2017

35%

9.00 am, 13th April, 2017

50%

4.00 pm, 13th April, 2017,

10%

Whole term

Evaluation of Course Program Evaluation Proposal Work This assignment aims to develop your skills in program evaluation, specifically how to develop an evaluation proposal. It is designed to test your understanding of issues related to evaluation questions, types of evaluation, study design, stakeholders and to encourage you to think about problems arising in a program evaluation in a practical setting. This will be a group project. You will work in teams of four or five students. You will select a health care program, or a health promotion program, either a domestic program or an international program and develop a proposal for its evaluation. The assignment is divided into four parts that should parallel your understanding of ideas and issues as presented and discussed in class 1. 2. 3. 4.

SPH 631 2017 Course Syllabus

Part Part Part Part

I: Define the problem and describe the program II: Decide the type of evaluation III: Methods used IV: How will the evaluation results be used?

University of Alberta, School of Public Health

4

The project mark is a group mark. Each team member will present a part of the proposal (15%) and the each group will submit one writeup of the proposal (35%). The distribution of the presentation marks are as follows: out of a total of 15%, 5% will be a group mark and 10% will be based on individual presentation skills as per criteria laid out in the presentation score sheet attached. The distribution of the written proposal are as follows: the total of 35% will be distributed in four parts: Part 1- 10%, Part 2 - 10%, Part 3 - 10% and Part 4 - 5%. Class room participation Students will be expected to participate in classroom discussions supplemented by background or assigned readings. Some classroom participation may be done individually or in small groups. Students will be expected to apply the relevant reading material to the topic and questions at hand. Quality is preferred over the quantity of contribution, and participation should facilitate group, as well as individual learning. Participation grades will be assigned based on an assessment of your preparation and contributions, as well as attendance in class. 10/10 •Attends regularly •Comes prepared, having read the assigned readings. •Comes with highlighted readings or written responses to the discussion questions to be shared with classmates but not handed in to the instructors. •May have done additional reading on the subject (optional readings or identified own sources of information), or makes an effort to relate the required reading to previous learning or work experience. •Contributes relevant comments, or asks questions during class. •Does not dominate whole class or small group discussions. Allows others to participate in the discussions. •Volunteers to act as spokesperson for small group feedback.

Grading

See ‘Assignments and Marking Criteria’.

University of Alberta Calendar Section 23.4 Regulations and Information for Students Evaluation Procedures and Grading System The University of Alberta Grading System The University of Alberta uses a letter grading system with a four-point scale of numerical equivalents for calculating grade point averages. SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

5

Grades reflect judgments of student achievement made by instructors. These judgments are based on a combination of absolute achievement and relative performance in a class. Some instructors assign grades as intervals during the course and others assign marks (e.g. percentages) throughout the term and then assign a letter grade at the end. Instructors must adapt their approaches to reflect the letter grading system. Grade distribution should reflect those shown in this document. (EXEC 03 FEB 2003) Descriptor Excellent Good Satisfactory Failure

Academic Integrity

SPH 631 2017 Course Syllabus

Grading in Graduate Courses Letter Grade Grade A+ A AB+ B BC+ C CD+ D F

Point Value 4.0 4.0 3.7 3.3 3.0 2.7 2.3 2.0 1.7 1.3 1.0 0.0

Plagiarism is a serious offence. The University of Alberta is committed to the highest standards of academic integrity and honesty. Students are expected to be familiar with these standards regarding academic honesty and to uphold the policies of the University in this respect. Students are particularly urged to familiarize themselves with the provisions of the Code of Student Behaviour (online at www.ualberta.ca/CodeofStudentBehaviour) and avoid any behaviour which could potentially result in suspicions of cheating, plagiarism, misrepresentation of facts and/or participation in an offence. Academic dishonesty is a serious offence and can result in suspension or expulsion from the University. University of Alberta policy about course outline can be found in Section 23.4(2) of the University Calendar. (GFC 29 SEP 2003)

University of Alberta, School of Public Health

6

SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

7

Schedule at a Glance Date/2017 Jan 12 Jan 19 Jan 26 Feb 2 Feb 9 Feb 16 Feb 23 Mar 2 Mar 9 Mar 16 Mar 23 Mar 30 Apr 6 Apr 13

Topic Foundations of Health Program Evaluation This is a ‘health program’ – student presentations Types of Program Evaluation Assessing the need for a Program Program Theory Process Evaluation Reading week Impact Evaluation Detecting, interpreting and analyzing program effects Economic Evaluation of Health programs Policy implications of Evaluation findings Making a difference: Ethics in Program evaluation Group Presentation of Program Evaluation proposals End of term exam and Final Program Evaluation proposals due – at 4.00 pm

SPH 631 2017 Course Syllabus

Instructor Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz Arto Ohinmaa Zubia Mumtaz Zubia Mumtaz Zubia Mumtaz

University of Alberta, School of Public Health

8

DETAILED COURSE OUTLINE BY WEEK WEEK 1 (January 12) – Introduction to SPH 631

Instructor: Zubia Mumtaz

Learning Objectives Overall Course objectives 1. Have an understanding of the importance and use of health program evaluations. 2. Describe the key aspects and types of program evaluation. 3. Understand the principles underlying the selection of appropriate strategies, methods and data sources necessary for evaluating a health program. Required Readings 1. Rossi, Peter, Lipsey, M. W. and Freeman, H. E. (2004) Evaluation: A systematic approach (7th edition). Newbury Park, CA: SAGE Publications, Chapter 1. 2. Chouinard, Jill Anne (2014). "Understanding relationships in culturally complex evaluation contexts." Evaluation 20.3 : 332-347. External Speaker: Jeanne Annett, Director Evaluation Services, Alberta Health Services.

WEEK 2 (January 19) This is a ‘Health Program’

Student presentations

Learning Objectives 1. Understand what is a ‘Health Program’ 2. Describe a program as a series of activities, supported by a set of resources, intended to achieve specific outcomes among particular target groups WEEK 3 (January 26) Types of Program Evaluation

Instructor: Zubia Mumtaz

Learning Objectives 3. To become familiar with a. Types of stakeholders; identification of stakeholder need. b. Nature of the evaluator-stakeholder relationship c. Politics of evaluation. 4. Understand types of program evaluation - needs assessment, program theory, assessment of program implementation, outcomes and impact, interpreting and analyzing program effects, measuring efficiency. 5. Identifying issues when formulating evaluation questions Required Readings SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

9

1. Abma, T.A. (2000). Stakeholder conflict: A case study. Evaluation and Program Planning. Vol 23: 199-210. 2. Vivianne E. Baur, Arnold H.G. Van Elteren, Christi J. Nierse, Tineke Abma (2010) Dealing with Distrust and Power Dynamics: Asymmetric Relations among Stakeholders in Responsive Evaluation 16(3) 233–248. 3. Dennis M. Gorman and Eugenia Conde (2007) Conflict of interest in the evaluation and dissemination of “model” school-based drug and violence prevention programs. Evaluation and Program Planning 30: 422-429. 4. Vivianne E. Baur, Tineke A. Abma, Guy A.M. Widdershoven (2010) Participation of marginalized groups in evaluation: Mission impossible? Evaluation and Program Planning 33: 238–245. 5. John M. Bryson, Michael Quinn Patton, Ruth A. Bowman (2010) Working with evaluation stakeholders: A rationale, step-wise approach and toolkit Evaluation and Program Planning 34 (2011) 1–12. Movie: The Silence Within Week 4 (February 02) Assessing the need for a program

Instructor: Zubia Mumtaz

Learning Objectives • • •

Gain a better understanding of the ways of defining the problem Become familiar with different sources of data Learn to validate and prioritize identified needs

External speaker: Birgitta Larsson, Principal, BIM Larsson& Associates Required Reading 1. Hawe, Penelope (2015). "Minimal, negligible and negligent interventions." Social Science & Medicine. 2. Lockwood A, Marshall M (1999) Can a standardized needs assessment be used to improve the care of people with severe mental disorders? A pilot study of 'needs feedback' Journal of Advanced Nursing 30 (6): 1408-1415. 3. Muela S., Ribera J. and Tanner M (1998) Fake malaria and hidden parasites-the ambiguity of malaria. Anthropology and Medicine 5(1)43-61.

SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

10

Week 5 (February 9) Program theory

Instructor: Zubia Mumtaz

Learning Objectives 1. Be able to express program theory- what it is and what it does. 2. Develop and use logic models 3. Assess program theory in relation to needs, its logic and plausibility. Required Reading 1. Chris L. S. Coryn, Lindsay A. Noakes, Carl D. Westine and Daniela C. Schröter (2011) A Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009. American Journal of Evaluation 2011 32: 199. 2. Harting J and Van Assema P (2010) Exploring the conceptualization of program theories in Dutch community programs: a multiple case study. Health Promotion International, Vol. 26 No. 1 Doi:10.1093/heapro/daq045 3. Gugiu C.P and Rodriguez_Campos L (2007) Semi structured interview protocol for constructing logic models. Evaluation and Program Planning 30: 339-350. 4. Kaplan S. and Garrett K. (2005). The use of logic models by community-based initiatives. Evaluation and Program Planning 28:167-172. External Speaker: Jeanne Annett, Director Evaluation Services, Alberta Health Services. Week 6 (February 16) Process evaluation

Instructor: Zubia Mumtaz

Learning Objectives 1. Understand the principles of process evaluation 2. Understand the different methods and data sources that enable a process evaluation.

Required Reading 1. Andre Maiorana, Susan Kegeles, Percy Fernandez, Ximena Salazar, Carlos Cáceres, Clara Sandoval, Ana María Rosasco and Thomas Coates (2007). Implementation and evaluation of an HIV/STD intervention in Peru. Evaluation and Program Planning 30: 82-93 2. Ruth P. Saunders, Dianne Ward, Gwen M. Felton, Marsha Dowda and Russell R. Pate (2006) Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP). Evaluation and Program Planning 29: 352-364 External Presenter: Jeanne Annett, Director Evaluation Services, Alberta Health Services

Week 7 – February 23 Reading week SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

11

Week 8 (March 2) Impact evaluation

Instructor: Zubia Mumtaz

Learning Objectives 1. Understand key concepts in impact assessment 2. Appreciate the importance, but the simultaneous limitations of randomized field experiments in assessing program impacts. Required Reading 1. Copestake, James (2014). "Credible impact evaluation in complex contexts: Confirmatory and exploratory approaches." Evaluation 20.4 : 412-427. 2. Green, Judith, et al. "Integrating quasi-experimental and inductive designs in evaluation: A case study of the impact of free bus travel on public health." Evaluation 21.4 (2015): 391-406.. 3. Victora, CG, J-P Habicht & J Bryce (2004) Evidence-based public health: moving beyond randomized trials. American Journal of Public Health 94(3): 400-405. 4. Smith, GCS & JP Pell (2003) Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized control trails. BMJ 327:1459-61. Exercise: El Arifeen S, Blum LS, Hoque DME, et al. (2004). Integrated management of Childhood illness (IMCI) in Bangladesh: early findings from a cluster-randomised study. LANCET 364: (9445): 1595-1602

Week 9 (March 9) Detecting, interpreting and analyzing program effects Instructor: Zubia Mumtaz Learning Objectives • • •

Become familiar with quasi-experimental impact assessment methods Gain an understanding of the limitations of the quasi-experimental study designs Develop a critical understating of the bases and errors when detecting, interpreting and analyzing program effects

Required Reading 1. Tones K (1997). Beyond the randomized controlled trial: a case for ‘judicial review’. Health Education Research, 12:1-4. 2. Habicht, J-P, CG Victora & JP Vaughan (1999) Evaluation designs for adequacy, plausibility and probability of public health program performance and impact. International Journal of Epidemiology 28(10-18). 3. Audrey E Pettifor, Catherine MacPhail, Stefano Bertozzi and Helen V Rees (2007) Challenge of evaluating a national HIV prevention program: the case of loveLife, South Africa. Sexually Transmitted Infections 2007;83:i70-i74. Published online 7th February 2007 at http://sti.bmj.com.login.ezproxy.library.ualberta.ca/cgi/content/full/83/suppl_1/i70 SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

12

Debate: Randomized controlled trails vs. Quasi-experimental impact assessment methods.

Week 10 (March 16) Economic evaluation of health programs Instructor: Dr Arto Ohinmaa) Learning Objectives • • •

Become familiar with the key concepts in efficiency analysis Be able to conducting a program cost-benefit analysis Appreciate the limitations of conducting cost-effective analysis in health program evaluations

Required Reading 1. Wasem J. (2007) Evaluation of prevention: A challenge for economists. International Journal of Public Health 52:337-338 2. Palmer, S., Byford S., and Raftery J (1999) Economic Notes: Types of economic evaluation. British Medical Journal 318:1349 3.Schwappach D. (2007) The economic evaluation of prevention - Lets talk about values and the case of discounting. International Journal of Public Health 52:335-336 4. Shiell A. (2007) In search of social value. International Journal of Public Health 52:333-334 5. Raftery J (2000) Economics notes: costing in economic evaluation. British Medical Journal. 320:1597.

Week 11 (March 23) Policy implications

Instructor: Zubia Mumtaz

Learning Objectives  

Understand the importance of dissemination and diffusion activities Develop a critical understanding of the issues of policy significance, o Of political time and evaluation time o Of policy space

Required Reading 1. Petticrew, M. et al (2004) Evidence for public health policy on inequalities: 1: The reality according to policy makers. Journal of Epidemiology and Community Health, Oct 2004; 58: 811 - 816. 2. Whitehead, M et al (2004) Evidence for public health policy on inequalities: Assembling the evidence jigsaw. Journal of Epidemiology and Community Health, Oct 2004; 58: 817 - 821. 3. Bjørnholt, Bente, and Flemming Larsen. "The politics of performance measurement:‘Evaluation use as mediator for politics’." Evaluation 20.4 (2014): 400-411. External Speaker: Consultant SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

13

Week 12 (March 30) Making a difference: ethics in program evaluation Instructor: Learning Objectives • • •

Gain an understanding of the importance of ethics in program evaluation Develop a critical understanding of value judgments Be able to apply the principle of ‘doing ethics’: Trustworthiness, justice, moral pluralism and cultural dialogue.

Required Reading 1. Newman D. and Brown R (1996) Applied ethics for program evaluation. Sage Publication Thousand Oaks , London New Delhi. Chapter 1. 2. Thomas A. Schwandt (2007) Expanding the conversation on evaluation ethics. Evaluation and Program Planning 30: 400-403

SPH 631 2017 Course Syllabus

University of Alberta, School of Public Health

14