Impact Evaluation in Student Equity

This stream will suit equity professionals working in Australian universities.

This stream comprises 5 modules designed for intermediate to advanced learners, who have some experience and understanding of the SEHEEF and are directly involved in impact evaluation projects or are looking to determine if programs are suitable for impact evaluation, and the types of evaluation approaches/methods that would be suitable.

Module 12: Introduction to Impact Evaluation

Overview

This module introduces the concept of Impact Evaluation and explains the difference between quantitative and theory-based impact evaluation, and the different questions they seek to answer. It discusses the conditions required for effective impact evaluation (as distinct from CQI) and how to select suitable student equity programs for impact evaluation, using the Impact Evaluation Prioritisation tool. The purpose and benefits (outcomes) of impact evaluation, including the importance of sharing and reviewing Impact Evaluation Reports for other student equity programs, are also highlighted.

Learning Outcomes

At the completion of this module Learners will:

• Understand what is meant by Impact Evaluation, Quantitative Impact Evaluation and Theory-based Impact Evaluation.
• Understand how quantitative and theory-based impact evaluation differ, and the different questions they each seek to answer.
• Be able to assess student equity program suitability for impact evaluation.

Module 13: Quantitative Impact Evaluation

Overview

This module discusses quantitative impact evaluation (QIE) in detail. It presents the key elements of quantitative impact evaluation, including some example quantitative data collection and analysis methodologies used in student equity program impact evaluation. Randomised Control Trials (RCTs), counterfactuals and quasi-experimental design are explained, with reference to ethical evaluation principles (see Module 3: Ethical Evaluation) and examples provided for different types of RCTs applied in student equity program evaluation. The module also discusses the specific conditions and program parameters that lend themselves to quantitative methodologies, or otherwise, and the key components of a QIE report are outlined.

Learning Outcomes

At the completion of this module Learners will:

• Understand what is meant by Quantitative Impact Evaluation and the potential benefits for student equity programs and practice.
• Understand the role of counterfactuals/comparison groups in conducting quantitative impact evaluation.
• Understand the limitations of quantitative impact evaluation.

Module 14: Theory-based Impact Evaluation

Overview

This module discusses theory-based impact evaluation (TBIE) and the important role it plays in determining the causal links between specific interventions and student equity outcomes. It introduces some common methodologies used in student equity program TBIE, the different questions they seek to answer, and highlights the value offered by mixed methods approaches involving both quantitative and qualitative data. The conceptual and empirical phases of TBIE are discussed, and key terms such as Theory of Change are explained. The specific conditions and program parameters that lend themselves to TBIE are presented, alongside discussion of TBIE limitations and guidance on when Quantitative Impact Evaluation may be more appropriate. An outline of the key components of a TBIE report is also provided.

Learning Outcomes

At the completion of this module Learners will:

• Understand what is meant by Theory-Based Impact Evaluation.
• Understand the role and limitations of theory-based impact evaluation.
• Be aware of some different approaches to theory-based impact evaluation, and the questions they seek to answer.

Module 15: Planning for Impact Evaluation

Overview

This module discusses the key steps and considerations in preparing for impact evaluation. It first walks practitioners through the process of assessing programs for impact evaluation, using the Impact Evaluation Prioritisation Tool. Having identified suitable student equity programs to be evaluated, the module then discusses how practitioners should go about determining appropriate methodologies and consider commissioning qualified internal or external evaluation specialists to undertake impact evaluation. It also discusses the importance of appropriately resourcing and planning for the management of impact evaluations, and sharing applicable outcomes, including through the Trials Registry.

Learning Outcomes

At the completion of this module Learners will:

• Be able to assess student equity program suitability for impact evaluation and determine whether QIE or TBIE should be pursued.
• Understand key considerations for effective impact evaluation, including resource allocation.
• Understand when to commission internal or external evaluation specialists to conduct impact evaluation.

Module 16: Data for Impact Evaluation

Overview

This module introduces the data requirements for conducting effective impact evaluation of student equity programs, with a focus on internal data systems and structures. It considers how 'success' may be measured/quantified for various student equity activities and programs, and where data on intended outcomes may be sourced. Mechanisms for linking data related to program participants collected via CQI activities with broader university systems and outcomes data are outlined, with relevant examples given. The module also discusses the specific challenges faced in linking project level and outcome data for pre-access/access programs.

Learning Outcomes

At the completion of this module Learners will:

• Understand the data and data linkage requirements for conducting effective quantitative impact evaluation.
• Understand the challenges faced in evaluating impact where participant data is not captured in university data systems, and how to mitigate these.
• Know some of the ways by which a counterfactual may be defined in practice, and comparisons with target (participant) populations made to estimate the impact of programs or activities.

Once you successfully completed this stream, you'll earn a digital badge that is instantly shareable to your social networks (including LinkedIn) which showcases your new skills and knowledge mastery.

Engage

This digital badge provides foundation knowledge in a discipline and doesn't require previous knowledge.

The Australian Centre for Student Equity and Success (ACSES, formerly the National Centre for Student Equity in Higher Education) is an evidence-based research and public policy centre funded by the Australian Government Department of Education and based at Curtin University.

ACSES is committed to providing the higher education sector with tangible proof of what works to improve the quality and impact of student equity practices in higher education.

For more information, visit acses.edu.au.

For more information on this stream, or any other questions about the Program, email us at acses@curtin.edu.au