Impact Evaluation in Student Equity

This stream will suit equity professionals working in Australian universities.

This stream comprises 5 modules, (modules 12-16 of the ACSES Evaluation Learning Program).

Module 12: Introduction to Impact Evaluation

Overview

This module introduces the concept of Impact Evaluation and explains the difference between quantitative and theory-based impact evaluation (QIE and TBIE), and the different questions each seek to answer. It discusses the conditions required for effective impact evaluation (as distinct from Continuous Quality Improvement) and how suitable student equity programs can be selected for impact evaluation. The purpose and benefits (outcomes) of impact evaluation, including the importance of sharing and reviewing Impact Evaluation Reports for other student equity programs, is also highlighted.

Learning Outcomes

At the completion of this module Learners will:

• Understand what is meant by Impact Evaluation, Quantitative Impact Evaluation and Theory-based Impact Evaluation.
• Understand how quantitative and theory-based impact evaluation differ, and the different questions they each seek to answer.
• Be able to assess student equity program suitability for impact evaluation.

Module 13: Quantitative Impact Evaluation

Overview

This module provides an understanding of quantitative impact evaluation (QIE), providing examples of various quantitative approaches, including trial and quasi-experimental designs. It covers in detail the fundamental concept of counterfactuals or comparator groups, and the ways in which such groups can be created or constructed, with reference to ethical considerations. Learners will gain understanding of the specific conditions and program parameters that lend themselves to quantitative impact evaluation methodologies, with examples of various QIE approaches used to student equity program evaluation provided. Key components of a QIE evaluation report are outlined.

Learning Outcomes

At the completion of this module Learners will:

• Understand what is meant by Quantitative Impact Evaluation and the potential benefits for student equity programs and practice.
• Understand the role of counterfactuals/comparison groups in conducting quantitative impact evaluation.
• Understand the limitations of quantitative impact evaluation.

Module 14: Theory-based Impact Evaluation

Overview

This module discusses theory-based impact evaluation (TBIE) and the important role such approaches can play in determining causal links between specific interventions and student equity outcomes. It introduces some common methodologies used in student equity program TBIE and the different questions each seek to answer. The specific conditions and program characteristics that lend themselves to TBIE are presented, alongside discussion of TBIE limitations and guidance on when Quantitative Impact Evaluation may be more appropriate. An outline of the key components of a TBIE report is also provided.

Learning Outcomes

At the completion of this module Learners will:

• Understand what is meant by Theory-Based Impact Evaluation.
• Understand the role and limitations of theory-based impact evaluation.
• Be aware of some different approaches to theory-based impact evaluation, and the questions they seek to answer.

Module 15: Planning for Impact Evaluation

Overview

This module discusses the key steps and considerations in preparing for an impact evaluation. It first walks practitioners through the process of assessing programs for impact evaluation, using the Impact Evaluation Prioritisation Tool. Having identified suitable student equity programs to be evaluated, the module then discusses how practitioners should go about determining appropriate methodologies and consider commissioning qualified internal or external evaluation specialists to undertake impact evaluation. It also discusses the importance of appropriately resourcing and planning for the management of impact evaluations, and ways in which evaluations can we socialised and shared.

Learning Outcomes

At the completion of this module Learners will:

• Be able to assess student equity program suitability for impact evaluation and determine whether QIE or TBIE should be pursued.
• Understand key considerations for effective impact evaluation, including resource allocation.
• Understand when to commission internal or external evaluation specialists to conduct impact evaluation.

Module 16: Data for Impact Evaluation

Overview

This module introduces data requirements for conducting effective impact evaluation of student equity programs, with a focus on internal data systems and structures. It considers how 'success' may be measured/quantified for various student equity activities and programs, and some sources for data on intending primary/secondary outcomes. Mechanisms for linking data related to program participants collected via CQI activities with broader university outcomes data are outlined, with examples given.

Learning Outcomes

At the completion of this module Learners will:

• Understand the data and data linkage requirements for conducting effective quantitative impact evaluation.
• Understand the challenges faced in evaluating impact where participant data is not captured in university data systems, and how to mitigate these.
• Know some of the ways by which a counterfactual may be defined in practice, and comparisons with target (participant) populations made to estimate the impact of programs or activities.

Once you successfully completed this stream, you'll earn a digital badge that is instantly shareable to your social networks (including LinkedIn) which showcases your new skills and knowledge mastery.

Engage

This digital badge provides foundation knowledge in a discipline and doesn't require previous knowledge.

The Australian Centre for Student Equity and Success (ACSES) is an evidence-based research and public policy centre funded by the Australian Government Department of Education and based at Curtin University.

ACSES is committed to providing the higher education sector with tangible proof of what works to improve the quality and impact of student equity practices in higher education.

For more information, visit acses.edu.au.

For more information on this stream, or any other questions about the Program, email us at acses@curtin.edu.au