Strategy for community engagement evaluation
(1) Evaluation as part of performance monitoring and reporting | (2) Evaluation as part of project management and development | (3) Evaluation as part of developing a shared evidence and knowledge base | Key outputs of the Evaluation Strategy
This evaluation strategy recognises three key roles for community engagement evaluation in Government:
- as part of performance monitoring and reporting for public sector accountability at the project, departmental and Whole of Government level;
- as part of project and program management and development and on-going project and program improvement; and
- for developing a shared evidence and knowledge base for good community engagement practice.
These roles are not mutually exclusive, rather evaluation, particularly at the project and program level, should attempt to recognise and provide for all three purposes. However, the scale and scope of evaluation should reflect:
- the purpose for which the evaluation will be used and the intended audience of the evaluation; and
- the scale and significance of the project or reporting level.
(1) Evaluation as part of performance monitoring and reporting
Evaluation as part of performance monitoring and reporting is generally referred to as summative evaluation or evaluation for accountability. This type of evaluation is used to report on achievements, increase accountability, and track progress.
Evaluation for the purpose of performance monitoring and reporting is usually focused on reporting on productivity and 'outputs' and on the achievement of certain objectives or outcomes. It may also be concerned with certain process requirements or practice principles being met where relevant.
Summative evaluation generally relies on measures or indicators, which can be used to report progress. These can be objective measures of quantity or subjective measures of quality. Indicators can also be classified according to whether they measure process, outputs, or outcomes. In some cases, the term 'impact' indicator is used to refer to measures of the short to medium term effects of interventions and the term 'outcome' indicator is reserved for measures of longer term effects.
In general, it is much easier to find ways to measure productivity and outputs, and subjective measures of the quality of those outputs, than it is to find ways to measure the longer term outcomes of activities. However, most government community engagement activities are focused on improving broader government outcomes, therefore, a best attempt needs to be made to measure the effects of government community engagement activities in those processes.
An important first step is to be clear about the intended economic, social and environmental outcomes of a project, program or policy. These outcomes also include immediate community engagement impacts, such as increased levels of community awareness or input of information from the community into a planning process; or longer term community engagement outcomes, such as improvements in the relationships between government and community, as well as improvements to the programs and policies that the community engagement process is feeding into.
The development of an outcomes hierarchy can assist this process. An outcomes hierarchy guides the planning and evaluation of activities by making explicit how a program is expected to work. This is done by identifying the cause and effect chains that lead between the immediate intended community engagement project outcomes and the desired broader government community engagement outcomes. An example of an outcomes hierarchy is provided in Figure 1.
Figure 1. Example of an outcomes hierarchy for a community engagement online consultation website project.
Performance monitoring and reporting occurs at three levels:
- micro-level: project/ program;
- macro-level: divisional/ department; and
- mega-level: whole of government.
The role of summative evaluation in these three reporting levels and some examples of common indicators, are presented below. The example indicators attempt to reflect a range of process, output, and outcome indicators; however, given the difficulty in identifying measures of outcomes, the weight is on process and output indicators.
Micro-level - Evaluation of community engagement project or program
Purpose is to monitor and track performance and the delivery of outputs and outcomes against individual project objectives and for the purpose of reporting against business and branch area business plans.
Requires individual performance indicators to be developed that are relevant to the objectives and context of the individual projects and programs, as well as the use of relevant indicators for reporting divisional/ departmental performance.
Macro-level - Division/Departmental performance (community engagement programs and policies)
Primary purpose is to evaluate divisional/departmental community engagement performance against the objectives in the Strategic Plan. Reporting on performance is often done through the Ministerial Portfolio Statement.
Requires common indicators that reflect the department's and the government's reporting framework and data availability. The current focus in the Ministerial Portfolio Statement and Department of Premier and Cabinet Strategic Plan is on Key Performance Indicators of quality, quantity and timeliness of outputs.
Mega-level - Whole of Government performance (community engagement programs and policies)
Primary purpose is to evaluate performance and report on the community engagement activities across Government (outputs) and to examine some of the presumed outcomes of community engagement activities at the Whole of Government level. This enables reporting in Priorities in Progress.
Evaluation at this level attempts to both measure levels of community engagement with the Queensland Government and tie together community engagement outputs with intended policy outcomes across a spectrum of government programs and policies. As part of measuring levels of community engagement, the Queensland Government has commissioned household survey data to provide benchmarks indicators of community engagement.
(2) Evaluation as part of project management and development
The second role for evaluation is as part of community engagement project management and development. This is generally referred to as formative evaluation or evaluation for development, and differs from summative evaluation in that the purpose of evaluation is not about reporting on performance to ensure accountability but rather to integrate learning and continuous improvement into the community engagement project/ program/ policy cycle.
Formative evaluation must be integrated throughout the project, program or policy cycle so that the results of evaluation can be used to improve the project/ policy development as it is on-going (continuous improvement).
Data collection may be based on qualitative data or indicators developed at the project/ program/policy planning stage. Ideally, indicators should be identified in collaboration with community representatives and other stakeholders. These will generally include both process and outcome indicators and be based on overall government community engagement objectives, as well as specific objectives for individual projects.
The data collection and analysis can be undertaken by: community engagement practitioners, ideally through a participatory monitoring and evaluation approach between practitioners and stakeholders, or alternatively by external evaluators, who can also use a participatory research approach. Overall, the scale of the evaluation and the resources committed to the evaluation process should reflect the scale and importance of the activity. In the case of small-scale projects, limited data collection by officers from the relevant department may be appropriate.
(3) Evaluation as part of developing a shared evidence and knowledge base
Evaluation can also have a role in developing a shared knowledge and evidence base on community engagement. Evaluation provides a unique opportunity to collect data that will be useful in achieving the objectives of the Research and Evaluation Agenda around:
- strengthening representative democracy;
- widening and deepening citizen participation; and
- building community capacity and public sector capability.
For example, in the Research and Evaluation Agenda one of the key research areas under the widening and deepening citizen participation theme is the evaluation of current models and processes for engagement (including models of eDemocracy):
- to improve our understanding of their scope (the purposes and issues for which they are appropriate); breadth (the number and types of people they engage with) and depth (the level of involvement people have in the activity) of involvement; and
- to examine the links between the context, methods, practice and outcomes of engagement, with particular attention to testing the current assumptions about best practice and the potential benefits of community engagement (such as building trust and confidence in government).
Case study evaluations could be designed to explore some of these research topics as well as achieve the objectives of performance monitoring and project/program development.
Key outputs of the Evaluation Strategy
- A set of indicators for measuring community
engagement performance across Government
The first key output of the strategy will be a suggested set of indicators for measuring community engagement performance across a range of sectors. Example project and program level indicators will be included in the community engagement evaluation guidebook. In addition, the Research and Evaluation Unit will help facilitate and contribute to the dialogue around Whole of Government community engagement monitoring and reporting. - A project-level community engagement evaluation guidebook
A project-level evaluation guidebook will be developed as part of the Community Engagement Implementation Strategy 'suite of resources'. This guidebook will focus on the three roles of evaluation while placing particular emphasis on the importance of integrating evaluation into community engagement planning and conducting formative evaluation for on-going project improvement. - Training materials and a training program to support the evaluation guidebook
Training materials and a training program will be developed on community engagement evaluation to support the information provided in the guidebook. These materials and the training program will both stand alone and be available for incorporation into the community engagement training initiatives being led by the Capability Development Unit. - Establishing a community engagement research and evaluation network across government and the tertiary sector
A network of community engagement research and evaluation practitioners and experts will be developed across government and the tertiary sector. This network will begin with the formation of a research and evaluation sub-committee of the Community Engagement Improvement Strategy Implementation Team, which includes senior representatives from all government agencies. - Developing a framework for the evaluation of the Community Engagement Improvement Strategy
A framework for the evaluation of the Community Engagement Improvement Strategy (CEIS) will be developed by the Research and Evaluation Unit and the Capability Development Unit, in conjunction with the CEIS Implementation Team. This framework will be used to report on the performance of the Strategy, provide information for future decision about public sector community engagement capability development needs, and contribute to the knowledge base around community engagement capability development.
Other languages