Evaluation of policies and programs together with a culture which learns from success and failure is pivotal in producing evidence for effective policy development and public sector management.
This brief outlines research prepared for the Independent Review of the APS on evaluation and learning from failure and success. The research takes a historical approach to frame the issues for the APS Review panel and suggests a number of reform directions for their consideration.
In ANZSOG research commissioned for the APS Review, Rob Bray (ANU), Matthew Gray (ANU) and Paul ‘t Hart (Utrecht University) explore the following questions:
The paper concludes the APS currently does not learn well from experience and its approach to evaluation is piecemeal both in scope and quality. The issue is not one of skills and capacity but first and foremost, one of culture and institutional practices.
The research paper is informed by extensive literature reviews and consultations with experts in the field, including current and past senior public servants.
A key issue is how evaluation and learning activities relate to the principle and practice of accountability. Is evaluation simply framed in the context of the government goals for programs and reports to government? Or is there is a broader accountability role for evaluation and if so, how this can be fitted into the structure of the APS? The paper finds the APS has a piecemeal approach to evaluation. This diminishes accountability and is a significant barrier to evidence-based policy-making.
In their day-to-day activities, departments and agencies tend to focus on achieving the ‘here and now’ priorities of program and policy implementation. In examining their past performance, departments and agencies are often more concerned with reputational risk instead of learning from experience and feedback. Evaluation is often seen as a second-order and lower-priority issue.
The problem is not a lack of skills and capacity of public servants. It is a product of the environment in which the APS operates and cultural practices. This is exacerbated by the lack of an institutional framework that embeds the strategic importance of institutional learning.
The APS would take a rigorous evidence-informed approach to designing and implementing policies and programs, and providing advice to government. This would be grounded in:
The effectiveness of programs and policies would be tested against clearly articulated program objectives as well as a systematic evaluation framework. This framework would:
As to who evaluates, the paper sees this question as being related to the broader question of the accountability framework in which evaluation is undertaken. While most evaluations would still be undertaken within departments, the paper suggests a need for much stronger central coordination and leadership.
This centralised evaluation role would be both at an APS-wide level and in departments. An APS-wide function would:
The paper canvasses two options for the centralised evaluation function – within the current APS structure or as an independent statutory agency accountable to Parliament.
Within the wider framework of departmental learning, evaluation is just one of a range of inputs. Organisational learning can also derive from:
Other options to mine the learning potential of both successes and failures include:
Drawing these inputs together is an often neglected function, and would be supported by departments developing custodians of these learnings.
Underpinning these practices would be effective information systems, providing quality and timely data to departments and evaluators.
Policy and program failures should not be approached from a perspective of allocating blame, or successes from the perspective of reputation-enhancement and credit-claiming. Both successes and failures could instead be treated as sources of data, insight and lesson-drawing.
Evaluation and learning would become an integral element of the APS accountability framework, including:
While the APS will continue to work in the era of the 24-hour news cycle, demands for instant grabs and a focus of ‘gotcha’ reporting, a systematic approach to policy and accountability will reinforce the capacity and confidence of the APS to ‘stand its ground’ with authority.
This article originally appeared in The Mandarin.