National Research and Developments
NHMRC additional levels of evidence and grades for recommendations for developers of guidelines, National Health and Medical Research Council, Canberra, 2003
All evidence is not created equal. The major problem confronting someone wanting to apply evidence based management is to determine the quality of the evidence you need and to recognise the quality of the evidence you have.
Over recent years, the Health Advisory Committee (HAC), a principal committee of the National Health and Medical Research Council (NHMRC) has developed a suite of handbooks to support organisations involved in the development of evidence-based clinical practice guidelines (www.nhmrc.gov.au/publications/synopses/cp65syn.htm).
The NHMRC has been reviewing the use of the principles and processes set out in the handbooks. It now appears that the current ‘levels of evidence’ may be too restrictive for use in public health and community service settings.
Grades of recommendations
A new grading system for recommendations has also been developed and the approach is detailed in the document “NHMRC additional levels of evidence and grades for recommendations for developers of guidelines” . In addition to the current NHMRC levels of evidence for intervention studies, this document provides new ‘interim’ levels of evidence for some common research questions. The results of each included study must be assessed in three dimensions:
- Strength of evidence
- Effectiveness of evidence is based on the probability that the design of the study has reduced or eliminated the impact of bias on the results
- Quality of evidence: each included study is critically appraised as to its methodological quality. The study is assessed according to the likelihood that bias, confounding factors and/or chance has influenced the results.
- Statistical precision: the primary outcomes of each included study are critically appraised to determine whether the effect is ‘real’ as opposed to being due to chance (using the p-value and/or confidence interval).
- Size of effect
This dimension is useful for assessing the clinical importance, as opposed to statistical significance, of the primary outcomes of each included study. This is calculated on the basis of the size of the effect and its corresponding 95% confidence interval
- Relevance of evidence
This dimension assesses the relevance of the results of each individual study with respect to:
- Outcomes: the appropriateness of the outcomes. Are they relevant to the patient?
- Population: are the outcomes of the study based on a similar population and therefore generalisable or applicable to the population of interest?
- Intervention: are the outcomes of the study a consequence of a similar intervention and therefore generalisable or applicable to the intervention of interest?
Feedback
The ‘interim’ levels of evidence and grading system for recommendations do not have official NHMRC status, but are being piloted until mid-2006 with feedback being sought until 30 June 2006 on their usability and applicability.
The NHMRC draft guideline is at: http://www.nhmrc.gov.au/consult/docfeedback.htm.
Spencer, L; Ritchie, J: Lewis, J; Dillon, L; (2003) Quality in Qualitative Evaluation; A framework for assessing research evidence, Government Chief Social Researcher’s Office
This report presents the findings of a study carried out by a team of researchers based at the National Centre for Social Research, on behalf of the Strategy Unit in the Cabinet Office. The objective of the study was to develop a framework which would guide assessments of the quality of qualitative research evaluations. The study was a response to the fact that, despite their growing use, there are no explicitly agreed standards regarding what constitutes quality in qualitative research evaluations.
The framework draws heavily on existing frameworks, on the wider literature and on the contribution of those who participated in the study. The development of the framework involved:
- a comprehensive review of the literature on qualitative research methods relating to standards in qualitative research
- a review of qualitative research methods used in government funded evaluation studies
- a review of existing frameworks for assessing quality in qualitative research
- exploratory interviews with a range of people who have an interest in quality assessment of qualitative research and/or policy-related evaluations. These included academics who have written about qualitative research from either a theoretical or empirical perspective; authors of existing frameworks; research practitioners; commissioners and funders; and policy-makers who have used qualitative research evidence in the development and evaluation of policies
- a workshop, involving the above groups, to refine the framework initially developed
- a trial application of the framework to a small number of studies
The Framework includes four guiding principles, 18 appraisal questions as well as quality indicators.
The Framework can be accessed at: http://www.policyhub.gov.uk/docs/qqe_rep.pdf
O’Dwyer L, A Critical Review on Evidence-Based Policy Making, Australian Housing and Urban Research Institute (AHURI), Melbourne, Southern Research Centre, May 2004
This review has the following objectives:
- To show how evidence-based policy is distinct from other policy
- To describe how it is formulated and developed
- To show how the outcomes of academic research are communicated to policy makers
- To report how policy makers receive and interpret the outcomes of specific types of research
- To find why evidence-based policy has become prominent in a range of portfolios in other countries, but is largely restricted to health in Australia.
The material under review is largely of a conceptual or theoretical nature rather than empirical research and so the review itself is an amalgamation of the strengths of the narrative review and the systematic review formats.
Evidence-based public policy is based on research that has undergone some form of quality assurance and scrutiny. This distinguishes it from public policy based on more conventional policy development processes where intuitive appeal, tradition, politics, or the extension of existing practice may set the policy agenda. The assumption that evidence itself is a good thing, that it is meaningful, reliable and trustworthy. A difficulty is that in the social sciences in particular, interpretations of the strength and quality of evidence are fraught with disagreement. Even where there is consensus, the best available evidence may not meet the highest standard, or event an agreed standard.
Evidence-based policy making shares many of the features of “ordinary” or traditional policy making but has a number of distinct characteristics. Four basic assumptions are that:
- evidence based policy is a meaningful concept
- evidence should be available to policy-makers
- evidence can be interpreted and used to inform policy development
- policies based on evidence are better than policies that are not based on evidence.
The review finds there are broadly three types of policy fields which make different uses of evidence and research:
- Stable policy fields (areas where knowledge is reasonably settled; theoretical foundations are strong; governments broadly know what works; there is a strong evidence base and incremental improvement).
- Policy fields in flux (where the knowledge base is contested and there is disagreement over the most basic theoretical approaches).
- Inherently novel policy fields (the newness means there is no pre existing evidence base, e.g. regulation of biotechnology; privacy on the net).
Only in the first of these fields is policy really based on evidence, rather than just informed by it.
The full report can be found at:http://www.ahuri.edu.au/publications/projects/p40185
Mc Donald C,Forward via the Past? Evidence-Based Practice as Strategyin Social Work in “The Drawing Board: An Australian Review of Public Affairs” Vol. 3, No. 3 March 2003 pp. 123-142, School of Economics and Political Science University of Sydney
The concept of evidence-based practice is enjoying resurgence across the applied social sciences. This paper examines the concept’s deployment in social work, an activity that exemplifies the optimistic institutionalisation of the applied social sciences within post war welfare states. Employing the notion of the professional project, McDonald charts the development of Australian social work in the 20th century, noting the ambiguities and tensions of working in a humanist profession acting on behalf of the state.
The author finds that evidence-based practice is one way some social workers have attempted to manage these tensions, a means congruent with the professional project. As the welfare state is destabilised, and in response to managerialist-inspired modes of reform, evidence-based practice has been revitalised. McDonald assesses the capacity and merit of evidence-based practice as a political strategy articulated by sections of a destabilised occupational group to promote the goals of social work in a context of institutional upheaval.
This report can be found at: http://www.australianreview.net/journal/v3/n3/mcdonald.pdf
Marston G and Watts R,Tampering With the Evidence: A Critical Appraisal of Evidence-Based Policy-Making, in “The Drawing Board: An Australian Review of Public Affairs” Vol. 3, No. 3 March 2003 pp. 143-16, School of Economics and Political Science University of Sydney
Recent enthusiasm for evidence-based policy-making in Australia has many sources. So-called ‘managerialist’ reforms to public administration have been significant, as has the diffusion of particular bio-medical models of research. However, the meaning and practice of ‘evidence-based policy’ are contested.
The authors offer an account of the design of arguments to identify and critically assess the value of evidence-based claims and their relationship to evidence-based policy. Their critique indicates the very wide range of what can — properly — count as evidence, based on a premise about the irreducible richness and complexity of social reality. They highlight the importance of being thoughtful about the assumptions that shape policy research questions and ‘warrant’ the conceptual connections that constitute knowledge claims. Finally, the authors illustrate their arguments with a policy research case study on juvenile crime.
This report can be found at: http://www.australianreview.net/journal/v3/n3/marston_watts.pdf
Other languages