London School of Hygiene & Tropical Medicine

Lecture 4 (13.00 – 14.00, 31 October 2018,  Jerry Morris B, Tavistock Place): Uncertainty and its consequences in social policy evaluation and evidence-based decision making, Matthew Jukes ( Fellow and Senior Education Evaluation Specialist at RTI International) and & Anne Buffardi (ODI)

The methodologies of RCTs and systematic reviews imply a high standard for the level of rigour in evidence-based decision making. When these standards are not met, how should decision-makers act? When a clear body of evidence is not available there is a risk that action is delayed while further research is conducted or that action is taken without optimal use of the evidence that does exist. In fact, all evidence-based decisions involve a degree of uncertainty. The question we address in this paper is: What level of certainty is required for which kinds of decisions? Scientific skepticism demands a high degree of certainty for sure and steady advances in knowledge. Medical interventions with a risk of death require a high degree of certainty. But what about decisions in social policy? We argue that decisions should be made based on a consideration of both the uncertainty and consequences of all possible outcomes. Put simply, if severe negative consequences can be ruled out, we can tolerate greater uncertainty in positive outcomes. We present a framework for making decisions on partial evidence. The framework has implications for the generation of evidence too. Social policy evaluations should systematically consider potential negative outcomes. Sources of uncertainty – including assumptions, methods, generalizability of findings as well as statistical uncertainty – should be analyzed, quantified where possible and reported. Investment should be made in reducing uncertainty in outcomes with the biggest consequences. Uncertainty can be managed by placing small bets to achieve large goals. Overall, more systematic analysis of uncertainty and its consequences can improve approaches to decision-making and to the generation of evidence.

Matthew Jukes has two decades of academic and professional experience in evaluating education projects, particularly in early-grade literacy interventions and the promotion of learning through better health. His research addresses culturally relevant approaches to assessment of social and emotional competencies in Tanzania; improving pedagogy through an understanding of the cultural basis of teacher-child interactions; frameworks to improve evidence-based decision-making; and methods to set reading proficiency benchmarks. He is contributing to projects in Malawi and Tanzania aimed at improving the quality of pre-primary and primary education in those countries.

Lecture 5 (13.00 – 14.00, 28 November 2018,  Jerry Morris B, Tavistock Place): Contextualised structural modelling for policy impact, Orazio Attanasio

Lecture 6: Stakeholder Engagement for Development Impact Evaluation and Evidence Synthesis, Sandy Oliver

Lecture 7: The need for using theory to consider the transferability of interventions, Chris Bonell

Previous lectures in this series are available to watch online:

Lecture 1: The Four Waves of the Evidence Revolution: Progress and Challenges in Evidence-Based Policy and Practice, Howard White (research director of CEDIL and Chief Executive Officer of the Campbell Collaboration)

The evidence movement has rolled out in four waves since the 1990s: the results agenda, the rise of RCTs, systematic reviews, and developing an evidence architecture. This revolution is uneven across sectors and countries and is an unfinished revolution. Drawing on experiences from around the world, this talk will provide a historical overview of the evidence movement and the challenges it faces. Response to these challenges will be considered, including those offered by the work of CEDIL. Watch the lecture online.

Lecture 2: Representing Theories of Change Technical Challenges and Evaluation Consequences, Rick Davies (independent Monitoring and Evaluation consultant [MandE NEWS], based in Cambridge, UK )

This lecture summarised the main points of a CEDIL inception paper of the same name. That paper looks at the technical issues associated with the representation of Theories of Change and the implications of design choices for the evaluability of those theories. The focus is on the description of connections between events, rather the events themselves, because this is seen as a widespread design weakness. Using examples and evidence from a range of Internet sources six structural problems are described, along with their consequences for evaluation. The paper then outlines six different ways of addressing these problems, which could be used by programme designers and by evaluators. These solutions range from simple to follow advice on designing more adequate diagrams, to the use of specialist software for the manipulation of much more complex static and dynamic network models. The paper concludes with some caution, speculating on why the design problems are so endemic but also pointing a way forward. Three strands of work are identified that CEDIL and DFID could invest in to develop solutions identified in the paper. Watch the lecture online.

Lecture 3: Development Impact Attribution: Mental Models and Methods in ‘Mixed Marriage’ Evaluations, James Copestake (Professor of international development at the University of Bath)

Using the marriage metaphor to explore collaboration that spans academic traditions and disciplines, researchers and managers, public and private sector agencies. Mental models are used to explore the ontological, epistemological, contractual and socio-political tensions created by formalised evaluative practice. The lecture focus particularly on experience with mixing qualitative impact evaluation with other approaches to generating evidence, learning and legitimising public action. It draws on case studies from the garment industry, medical training, housing microfinance and agriculture spanning three continents. Watch the lecture online.