Published by Policy Press in November 2022, this practical handbook is available now.
About the book
Watch our launch webinar
Related content – practice briefing, blogs and podcast
Authors Sarah Morton and Ailsa Cook on the Transforming Society podcast
Where to buy (and how to get a discount!)
About the book
Why is it hard to know if you are making a difference in public services? What can you do about it?
Public services throughout the world face the challenge of tackling complex issues where multiple factors influence change. This book sets out practical and theoretically robust, tried and tested approaches to understanding and tracking change that any organisation can use to ensure it makes a difference to the people it cares about.
With case studies from health, community, research, international development and social care, this book shows that with the right tools and techniques, public services can track their contribution to social change and become more efficient and effective.
Engaging, enlightening and practical – this is a must-read for public service workers wanting to know and demonstrate that they are making a difference.Sandra Nutley, University of St Andrews
Why is it so hard to know if you are making a difference?
This chapter explores why it is so hard to know if projects, programmes, partnerships and organisations are making a difference, and what is needed to go on the journey to be able to make evidenced claims that really demonstrate impact. It introduces why we have written this book, and what will be covered in the rest of the chapters, including an overview of our approach.
Why complexity thinking can help understand public services
Public services sit in complex webs of programmes, policies and practices and are not in the control of any one organisation. Change happens through a web of interlinked and overlapping approaches and services, which makes it difficult to understand. Engaging with this complexity is essential for knowing whether any initiative is making a difference but can be challenging. This chapter explores what needs to be taken into account to work within complex systems when seeking to understand and track outcomes and impact.
What data and evidence do you need to see what difference you are making?
This chapter explores what data, evidence and feedback practices are helpful or unhelpful for knowing whether initiatives are having their intended impact. It considers some of the cultures and practices that can get in the way of having the required feedback for understanding change.
It sets out the kinds of data and evidence that are best suited to understanding complex change. Drawing on experience of working in different settings this chapter explores the concept of ‘data cultures’ and how these data cultures can help or hinder initiatives get the data they need. Examples highlight ways of applying this thinking to any specific work, especially making the case for using mixed data that includes qualitative data. The concept of a data improvement journey is introduced, along with practical ways of thinking about the data and evidence needs in any specific setting.
Owning your initiative’s outcomes and impacts
Being clear about the difference an initiative wants to make and the outcomes or the impacts of the work is the first step in knowing whether it is making a difference. Many organisations need to report to or align with multiple outcome frameworks and indicators. This can result in organisations feeling caught in a tangle of outcomes. Being able to navigate around this complex outcomes and impacts landscape essential for anyone leading public services where they are expected to work towards, report on or be evaluated against outcomes or impacts. This is what we call ‘owning your outcomes or impacts’.
This chapter offers accessible ways to understand and operationalise outcomes and impacts and unpack some of the contradictions surrounding the use of outcomes and impacts within wider policy and funding systems.
Embrace the complex context
Whether supporting people to move on in their lives, implementing new policy initiatives, providing training or spreading good practice, the evidence shows that the most effective approaches are those that are tailored to the specific context. Rather than ignoring it, trying to work well with the challenges and opportunities of any initiative’s context can help improve delivery and really make a difference to people and communities. It’s also essential for shaping how the initiative will make a difference, and for tracking that difference. This chapter discusses why context matters and sets out some practical ways of working effectively with the complex context in which anyone seeking to make a difference finds themselves.
Clarify the change you want to see
A meaningful connection between activities and outcomes is often vague or unexplored – but this magic ‘how’ change happens is essential for driving programmes and for understanding their impact. Outcome or impact maps are the core of the approach in this book, as the building block for setting out, understanding, learning about and evidencing change. This is one type of ‘theory of change’ approach which is common for understanding change in complex systems. We believe that process-driven theories of change are most effective for public services, and in this chapter we set out our version of this – outcome or impact mapping – and illustrate how they can be used at different levels.
Get going on your data, evidence and feedback improvement journey
Picking up on the issues from chapter 3 about getting the data, evidence and feedback needed to understand complex change, this chapter presents some practical ways to improve and use data. It includes how to undertake a data audit, filling gaps in data, and practical data collection techniques. It provides some examples of how these have been applied in different public services settings. It aims to help organisations get all of the data and information they need to evaluate their progress towards outcomes and demonstrate their impact.
Tracking progress towards outcomes and impacts
The journey to improving outcomes or maximising impacts is never over. Effective organisations that own their outcomes are always seeking to improve, reflect and learn, to align resources and develop systems and processes to make the best contribution possible. Taking time to systematically analyse and assess progress is a linchpin of this approach. This chapter outlines what is required to analyse, assess and track progress towards outcomes or impacts ahead of reporting, and shares some simple and accessible ways of thinking that can help even people with no experience of analysis get going.
Telling the story of the difference your work makes
Building a strong contribution story that explains and can evidence the difference your work makes to people, policy, practice or communities is not an easy task. This chapter brings together learning from working with many organisations and clients, as well as the authors’ work in outcome and impact reporting, using the approach set out in this book built on contribution analysis. The chapter considers reporting in an outcome or impact-focused way, for different audiences and purposes, with practical tips of how to do this well. This is the final part of the process described in chapters 5-8. It considers reporting for learning and improvement as well as making impact claims.
Using this approach in different contexts and sectors
The approach to outcome and impact evaluation in this book works for the complex problems and issues that public services face. It has been used in settings such as health, social work and care, policy, international development, research, and children and adult services. There is more in common across these contrasting sectors and contexts than divides them in relation to outcome and impact evaluation. Whatever the setting, the key challenges remain: how to work well with complex contexts; how to understand and work well with outcomes and impacts; how to get the relevant data, evidence and feedback; and how to embed this in day-to-day work. This chapter explore those challenges and gives practical tips for overcoming them. It includes case studies of the approach in different organisations and settings.
Becoming an outcome and impact-focused organisation
This chapter shares insights from some of the organisations who have experience of embedding this approach in their work. It has a resources section setting out some useful information for working in this way. It includes a description of the software OutNav developed by the authors and used for embedding this approach.
This book guides and encourages those of us working in public services. It does not gloss over the complexities and the commitment required but reassures us that it is worth it. People deserve to see their experiences captured.Mairi Simpson, Health Service Manager
Virtual book launch
Catch up with our webinar celebrating the launch of the book.
Authors Sarah Morton and Ailsa Cook highlight key themes from the book and, together with one of our clients, Susan Myles of Health Technology Wales, share their experiences overcoming the challenges of tracking the outcomes and impact of complex change processes.
This book is easy to read. More importantly it’s clear, it’s succinct and it’s practical. And to any organisation that’s interested to understand and demonstrate the difference they make day to day; I would absolutely recommend it to you.”Dr Susan Myles, Director, Health Technology Wales
Why is it hard to know you are making a difference?
Transforming Society podcast
How Do You Know If You Are Making a Difference?
Authors Sarah Morton and Ailsa Cook speak to Transforming Society podcast host Jess Miles about the challenges of tackling complex issues and trying to achieve change in the public services.
Listen to the podcast.
Evidence & Policy blog
How do you know if you are making a difference? Is your data culture getting in the way?
Sarah Morton and Ailsa Cook look at how organisation is influenced by their data culture. Read the blog post.
LSE Impact blog
Putting public data to use is a fundamentally human challenge
Sarah Morton and Ailsa Cook discuss how the fundamental challenges faced by organisations using data are largely human. Read the blog post.
Third Force News opinion article
Why is it hard to know you are making a difference?
Sarah Morton highlights a community wellbeing service case study from the book. Read Sarah’s article.
Where to buy the book
How Do You Know If You Are Making a Difference? A Practical Handbook for Public Service Organisations retails for £16.99.
Order the book from Policy Press
Sign up to the Policy Press mailing list to receive a 25% discount on this and all their books.
We are delighted that people are enjoying the book and taking the time to let us know on social media.
If you like to share your thoughts online, we’d love to hear your feedback about the book: on Twitter @matter_of_focus and on LinkedIn /matter-of-focus/.
Making time to read through @sasmort & @AilsaCook ‘s great book – chimes strongly with #PublicEngagement best practice i.e. understanding situations from the perspective of the people you are working with being key: ‘their definition of success and failure is what is important’ pic.twitter.com/A6TeJGn5xB— Ben Littlefield (@ChemField) January 4, 2023
This came today. Said I’d only read one chapter but then the highlighter came out and all hope was lost. @sasmort and @AilsaCook just loving it so far, so much to reflect on and action pic.twitter.com/JV54yIak9S— Suzanne Steel (@SuzanneSteel7) December 20, 2022
Mini break reading- how to measure relationships at the heart of complex system change- it’s speaking to my principles from page 1! @ARC_EoE pic.twitter.com/zauPlV2WCH— Professor Wendy Wills (@Wendy_J_Wills) December 1, 2022
Really useful session from @matter_of_focus on evidencing how what you do is making a difference in the context of a complex system 💭 Having read the book already, it’s a helpful & accessible read for anyone grappling with evidence and impact! 📖 Thanks @sasmort @AilsaCook! pic.twitter.com/0DTz2aBXRq— Katie Cramphorn (@KatieCramphorn) January 12, 2023
To receive a regular round-up of our insights and news please sign up to our mailing list.Sign up now