Embedding meaningful outcome evaluation: tips from the OutNav Community

This post captures the learning from a 2019 OutNav Community meeting that focused on embedding the practice of meaningful outcome evaluation across organisations.

Our clients have overcome many practical and conceptual challenges, like reconciling the outcomes of their work with those specified by funders and working out mechanisms for staff to systematically reflect on practice. In so doing, these organisations are cementing their place at the forefront of best practice internationally in meaningful outcome evaluation.

We are grateful to two of our clients, Julie Gardner from the Thistle Foundation (discussing her work with Lothian House of Care) and Franca MacLeod from the Scottish Governments’ Children and Young People’s Improvement Collaborative, for sharing their work with the group.

Embedding reflective practice

Julie Gardner opened the morning by sharing the work of Lothian House of Care team to embed reflective practice in their work.

Time and space

As a team, Lothian House of Care believe that giving practitioners the time and space to reflect on their practice is critical to enabling them to work in new and more person-centred ways. In addition, the team recognise the value of this as evidence to inform their evaluation of the programme.

Lothian House of Care has taken a flexible approach to making space for reflective practice. For example, creating protected time for reflection in training sessions, incorporating reflective practice in team meetings and even holding lunch clubs to enable staff to reflect.

Focus

In order to keep reflections consistent, the team use a set of questions to focus their discussions:

1. What are you noticing?
2. What has been better?
3. What is working?
4. How did you do that?
5. What are you learning?
6. What will you be pleased to notice in the coming weeks?

Capturing reflections

The staff reflections are captured in a variety of ways so that they can be used by the Lothian House of Care Team to contribute as evidence to their ongoing evaluation.

In some cases reflective notes are captured on a flip chart and then photographed, at other times staff are asked to write their reflections in a reflective impact log. Reflections are also captured by other team members as they write up meetings and sessions. These sources of evidence can then be uploaded to OutNav and used to inform the evaluation.

The difference reflective practice makes to the team

Julie made a strong case for the value of reflective practice as a tool in itself as well as to contribute to evaluation. She summarised the difference that reflective practice makes to the team:

• The data captured through reflective practice allows the team to tell the story of the difference they make in all its complexity and subtlety​, in the way they want to tell it rather than focusing only on the story the ‘system’ asks them to tell.
• Reflective practice can support learning and development​.
• When people are enabled to do so, reflection provides time and space to notice things they may not otherwise have paid attention to​.
• Reflective practice can result in improved morale and resilience among the team as they are able to see a piece of work as more about telling the story of the difference they are making, rather than a ‘tick-box exercise’.

The time invested in reflective practice at Lothian House of Care has resulted in lots of smiles.

Julie Gardner, Lothian House of Care

The benefits of using OutNav to embed evaluation in improvement

Franca MacLeod shared some of the work they have been doing using OutNav to evaluate a Scottish Government improvement programme, the Children and Young People’s Improvement Collaborative, CYPIC.

Franca highlighted three aspects of working with OutNav that she found particularly helpful:

A clear focus on activities and engagement

As with many improvement programmes, the work of the improvement advisors is complex and diverse. Going through the process of outcome mapping has enabled them to develop a very clear and shared understanding of what they do and who with, to underpin the evaluation.

this approach allows you to be closer to the activities and engagement.

Incorporating different kinds of data and feedback

When evaluating improvement it can be hard to gather as much data as you would ideally like in a robust and systematic way. Franca highlighted the benefits to them of using OutNav to incorporate the observations and reflections of improvement advisors. Most importantly from a research perspective, being able to assess confidence in the evidence allows CYPIC to include less robustly collected data and highlight that there are limitations.

Capturing data and feedback directly from improvement advisors

CYPIC are making the most of the flexibility afforded by OutNav and have set every improvement advisor up on OutNav with their own project. This has filled an important gap for them, enabling each improvement advisor to gather and analyse the management information that they need to ensure their projects are progressing well. Holding this in OutNav gives Franca a single space where she can easily access all the data and use it to inform the evaluation.

If you would like to know more about how evidence is rated within OutNav, read our post Track your progress and impact in OutNav.