View and download this case study in pdf form or scroll down to continue reading on the page.
Download pdfAbout Health Technology Wales
Health Technology Wales (HTW) is a national health technology assessment body that appraises health and social care non-medicine technologies to drive improvements in health, wellbeing and value for the people of Wales.
Established in 2017 in the context of an ambitious and evolving health and social care policy agenda for Wales, their remit is to critically appraise evidence on clinical and cost-effectiveness of the technologies and produce national guidance to support their adoption or otherwise.
This encourages the best use of the limited resources available and maximises the health gain new technologies offer for the people of Wales.
A vision to demonstrate impact and value from the outset
In heading up the new organisation, Director Susan Myles recognised the unique opportunity for HTW to be explicitly outcome-focused in their approach and to build in a framework to assess the impact of their research, guidance and associated activities from the outset.
It was also critical for HTW to start clearly demonstrating the impact and value of its work as soon as possible, to provide assurance and confidence to those drawing on their research and guidance, and to their funders the Welsh Government.
What we did and who with
Following a successful tender process, Matter of Focus was commissioned to help HTW develop their prospective evaluation strategy and build the capacity of core staff in evaluation.
We really wanted to go with specialists that had an evidence-based approach and the Matter of Focus methodology approach really stood out in that regard.
Susan Myles, Director of Health Technology Wales
We worked closely with HTW as they established their new team. To begin, we conducted a high-level review of the evidence of impact assessment approaches within the field of technology assessment specifically and how these would fit with evaluating HTWs remit and work programme.
To develop an in-house impact assessment framework for HTW we held a series of outcome and impact mapping workshops with their new staff team to:
- explore and agree key outcomes and impacts for each area of HTW’s work,
- identify and map how specific activities contribute to outcomes and impacts,
- explore how they could evidence their activities, outcomes and impact and suggest an approach to evaluation.
What is outcome mapping?
Outcome – or impact – mapping is our participatory approach to setting out a theory of change based on a framework we express using our headings:
We believe that this plain language approach helps to refine thinking about the initiative in question.
Through engaging online or in-person workshops we bring together staff and stakeholders, to separate out outcomes and impacts into different levels, which helps people understand the change mechanism that underpins their work – linking their activities to the outcomes and impacts that matter.
We think about outcomes and impacts at the level of reactions; knowledge; skills and capacities; changes in behaviour, policy or practice; and at the level of longer-term social change.
Read more about outcome mapping
We supported the team with an audit of what data and feedback HTW was already collecting, which was mainly communications data and the number of downloads on the website. They also had feedback from in-person events.
Working in OutNav to assign existing data and evidence to their impact maps, we considered which parts of their impact story it could inform and where the gaps were. We suggested some practical steps to start filling the gaps, specifically to:
- complete reflective impact logs at the end of each appraisal process to assess immediate and potential impact (this included identifying any information about potential cost savings or reduction of harm where it was quantified);
- add a web feedback form when people download HTW evidence and guidance to capture information about them and what they hope to do with the information;
- set up an annual stakeholder survey;
- apply the evaluation framework to all other ways of getting feedback, e.g. event questionnaires, case studies.
We worked closely with two members of HTW staff to develop their competence and skills in mixed method data collection and analysis to support their routine assessment of the impact of HTWs work.
How HTW felt about this work
The advice and support provided by Matter of Focus has been indispensable in kick-starting our impact evaluation work. They are highly responsive to our needs, for example in making changes to the reports which can be generated in OutNav.
Sophie Hughes, Senior Health Economist, Health Technology Wales
Matter of Focus was incredibly easy to work with, the approach was intuitive and really easy to follow. It did require focused effort and I think that’s important to say.
Susan Myles, Director of Health Technology Wales
What they learned and gained
An overall evaluation framework
HTW now has a coherent way to demonstrate impact by applying the Matter of Focus evaluation approach across different activities, including case studies, their annual reports, impact flash reports and a dedicated web page.
Initially HTW had four impact maps that directly mapped to the four key pillars of their work: Identification, Appraisal, Adoption and Engagement. In 2022 they rationalised this into one map with four pathways representing those strands of work to avoid duplication and streamline the evaluation process.
Read more about making use of pathways through outcome maps to understand change
Matter of Focus helped us to streamline our impact evaluation strategy, to ensure that implementation of the strategy was achievable.
Sophie Hughes, Senior Health Economist, Health Technology Wales
A software to support and embed evaluation
OutNav holds the Matter of Focus evaluation framework in one place. It’s a secure, cloud-based software in which the team can bring together and assess all their data, information and feedback to evidence their impact map.
Matter of Focus’s unique OutNav software was a really important addition for us in terms of giving us a structured tool to routinely collect and analyse our impact data and information.
Susan Myles, Director of Health Technology Wales
Improved data collection and analysis
Overall, the data improvements led to a rounded picture of the different ways that HTW contributes to learning, skills, knowledge, and actions that people take as a result of the advice, opportunities for discussion and guidance HTW gave.
The website feedback form has been very successful despite initial uncertainty about how many people would be willing to give feedback via this method.
There is an option to skip the form to ensure it doesn’t prevent anyone from getting the information they needed. However, many people are happy to provide information and give contact details so they can be followed up. This gives insight into who is downloading guidance, what their reasons were and what the impacts potentially could be.
The Matter of Focus Approach and OutNav has really helped us consider the difference our work has made. It gave us a way to collect and use continuous feedback, reflection and improvement and the opportunities that arise from that, particularly because we were a new organisation and that was really important to us.
Susan Myles, Director of Health Technology Wales
What they do differently
HTW was fortunate to embed this way of working early on in their development. In addition to the outcome-focused vision of their leadership, they had two members of staff that championed the approach, putting it into action and building it into the day-to-day work of the organisation.
HTW takes a proportionate approach to data collection and analysis
Matter of Focus helped us to understand the importance of pragmatism in our approaches. As a research organisation, we had a tendency to be overly ambitious in our data collection.
Sophie Hughes, Senior Health Economist, Health Technology Wales
Matter of Focus was really helpful in advising us on proportionate data collection and analysis tools. We are a relatively small organisation, so we needed something that was fit for purpose and that we could build into ‘business as usual’ and be proportionate in terms of our efforts in relation to other demands on our time.
It doesn’t take armies of people: it is achievable and doable. Just set your own goals and stick to what you want to achieve from it. Just focus on the areas that will bring most value to you.
Susan Myles, Director of Health Technology Wales
HTW has aligned their reporting mechanisms, making their work and its impact come to life
One of the key decisions HTW made was to structure their annual report and quarterly government report mechanisms around the Matter of Focus evaluation framework, communicating highlights using the headings: what did we do, who with, what were their reactions, what they learn and gain, what difference does this make.
This is well received by stakeholders, who are often co-producing the case studies. Reports are accessible and interesting, making HTW’s work and its impact come to life.
View HTW’s 2021 Annual Report.
What difference does this make?
HTW is doing a great job of capturing how they are making a difference and how they can continuously improve on their work. Having a wider suite of feedback enables them to evidence their contribution to the outcomes they were commissioned on and show their role in supporting health and social care in Wales.
In 2021, HTW’s work on demonstrating impact was recognised with the prestigious David Hailey Award for best impact story. Voted on by their international peer organisations, the award identifies the best example of the impact a health technology assessment has had and what lessons were learnt from the process. To receive this recognition was really motivating for the HTW team.
I think our success was heavily based on our decision and effort on taking onboard the Matter of Focus approach and methodologies. As a relatively small organisation, it was wonderful to be able to show to our peer organisations – many of which have hundreds of staff – that this was possible (because there’s a lot of talk about it but few people do it) and that it can be done in a proportionate way.
Susan Myles, Director of Health Technology Wales.
Of critical importance is that HTW has provided their sponsor the Welsh Government with assurance and confidence to continue with and significantly increase their funding, and in doing so, allow them to grow their contribution to improve the quality of care in Wales.
