Man and woman working on white board with sticky notes

Monitoring and evaluation can help us act on the lessons from a year like no other

Our Thinking | insight

Published

Authors

6 Minute Read

RELATED TOPICS

Share insight

insight

One strange reality of 2020 was that I sat at home while the world changed. Life felt discontinuous, even though each day was hard to differentiate.

What did I learn during this experience? In the midst of it, I was reminded of what I value. I also learnt about my own adaptability and that of my clients and colleagues as we found new ways to connect, collaborate and do good work.

In terms of deeper change, it is hard to say yet. It takes time for the meaning of intense experiences to filter through the subconscious and come to our attention. Learning also takes stimulus: active reflection with other people and time moving in the world.

Reading Ian Buruma’s “Year Zero: A History of 1945” during lockdown, I was conscious that global upheaval brings destruction – destruction that contains creative seeds, but destruction nonetheless. Dealing with that means understanding the historical roots of inequality and insecurity as we react to the present moment.

In “A Paradise Built in Hell: The Extraordinary Communities that Arise in Disaster”, Rebecca Solnit speaks to a similar opportunity to learn through difficult times. She argues that, if we are willing, we can learn what matters to us. Specifically, we can learn new, inclusive and democratic ways of working together, and do things differently as a result.

2021 therefore is a year for making sense of the continuities and disjunctures of 2020 and learning what they mean for this year – and the years to come.

For policy-makers, regulators and program managers, evaluation and its sidekick, monitoring, will be essential tools to drive this sense-making process. By collecting and analysing data to regularly track how a program or service is performing (monitoring), and to periodically assess the value of what has been delivered and identify what is required to deliver better outcomes (evaluation), we can develop the evidence base needed to steer for what matters.

Based on Nous’ extensive experience supporting government, not-for-profit and private businesses in times of change, we have identified three questions for organisations to ask as they design their monitoring and evaluation approaches to learn as much as possible in this new year.

Three questions: to where we want to go, why and for whom

What do we want to learn?

Learning is most powerful when there is a shared agenda for it.

2020 challenged what we thought we should, could or would do, and how. For example, the community conversation about racism and how we recognise Aboriginal and Torres Strait Islander peoples changed irrevocably. Policy questions long discussed in Australia were solved, if temporarily – people who were homeless were housed and unemployment benefits were increased. Pilot solutions were rolled out at scale, including telehealth and online court hearings.

The opportunity in 2021 is to build on this proven adaptiveness among policy-makers, service providers and the community to deliver public good in innovative and effective ways. The associated challenge, however, is to make hard choices about priorities, about the best allocation of resources and about what roles government, businesses and service partners should play.

With policy opportunities and challenges in mind, monitoring and evaluation should help us learn about:

  • Context: What are the evolving circumstances and how do they affect what we want to achieve?
  • Delivery: Where are we at, compared to where we thought we would be? How did we adapt? What should we focus on now? How will we know if we are doing the right things, and doing them well, and if we are the right people to do them?
  • Outcomes: What is changing for the people who have most at stake? In the face of the differentiated impacts and opportunities of COVID 19, we particularly need to find out what works for whom, how, and why or why not. For example, Nous is using these ‘realist’ evaluation questions as we evaluate suicide prevention and mental health programs to understand how services can best assist people with different needs and preferences.

How will we learn, and from whom?

Answering strategic questions meaningfully in an evolving context requires both robust and creative methods of data collection and analysis.

In 2021 we should continue agile learning while also building the foundation to learn systematically over time what works and why. For example, in evaluating an innovative education program, in 2020 Nous adapted our original plans to focus on capturing real-time data (using Nous’ COVID-19 Recovery Monitor, public datasets and virtual workshops). We used this data to confirm the program’s importance and identify the strategies used to adapt it to changing conditions. In 2021, we will start in-depth, longitudinal consultation, with a carefully chosen sample, to understand what is changing for school students and why.

Who we learn from is as important as how we learn, because the collection and use of knowledge is an exercise of power. This was highlighted by the Productivity Commission’s release in 2020 of the Indigenous Evaluation Strategy, a whole-of-government framework that requires evaluators to centre Aboriginal and Torres Strait Islander people, perspectives, priorities and knowledges in evaluation design, delivery and reporting. In a recent evaluation of a legal support service for Aboriginal and Torres Strait Islander young people, Nous used culturally safe engagement to hear from the young people involved what mattered to them, and why.

What will we do with what we learn?

Finally, the data we collect and analyse is only as useful as what we do with what we learn. Evidence-based reflection, problem-solving and decision-making will matter even more in 2021, as we are required to keep adapting.

At Nous, as the pandemic took hold, we knew we needed to keep hearing what was important for our clients and employees. We intensified our check-ins and adapted our modes of engagement. We used this regular supply of information to make adaptive business decisions, continue serving our clients and look after our people.

Organisations can also learn from each other. It can be risky to publish information when things do not work as expected. But, as a roundtable of NFP leaders hosted by Nous highlighted, the exchange of lessons about what works well, for whom, how and why contributes to a wider culture of learning and responsiveness in a sector or community. It encourages the use of evidence to make good decisions about what to do more of, less of or differently.

We can harvest creative seeds in the destruction

We need to learn from 2020 and keep learning in 2021 so we can harvest the creative seeds in the destruction and make decisions that lead to better outcomes for communities, customers and businesses in an uncertain world.

Organisations that use targeted, high-quality monitoring and evaluation strategies to understand what is working, or not, for whom and why, will be well-positioned to take up new opportunities, adapt their approaches and leave behind ways of working that are no longer relevant.

Find out more on NousCast Shorts

Get in touch to discuss how Nous can help you enhance your monitoring and evaluation efforts.

Prepared with input from Tanya Smith