🎯 Evaluation! A make or break thing in the digital health.

I sat down with Dr Shay Soremekun from the London School of Hygiene & Tropical Medicine to talk about the very hot topic of evidence generation and evaluation.

Everyone's talking about LLM evals, technical performance, benchmarking.

But ultimately people care about impact. Yet impact is rarely a neat, linear path to a yes/no answer. Anyone who's actually implemented something in the field knows about all the other contributing factors, the daily challenges and how hard it is to move the needle on big clinical outcomes with robust clinical evaluation.

To understand why it improved care, if it did, was as important, if not more important, than understanding that it did.

Because that will help us to understand how we need to potentially modify or adapt the intervention either in the same place or in future places to be able to achieve the same success.

🌟 Who will benefit from listening to this episode?
Digital health companies : Founders, data scientists, AI/ML scientists, PMs, designers, clinical evaluation/study teams, clinicians
Funders : donors, investors
Global and public health professionals
Implementers
Researchers
Regulators

Shay explains the use of Program Theory and Logic models to visually connect your intervention, all the intermediate steps (not just technology, but people and change) to outputs, outcomes and long term impact. Making space to observe other levers in the system you didn't initially anticipate.

The work that Shay and her team have done with The Malaria Consortium and LSTM's Centre for Evaluation looking at a digital health tool for community health workers in villages and facilities showed this perfectly. Because they were intentional about observing the whole system, they discovered other factors contributing to impact and could redirect efforts accordingly.

In this age of tightening budgets, and pressure to show clean shiny KPIs, how do you make room to observe these things?

We also discussed decolonizing evaluation: capturing what's locally valuable, not just paradigms of success developed in comfortable offices and ivory towers.

So many learnings here for understanding good co-design, implementation, and how to measure what matters.

💡 Keep learning!
If you found this episode helpful listen to:
Episode 15 : Implementation 101 and how to Fail well with Caroline Perrin
Episode 12 : Health First, Innovation second, with Smisha Aggarwal
Episode 5 : What is the right approach for regulation and evaluation of digital health technologies?

I have also written a Substack article summarising what I learned from this episode. Check it out for easy reference to the concepts outlined here

Found it useful? Know someone who is struggling with this very thing? Share and keep raising the profile of people impacting underserved community globally.