Digital innovation in humanitarian settings

GLOBAL PERSPECTIVES ON DIGITAL HEALTH
Interview with Javier Elkin - Digital Director, ICRC

Global Perspectives on Digital Health, welcome. This is a podcast for policymakers, donors, investors, builders, founders. If you're creating impact and it touches the lives of underserved people, then this is a podcast for you.

Today we're talking to Javier Elkin. Javier is the outgoing digital director of the ICRC, the International Committee of the Red Cross. And I'm really excited to speak to Javier because the ICRC is this big behemoth of an organization with a clear global mission, a humanitarian mission. But one with unique challenges, working in conflict zones, working in some of the most difficult parts of the world, where people are having to deliver in very, very difficult environments care for people who are in extreme conditions.

And each of those places that they're working have their unique constraints in terms of infrastructure, the actual clinical problems that they're trying to solve, the logistical problems, and the environment itself, whether that's from a particular political or geographical or meteorological perspective.

How does an organization like this implement digital technologies in a way that adds value, in a way that's sustainable, that can meet the local demands but still be maintainable from a global perspective. So Javier, who has been at the heart of that innovation journey that the ICRC has, is going to be so invaluable for many of us who are working in this space, trying to wrestle with the trade-offs between how do we create value locally and how do we think about scaling globally, et cetera?

So I think there are lots of transferable lessons to learn, especially from some of the most difficult contexts around the world. I'm really, really looking forward to this conversation. So let's get into it.

Welcome, welcome, welcome to Global Perspectives on Digital Health. What a delight to have you on the show. Thank you. Thanks for having me.

We met at the Geneva Digital Health Day for the first time at the FAIL Festival. You were talking about one of the challenges that you overcome, an on the field implementation that you had that something had gone wrong and what you and your team and the organization learned and implemented after that. And that's where we kind of started this conversation of like, you know, we should really get you on the podcast. So I'm really glad it's come to fruition.

Tell us a little bit about you. Yeah, sure, with pleasure. I actually started out as a student of the human condition. I studied psychology and I was always very curious about people and how they work. I was raised all over. I was raised in seven different countries by UN parents. So they were both international human rights lawyers. And so I was both curious about people and about the fact that people were so different all over the world and were living under different conditions. And so was quite sensitive to these things.

And I developed on to be completely fascinated by the brain and studied neuroscience, two masters and a PhD in neuroscience. And that was also when I first started working with artificial intelligence algorithms and digital technology. I didn't even know that I was doing digital health during my PhD, but apparently I was. I was doing mass experiments for public health with apps and tablets and things like that, mostly on mental health.

And so it was the beginning of me starting to see that I had a huge chance at being a comedian. And so then I explored a bit. I went to work on some startup incubators. I went to also do some research and strategy funding units of the UK government and opened up a bit my mind as to the wider innovation ecosystem.

And then I moved to Geneva. In Geneva I joined a life sciences technology company called IQVIA. So working with lot of different pharma companies and it was interesting work. But I wanted to do something a bit more valued. And so I joined Unitaid, which is a mobile health funder, part of WHO hosted by WHO here in Geneva.

And there I was in the strategy team and it was really interesting work. We were creating new funding mechanisms. They had a very, very rigorous approach to assessing who to give out funding to.

Then when COVID hit, I joined WHO proper. I actually started in the digital health and innovation department under Alain Labrique, another one of the people who was a guest on your show. Yeah, and so there I was advising Ministries of Health on their national digital health strategies. I was working in capacity building and assisting with implementation and guidance into how to implement behavior change interventions, especially through digital communication channels.

And for the COVID response, because everyone at the time had like the regular WHO hat and the COVID-19 hat, so for the COVID response, I was the product owner for the Messenger chatbots on Facebook. And so this was really cool because we could work with private sector companies that were trying to get a lot of global assistance.

We created a chatbot there that was really cool. It had all sorts of programs from different divisions such as a smoking cessation companion, mental health and support for stress program. We had flashcards that gave information on a daily basis of the different COVID-19 stats that people were looking at.

And so it had two million users and 100,000 subscribers. And I got a bit of a flavor for working with AI at the time as well, because we introduced the first natural language processing chatbot basically. And it was really interesting because people were engaging with it much more than with the other ones that were kind of press one, two, three, four to respond to get an answer.

And so from there, a position of a digital hub opened up, digital health coordinator at ICRC and it was an opportunity to build something from scratch, to give order where there wasn't any before. It was the first time this was going to be opened. And I started there three years ago. I'm now not part of ICRC. So last week, I'm not anymore.

But it was a good opportunity to have a chat with you and reflect a bit on what's happened and provide some lessons perhaps, share some things that could be helpful for other organizations.

Javier, thanks so much for sharing that journey. And yeah, what an opportune time to speak to you, because I guess you've got a lot of this learning and reflecting on fresh in your mind before you kind of sail off into your next role. So I'm really grateful that you've taken the time to speak to us. And so let's get into this particularly with the ICRC and the context that you came into.

You had this remit to kind of build something out from scratch. What was the context that you were coming into? My assumption is that you came into it you're like, oh my God, this is huge. Like, where do I even start? So how did you approach that start?

Yeah, so I've worked quite a bit before even starting the role because it was a bit of a daunting task. But I came in and it was the peak of excitement. There was this idea that there was so much being done already in digital health within the ICRC that there needed to be some order, there needed to be some systematic approach, there needed to be a program.

That should have been built around all the structure and really create a portfolio kind of approach to this. So that's how I started with this mission to kind of set the strategy, set the vision and create coherence.

The ICRC is a very innovative organization in that sense. The people that are in delegations in the field and country offices are always thinking of new ways of solving problems and digital solutions are part of that. The issue was that without a systematic approach, every time there was a problem, there was a solution for it. And there's so many problems that we ended up with a lot of solutions on our hands.

And so when I started, the first part was really trying to understand what the landscape looked like, what was happening well, what wasn't, from a technical perspective. Where was the organization going more broadly as well was very important to try to understand in what context does the ICRC work and where are we going?

There was a new technology strategy being drafted, there were new directions being set by the organization because this wasn't just a problem within the health units, but it was a problem more widely and ICRC is a very complex and large organization. Health is a small part of it, right? It can be very visible when you put them all in hospital somewhere or when you have patients coming in and you communicate about them, but there's a lot more going on at ICRC.

And so the first thing I did was kind of take a lay of the land and also there was a human element. So for the first time, there were a lot of people working in digital health in the organization, some of them in the technology department, some of them in the health units. And all of a sudden they had, you know, one person that became the manager of a team that didn't really exist and that were mostly naturally, almost in competition for funding and things like that.

So there was a whole leadership with a challenging aspect that came up from the beginning as well that was very important, very challenging, but I had a very good director at the time who was very supportive and understood what I was going into and really supported me through those initial steps.

That if you did feel daunted, I share it because I mean you're talking about getting the lay of the land and I feel like that must have been a really hard part of it. When you're talking about lay of the land of the ICRC, it's a lay of the lands, i.e. multiple countries, and then multiple layers. Don't know, supply chain and logistics, the infrastructure stuff, then the acute things that are needed in a conflict zone versus things that are needed in less acute zones but have been chronically facing issues and been ignored.

And so you probably had a lot of people in your ear saying, hey, like this is the most important thing you need to do actually. No, don't listen to anyone else. So how did you make sense of all of that?

Yes, so exactly, right? Exactly what was happening. The team was basically doing whatever was most convincingly communicated to them as the biggest priority, right? And so for most of first year, there wasn't a lot I could really direct because things were already ongoing, decisions had already been made.

And so it was really about trying to see what was working, what was not. Some things I was able to make decisions on and say, cannot keep going on this. And starting to provide a direction also allowing the rest of organization and ensuring we're developing all the scores, things that are scalable, that we're at the highest costs, we're at the hidden costs.

What is it that is based on partnership, what is vendor-based, and trying to understand all of this better, as well as unfortunately, things like health outcomes and things like that, were on top of the priority, because the reality is that two months into my role, we had our first financial crisis. And so a 700 million franc deficit was on the budget.

And so the priorities needed to be changed, right? But that didn't change the way. So, you know, they say never let a good crisis go to waste, right? So then you say, right, well, this is an opportunity for change. There's a requirement to change by the organization.

So how can we present this new kind of digital model strategy, multiple framework that we're trying to build? How can we present this in a way that encompasses and answers all the questions? And so that was basically the biggest challenge that I did most of the first year.

And just on this kind of business case side of thing and being financially constrained, there probably is so much to do and there's many, many things that the organization would have had to invest in, but it's a humanitarian context. So it's not like a, you're not making a financial argument for something that you get an ROI back for, but it's investing in for something to be delivered in a smoother way or kind of a more consistent way or reduce variability or something like that, or just saying, hey, like this is an investment that we need to make in this otherwise neglected thing, which is, guess, seen in a P&L sheet as a well, it's like a black hole.

So how did you kind of, how did you make business case for those types of things?

Sure. Yeah. Yeah. So as I was saying, it was something that we had to kind of adapt to on the go, depending on the needs of the organization. But I brought in some things with me from my previous roles that I had found useful.

So after building the logical framework, this was based on a theory of change approach. So that was the first. The first thing was to say, before we go into the specific solutions of which ones we should do or not, let's think about what our identity is, what our mission is, what are we supposed to actually be here for the organization, right?

And so this was built with the whole stakeholder engagement approach, and all that, but it was based on a theory of change approach, you have your outputs, your outcomes, and anymore. This is something I put at Unitarian for any sort of program, any sort of even from a digital health perspective, it's a useful tool both for internal clarity and also for external communication.

So this helped and provided the basis. But then when we were confronted with the fact that we have a lot of different ideas being proposed to us on a constant, and we didn't know which was prioritized, and we didn't even know where we were with the current ones.

So I proposed creating a structure. Now, with the team, we discussed this, and we created a structure that was basically a quadrant approach that whatever you propose to us will land somewhere in the quadrant that will define what we do with it.

And so there were really four criteria that we were looking at in order to ensure that we can either invest or not or consider or put on the back burner or whatnot, right? So they were strategic alignment, impact, technical alignment, and effort.

So what this actually means is from a strategic perspective, is this aligned with our logical framework? Is it aligned with the wider institutional strategy? Is it aligned with what our head, our director wants, what the leadership wants here in the country as well?

Is it something that will have sustainability, impact longer term, that sort of thing? Are these going to be local partners? Can we count on the next strategy, handle it? Are the commitments at least foreseeable? We don't have to have them all figured out in detail, but are they foreseeable?

And that would give us this have technical alignment. So from a technology perspective here, are we looking at something that is open source or does it require paid licenses? What about from a data security perspective? Is this going to be super risky? What does the cyber team think?

And does it require new expertise or do we have someone in house that can do this in like two minutes because they have the skill set already?

And then from an impact perspective, we have to look at what are the health outcomes. Is the return on investment from a financial perspective? And so here ROI doesn't necessarily mean do we make revenue because we don't make revenue, but it is can we make a financial case for the partner to then adopt it because they have savings compared to business as usual.

What for the ICRC because it will save them funds, it will save them money. How many people are going to reach? Is it going to be relevant beyond just this one country? Is it going be relevant across the whole organization?

So we're starting to introduce some global considerations as well, not just thinking about my context. Are other partners doing this? That was another one. And then how quickly we could get impact.

On the effort side, this was quite simple. It was how much money do we need to do this? And how many people do we need to do this? And can we integrate it within the daily work or is it something that comes on top?

And so with this in mind, basically we had a very strong framework for all of these. We documented this, we wrote it down, and it empowered the team really to be able to say yes or no quite confidently. Which was nice to see rather than yes, yes, yes.

And when we have to say no, Javier, can you please help me say no to this one? They could come to me with a case. And they were all making cases on the same level with the same criteria. There was no like, well, he asked me first type of thing. Or well, that one's more urgent because I think so. No, no, it fits on the quadrant somewhere.

And for leadership as well, they were quite happy because it provides clarity, it provides continuity and resilience of decisions. So if you think about it, at ICRC, people are rotating all the time. And so it's hard to keep track of decisions sometimes.

But with this, we were able to say, right, this is the decision that was made. If you disagree with it, we can revisit it. But this was the basis on why we're working on this one.

And so this was also very helpful because the rest of the organization then started looking at like, the digital health solutions optimizations. Digital health. So just digital solution optimization. And they asked me to be a part of that work stream for the whole organization after seeing this kind of work and how it led to a kind of stepwise clear rationale for decision making.

Theory of Change has come up in the podcast in a couple of episodes before, so the last one with Patti, also in kind of one of the ones before around measuring and clinical evaluation. And how you work backwards and therefore understand the lever points, what I take away from what you said, also such a useful tool in prioritization.

If you combine it with these other elements that you've talked about, which then gives you, thinking about my own challenges I've had in industry or kind of working with teams around prioritization being able to, yes, think about, okay, what outcomes are we going for?

Coupling it with all of these very clear objective criteria that you set that are important from your organization's perspective, right? And so it's then not this kind of generic framework, but here are the criteria that matter to us, and this is how we are applying them to prioritize things, right?

So I think that's a great takeaway. I can imagine at ICRC, the needs are so great so that we would have people come to us and say, oh, your support team needs to answer questions about staff health, because it's health and it's about a website. And so it's digital health.

And so these were the types of things where we could now point. And people would come to me, the engineers would come and say, do we have to actually work on this? And I'd say, no, no. We can all look back our key documents, we can look at this, does it fit, does it not?

And you know and it also I think it's good because it removes me from the equation as well as a decision maker and I was more there as a facilitator, right? For the team.

Yeah, exactly. If decisions, all decisions have to go through you, you're kind of creating a bottleneck, right? So yeah, yeah. That's really interesting.

And I think the other thing I take away is it's not only good for, you know, sometimes we think about health outcomes and we think about this work that we need to do from an external perspective, but actually such an important, organizational outcome from this work. Is internal clarity and driving good decision making.

So I really like that. So yeah, thanks for sharing with us some of the kind of tools that you've used to help you and your team kind of set that up.

You must have lots of examples of implementations. Do you have any examples of implementations that you could share with us that you've learned from or kind of created an impact that you're proud of?

Yeah, sure, sure. So you have to imagine, right, that the digital health portfolio at ICRC, once it was kind of made into a program, we had different ones, right? We had a lot of them. We had clinical decision making. We had solutions that were for pharmacy stock management. We had some that were for a digitalization of physical rehabilitation centers.

We had digital forms for ambulances, so for the carers and the ambulances to be able to put in all the electronic medical record of each patient and then hand over to a hospital. We had a telehealth solution as well, both for patients and for tele-expertise consultations. And then we had some medical databases as well.

So I think perhaps what's, I have two examples I think that are probably most compelling. I'll start with the one that I like to talk about because it's why I was able to bring in a bit of the development sector flavor to the work in the humanitarian sector. And it's with the Almanac.

So the Almanac is the algorithm for management of childhood illness. The Almanac itself was a solution that was developed in partnership with Swiss TPH, which is a part of the University of Basel here in Switzerland.

Is it tropical medicine in public health? That what it stands for? Yeah, tropical is Swiss tropical in public health. Yeah, okay. Awesome.

And so this was a solution that was basically a tablet with the IMCI, the Integrated Management of Childhood Illness, WHO guide on how to treat children integrated within it so that if you give this to a community health worker or someone that doesn't have full medical training, they can still provide the best possible care, or to provide wherever they need.

So why is this an interesting one? It was deployed in different countries, but we're going to specifically look at Nigeria. So within Nigeria, ICRC had been active since the 1980s because there had been a civil war already there. And so we had strong engagement with the Ministry of Health since 1980s. Imagine how far back up we're already at.

And the issue is that if we go to the northeastern states, here we were talking about Adamant states. In the northeastern states we have frequent incursions from armed groups, which we hear about in meetings all the time, even today actually. There's no access to the facilities. The communities in the areas are really affected by conflict.

They're in hard-to-use locations because we have to take our own flights to get there. It's not an easy place to access. Under-five mortality rate is quite high.

And what happens, which is quite common in conflict settings, that when you have a conflict, people with the skill set will emigrate. They will go somewhere else. They will find a job elsewhere in more ease. And the people who stay behind are always able to, right?

Or then sometimes they have family to care for and they can't, but otherwise they may not have the possibility to leave. And so it's, you have basically lack of high standard quality of care in remote areas.

So with Swiss TPH, ICRC developed the Almanac and was able to cover since 2016, 413 facilities in Adamant State. So primary healthcare centers with these tall buildings. And they had achieved 450,000 visits of children under five, so quite significant.

But in 2023, the conflict was deescalating in that area and moving elsewhere. And so the humanitarian actor that ICRC is, they said, great, we should exit from those facilities to support others because we don't have unlimited funds.

And so instead of exiting, I proposed doing a strategic handover. So the local district health had already shown a lot of desire for leadership. They were very involved. Proud of this. They were talking on the news about the Almanac program. They were talking to the Federal Ministry of Health as well.

So instead, we designed a two-year phased exit. Phase one, hand over both assets and the program to the Ministry of Health. And phase two, scale it up across the country.

And so for the first year, because we had an open source solution, we could transfer everything to local servers. We helped them obtain credits even through the server credits for them with they have a global health support program. We build capacity with our partners and the agency staff to really run the service themselves to train other states as well.

And it was really well received and adopted by the Nigerian Ministry of Health. So this was really great. And we had already started planting the seeds as well with the federal ministry. And so during phase two, the agency itself started leading the scale up to five other states who had shown interest in the neighboring region.

So the idea was to go from 400 to over 1,000 primary health care and covering over 25 million children and really open the door to digitalization. So we did a visit at the working room the federal ministry pediatricians who are the authority on pediatrics care in the country.

They visited a number of facilities. They came back with recommendations. We helped them draft the recommendations for to change policy. And Almanac was really approved at the federal level. It was called the National Council for Health. They were able to integrate the Almanac as a tool of choice to really provide high quality treatment in the country as a whole as well.

And we also helped create a roadmap to ensure that after we leave, they would keep integrating and follow this momentum.

If that hadn't happened it would have followed I guess like the routine or Standard flavor because like well you exit and then okay. Well, good luck now. All right. It's been good. See ya. Is, yeah. And it's, it's not desirable.

And to give credit to the ICRC, it's something that it's not, it's not standard either. People don't want to do that in general. They, they acknowledge that it's not, but then there's these internal tensions sometimes you have people that say, we're a humanitarian actor. We're not a development actor. We're there for emergency context.

And when we, when emergency dies down, we leave. And then there's others that say, well, no, we have to talk about this because we have a responsibility. We've gone in, we've disrupted the health system. Yeah, maybe we've strengthened it, but perhaps we've taken away from local capacity in some way. We've created dependence.

If we've been there for decades, you know, how do we, we can't just leave from one day to the next because the conflict has escalated and we just don't have the funds to keep supporting it. Right. So it's possible in these situations when you've been engaged for so long, I think, to open the door towards a more strategic mindset.

It's not necessarily the priority, especially when funds are shorter. Even after doing these handovers, it may be that then the local governments don't necessarily keep it up in the same way. So, yeah.

Yeah, that makes a lot of sense. It is a great use case for a technology which can help the handover process. And if we implement it in the right way, here's the impact that it can have. And it was so good, they then scaled up nationally. So not a really, really nice example. Do you have any others?

Yeah. So I think another example that might be interesting to share is about this one's a bit different because it illustrates a bit more how by adopting kind of the best practices from digital health that I kind of brought in with WHO. And all these things, why it had a positive impact in emergency contexts, right?

So the reason why this one's interesting is because in one of our operations, what happens often is that we plan for the year at ICRC. ICRC plans for the year. That's something called the PFR. For one year you plan and then you implement. Obviously there's always unexpected things. There's emergencies, there's escalations. And this is what happens in this case.

There was a specific context where the escalation suddenly increased, which means that you had more trauma patients. And so what a trauma patient is, you can imagine if you're on your day-to-day, your patients can come in for all sorts of reasons into your hospital, into the emergency room, right?

But when you have a war, you have buildings collapsing, you have cars exploding, you have people running out in the middle of the night. You have all sorts of things that lead to trauma, physical trauma in this case. And so we had to start supporting more trauma care centers in these places for a limited amount of time.

So we knew it was a limited amount of time because it's an escalation of the conflict. And we had to do it in different parts of the country, right? And at the same time, we're pulling out of another hospital in the region.

It was kind of very difficult for the field teams to monitor exactly what our contributions were and what was the flow of patients. And so they requested us a very simple digital solution to monitor what the activities are of our staff in that context.

And so what was currently happening was that our hospital staff would go with pen and paper and they would capture on pen and paper everything that was be registered. Either on pen and paper or on computer systems within the different centers that we were supporting.

How many centers were there in this context? It could be three or four. But they could be very far away. And it would take a long time to get that data. So you might have only someone doing it on a weekly basis.

So any event you go to the center, and there may be multiple systems, how do you collect data? Okay, so you've got to talk about a phone and paper, paper and an Excel on a computer.

And so they have to go around and collect all this data and then put it into a tally chart or something. And then they have to send this data to someone more in a central function that will then clean it, put it into an Excel. And then they would share this data back to the delegation, to decision makers, to have a full overview, and even regional level and headquarter level to have a view of this, right?

So you can imagine that this kind of created a lot of fragmentations. And so what we developed for them was very simple. We created a trauma center form that was configured in DHIS2.

So this is open source solution. We're already using it for our medical databases and other things. We custom apps built on it. So we have already been in for a lot of it. And we created dashboards linked to it.

And so within offline app, the DHIS2 they could start just collecting the data directly on the phone. And we tested, validated, and set this up in about two weeks.

So from when the field requested it to when we actually deployed it and got through all the internal approvals, it was just two weeks.

And then basically all that happened was that in the hospitals, in these trauma care centers, the hospital staff would just capture the data directly on the phone or the web app or whatever digital device they had at hand, simultaneous and parallel.

And then this would automatically sync to the reporting systems, dashboards that would then be used directly for the discussions. So we completely cut all the steps that were involved before, right?

So the impact was immediate, right? The data was captured rather than being in multiple systems. It was now captured in one system. We could coordinate teams better. And ultimately, all of this meant that there was higher quality support for patients, which is really the end goal here, because people are under pressure and don't have to spend time dealing with these systems, basically.

And so for me, what I like about this is that it shows, first of all, investing in open source was the right call. And it led not only to savings in the long term because we don't have license costs and all of that, but also because it allows us to react quickly in an emergency and that sometimes the solutions are, the simplest solutions are the ones that have the most impact.

It was simple. We were able to do it in a very fast way. And it fit the reality of the field.

So yeah, there's a lot of lessons, I think also for from a humanitarian perspective, kind of to go back the other way and to the WHO context. We were using global health designed tools for global health, but in a humanitarian conflict context.

And a lot of the time if you look at the work of the WHO, of the human organizations, are working in this space, they design and they think about peaceful times. But the world is evolving. There are more conflicts today than there were when both of us were born, and even before that, the past 40 years and more, right?

So we have to start thinking as well about how can we learn to integrate lessons from designing for conflicts as well into the development sector.

It really touches on the episode I've done with someone called Nadia two episodes ago, where we were talking about refugees and the need for digital identities. And were talking about, you know, it touched on a similar theme that, you know, the chances and the likelihood of us being refugees are also becoming higher.

How would we want to be treated? And kind of really thinking about the resilience of health systems that we're talking about in this context, data infrastructure, but absolutely it applies to what you're saying.

That's exactly right. That's exactly right. And I myself, I myself am a descendant of refugees. It's not, my wife is one generation. So her father was a refugee and I am two generations away, right?

So this is also was a big motivating reason as to why I wanted to join the ICRC. But it is something that we have to keep in mind that it can happen to any of us and that we should really try to make sure that the international global systems that we have in place can function beyond peaceful times.

Absolutely. And a lot of people in the global health space by the way will know DHIS2 but for those of you who don't it's I'll try and give a good summary and maybe you can improve on it but essentially it's an open source tool that's used for actually lots of different types of reporting so that public health and other kind of health ministry level or other kind of development use, there are lots of different types of use cases.

And so this platform is kind of readily available in lots of settings around the world. And it sounded like you thought, hey, you know, there's this infrastructure that already exists, there's a tool that already exists, how do we leverage that that's already there rather than trying to reinvent the wheel, essentially?

Yeah, exactly. So DHIS2 is an open source solution built by the University of Boston. Digital health information systems.

And it's true that in the global health space, it's something that is really, really utilized for handling basically your Ministry of Health data and having an overview from an epidemiological perspective of what's happening in your health system and that sort of thing.

But for us, we were using it to create custom apps. So for example, we created a pharmacy stock management tool based on it because it's just very powerful as a platform, right?

Then we donated that back to DHIS2, which is a wider community of practitioners who can download them and integrate them within their own. The beauty of open source, right?

Kind of, you you take things become modular and then it's like, hey, we built this thing in this use case that were useful to us. It might be useful for others, you know, go and go forth and prosper kind of thing.

So, Javier, that all sounds great. And those are really great examples of how you used ingenuity and good thinking to consider what was needed on the ground and adapt things that existed and that you had to say, how do we A, scale this thing nationally or be using existing technology in a way that really matters in this context.

But of course you've been at ICRC in a time and an age where there's been this external pressure of hey, LLMs are gonna like change the world and everything. And so, I don't know, do you have any examples of how you thought about in a good way, not how do we shoehorn LLMs into everything, but how do you do you have any examples of how you thought, this might be a great use case for using this within the ICRC context?

Yeah, you're completely right, by the way. As soon as I started very little time after, basically everyone started using, you know, LLMs and everyone was very excited about them. And there was this almost pressure to start saying, right, can we just use these for clinical decision making?

And no one knew any better at the time. And there was a lot going on there trying to make sure that we would do things responsibly.

So luckily ICRC had a standing partnership with EPFL, which is an engineering university about an hour away from Geneva. And we had a brilliant partner there called Annie Hartley.

And she had been developing a large language model called Medotron and wanted to make this relevant for the humanitarian health.

Medotron. Yeah, that's very creative with their names. And so it sounds like something coming out of Transformers or something.

But what her and her army of students had done was take the open source LLMs that have come out from the big kind of tech companies and retrain them on medical data. You can imagine textbooks, journals, and on PubMed and all of that.

And that's what made basically this open source LLM called Medotron.

And for us, we thought, well, this is a good opportunity to start thinking about how do we actually adopt this responsibly because we're starting to think about how AI is going to make potentially influence on ICRC's decisions to save someone's lives, right?

And it was helpful for us to see where we can work with the EPFL, they can read on the evaluation. They're going to take a science driven approach. They're going to be very cautious.

And, you know, it's not going to be tied to like our operational needs. And we can join this kind of broader evaluation initiative.

And so we adopted this Medotron through what's called the MOOVE. So the MOOVE is an acronym for Massive Online Open Validation Evaluation.

And it's basically a front end for how to evaluate different LLMs. And we started using this to see how we could do it at ICRC.

And so this was important because we needed a different way of evaluating LLMs, right? So if you look at standard AI benchmarks, all the multiple choice and things like that, they just don't translate for our humanitarian context.

So correctness in our context is very situational. So if you've asked Chat GPT, how do I treat a gunshot wound when I'm in a very remote area? It might say, well, dial 911 to call a doctor? Something like that.

But what we needed to say is actually put that person between two people on the back of a motorcycle and drive to your nearest health center, you know? Or if you have a broken arm and you're again very far in remote area, to say grab two pieces of wood and stabilize the arm and then wrap it up and go.

But that doesn't be technically correct without the contextual relevance. Right, right?

So you can have a clinically correct answer and the clinically correct answer can be very unsafe or it can be completely unusable if you're in our context.

So it's only hypothetical questions. There's no treatment decisions. It's just a platform where you log in as an expert to the support team.

In our case, have nurses, therapists, doctors, surgeons that will come in and ask the question. And they will get two answers, usually from two different models.

And they will vote which model gave the best answers. They will vote this answer is better. And then they will say why it's better.

And they will talk more on different criteria for fairness, bias, understanding, length, all these things, clinical accuracy, right?

Sourcing the experts who were experts in those contexts in the field there who were then asking questions, but also helping get closer to the right answer for those core contexts.

Exactly, because once you have the answer, the right answer, you're curating three targets, then you have one data point. When we get to 300, you can use those 300 validated answers to retrain the model and move on to the next phase, and then it should give you a contextualized chatbox.

I've got a question before you move on, just on this. So one of the things that I'm involved in lot of medical quality and benchmarking and evaluation work. And one thing that comes out often with lots of clinicians judging is a problem of ground truth and inter-clinician variability of what is correct.

Did you observe that? How did you deal with that?

So this is a beauty of the system, right? Because it's computer review, they can keep editing their answers until they are satisfied.

Got you. And editing afterwards as well. But with a minimum of three, you kind of correct for that as much as possible.

The other thing that we did was also the platform allows us to also introduce a RAG system, right? So Retrieval Augmented Generation Layer that we create our own knowledge base with our own institutional guidelines, and then can really start kind of reducing the kind of degrees of freedom for the algorithm to produce answers and to provide information.

So together, those things allowed us to create something that was helpful. And then we eventually even opened this up to the whole humanitarian sector, right?

We invited MSF and other partners to come and just use the platform because the reality is that to get to 300 here in the United States, takes quite a while.

I mean, that's quite an awesome data set. Yeah, definitely.

Because, you know, often people talk about the volume of the data set, but, you know, the quality and the depth of it for a certain context is also super valuable.

It up into a humanitarian validatathon. Again, very fancy words at EPFL, where it's basically an ongoing thing.

So anyone even now can even go and join themoove.org, click on humanitarian, and they can keep enrolling their experts into this.

And then eventually EPFL will give you your own organizational level chatbot that is specific to your institution if you're interested.

So it's a really cool and powerful model.

So that would have signified the end of phase one, right? Let's say, well, the algorithm works and we find that it's positive decisions. All of that would enter it.

Yeah, phase one. Then you have phase two, which becomes a silent move, right? So here we move to real questions from the field.

In this case, we've installed now GPUs at ICRC to host Medotron, to host the move internally. So really we can start asking confidential questions.

But in this case, the AI does not influence care. So the evaluations are still happening, but they're happening in parallel.

So the AI is kind of looking at the patient at the same time as the human, and it allows us to test, well, this is what the AI would have done. This is what the doctor would have done.

And it allows us to see we can evaluate if the treatment provided would have been the right one without any risk to the patients.

And then the last phase, the true move would be where you have, if phase two works out and it's safe and everything, then the true move is where you have the AI being used alongside standard care but in a controlled way.

And then you have the comparison of the outcomes level for the patients, care with AI, care without AI.

To do both phase two and phase three, we would need dedicated doctors to do this. And by partnering with EPFL, right, and in this case, Annie also has some other affiliations with medical schools in the US, universities like Harvard, and University of Boston, but they would be able to provide doctors that would go to the field and supervise and provide their care in the humanitarian settings as part of this kind of clinical trial.

So even if, for whatever reason, the AI evaluation didn't work out, we have been able to have a humanitarian impact by sending doctors to provide care and to reach more people with quality care.

And do you have sites that we're using phase two and three as well or like had phase two and three active?

I'm really hopeful that despite me leaving the team will continue doing so and there are strong signals all over that it should continue.

Absolutely, because if you think about it Shubs it's not about, there is no alarm system decision making type of thing in three years time, you're still gonna have to do this.

You're still gonna have to make sure that whatever chatbot out there is relevant to our context. For ICRC level settings, right?

And can really provide the quality of care that we are validated, that we trust. So there's no way around this really either way.

That in mind. 100%.

And I think, you the example that you gave about the trauma as well, you know, kind of this big advocate of like, you know, there's so much more to how good something is and the value that it gives.

And when we talk about quality, there's so much more to it than it just being correct, right? Your example was such a great encapsulation of that.

Something that can be technically correct on paper, but not valuable or useful in that, in a particular context. And that's the work.

I think the takeaway is like, if you're working in this space, you're not done just because you have, you know, a really well performing model, right?

You're not done. And there's so much work to do to make sure that it actually translates to people using it, people finding it useful, then it leading to being valuable in that context so that things in the world actually improve.

Exactly, because we have to remember that all of these systems are being designed as decision support tools, right?

The decisions themselves are with the humans, the clinicians themselves are fully accountable. These systems will not replace them. It supports them.

So it requires a human in the loop by design at all times.

But you need to go through these steps if you want to move from experimentation to actually build a responsible, scalable AI in your sector.

If you want to compromise safety or trust or ethics, you have to do it structured approach. It has to take a long time. It has to be cautious. It has to be stepwise. It's the responsible way of doing it.

In this way, are also in this book, you're also co-designing it with the people who are the end users. You're telling us, oh, I really need like a black button there so that if at any point the answers aren't correct, I want a quick and easy way to do it.

I want to go look for it. So just at all times, black button has to appear. You know, things like that.

We found also that within the organization, this has been very successful beyond health. EPFL is also now building a Legitron. So Legitron. Wow, nice.

Exactly, right? So now a lot of them are developing. It's the same thing. It's a move, and it's an LLM, it has the same name, right?

But it's just different. It's the same platform, it's just for different purpose.

Because the reality is that this is why we thought this was very powerful kind of approach is that we can use the same infrastructure, the same approach, standardize that.

But then everyone in the organization who has a kind of high risk use case, can do so in this way.

And luckily with Annie's team, they've been extremely supportive and helpful in achieving this.

What awesome learnings and insights like you've gone from the you've learned in the development context to as the ICRC were exiting Nigeria, identifying that's great use of technology to be able to help that health system sustain themselves with the Almanac product.

Then we went into the example of the open source DHIS2 tool, not over complicating things to getting the outcome you needed, which was like quick decision making, great, easy reporting for people who are in a very difficult situation.

And then you had a considered phased approach on assessing the quality and evaluating LLMs that were relevant for humanitarian contexts.

I mean, these have been such rich examples. I wondered if you had some reflections in general about the humanitarian sector.

Given everything we've seen with kind of USAID cuts has been such a tough year. You already talked about the financial crisis that you had at ICRC at the beginning of your journey.

But of course there was 2025 and the kind of cascading effects of that. I guess like, I wonder if you have any meta reflections on the sector of like what needs to happen, what needs to, like where does the humanitarian sector kind of go in this challenging time?

What do you think needs to happen?

Yeah, that's deep question. So, I mean, I can only reflect really on my own kind of experience on my point of view on this one, right?

So absolutely the funding cuts are, everyone's talking about the funding cuts, right? But to me, if you look at the whole sector, we're extremely diverse.

The whole sector is very diverse and every context in the field even is different. And I worry sometimes that the problem is there's been a bit more of crisis of trust which is into our sector and coherence about how our sector works, which has led to the funding crisis, which then is what we really talk about because that's been what we're most affected by.

But to me, it's more the symptom than the cause.

I think that there's a lot of organizations in the humanitarian sector and they all claim to be abiding by the same values, the same principles of humanity, impartiality, neutrality, and so on. But they don't act or communicate in the same way.

This creates a credibility problem for the sector, right? Because it becomes harder to convincingly uphold these principles when they're being applied differently.

So I think it's in two parts. So externally, I think, you know, they look in and there's this crisis of trust and then internally, we are under pressure because the organizations have tendency to retreat into what's familiar when there's funding and then cut the digital aspects, cut innovation functions and hope that simplification will bring control.

But I don't think that complexity will disappear because we kind of put it to the side.

And so I think there's two things. The first thing basically is there's been a bit of an erosion of trust. And I think we have to really look into that a bit more.

And the second one is we have to ensure that these organizations, ICRC is 163 years old, right? It's such an organization.

It has to be able to face these challenges and not kind of react to funding cuts in a way that will potentially make it less future ready.

When you talked about trust, you mean from a public perspective? Do you mean with a relationship with multilateral organizations or even like government organizations which entities in relation to in terms of trust?

There has been a huge shift from when I joined to when I left, how people react and how the general public sort of perceives our work.

So there's this kind of, there's a bit of a dogma. These organizations because we've been around for such a long time and I mean that as a sector, not in any single organization.

And there's this deeply held belief sometimes that you see that is that because humanitarian work involves helping people in very concrete ways, you're providing assistance, you're helping someone in a horrible situation, then the work is inherently good.

And on an individual from a human level, it's completely true when you see it as one of those people who have dedicated themselves to their whole lives and sacrificed so much, being in the field, that we're helping someone in need, that that matters, that has an impact, that was a good thing.

But at an organizational level, at a sector level, I don't think that's enough anymore. We're seeing it, it's not enough anymore.

So I think we need to start really looking critically at what it is that we do, having more scrutiny. There are no scrutiny mechanisms for the humanitarian sector.

All the claims are allowed. And it's important to have those uncomfortable conversations, I think.

So I would say, yeah, because trust is something that is earned and is currently lost in many ways.

So before, when my parents were in the sector, everyone trusted the international sector and today, you know, exactly.

And so I would say, like, you know, if you're talking to like leaders in this field, you'd say invite external and independent scrutiny.

It's not a threat. It's a necessity. We need to rebuild and maintain public trust.

We can't look at the past and say we've been around for a long time. Therefore, the values themselves and talking about them are not enough.

The world is making decisions that are going not necessarily aligned with those values and we have to adapt to that.

And we need to also be more transparent about where we fall short. We have to be open to change and questioning some long-standing assumptions about, you know, where are we doing good and where are we there for political reasons and things like that.

So these are all conversations that are happening all times internally.

The humanitarian sector is very kind of philosophical, I think. You talk about it, they're very philosophical, very deep in that sense, but it's also sometimes very short-termistic as well, right?

One-year planning and so on, and reacting to the latest emergency.

I mean, and to some degree, you need a big proportion of it to be like that because that's the nature of conflicts and humanitarian disasters.

At some degree, there's always going to be some level of reaction.

Javier, what an insightful and valuable conversation. You don't have to be in the humanitarian sector to have kind of taken really a lot away from this around leadership, around creating things from scratch around what it takes to prioritise with transparency, what it takes to create value locally.

So Javier, thank you, thank you so much. It's been a real pleasure to talk to you and yeah, wish you luck in your new endeavours as well.

Thanks, Shubs, really appreciate it. Awesome, awesome.