27 – The Evidence Based Principle

[00:00:07] Santiago: Hi and welcome to the IDEMS Principle. I am Santiago Borio, an Impact Activation Fellow, and I’m here with David Stern, a Founding Director of IDEMS. Hi David. 

[00:00:16] David: Hi Santiago. Another principle. What have we got today?

[00:00:20] Santiago: Today is Evidence Based, which is probably second nature to you, to be evidence based.

[00:00:28] David: Well yes and no. I love the opposition of this with the Informed Decision Making. 

[00:00:33] Santiago: What do you mean by opposition? 

[00:00:35] David: Well, Informed Decision Making is deliberately quite weak in our decision making. But Evidence Based is extremely academic and strong. And so, those two together are actually, they’re nicely contradictory amongst themselves in some way, which I quite enjoy.

[00:00:49] Santiago: Okay, let’s have a look at what the principle means for us. So, ‘this principle highlights the importance the company attaches to rigorous evidence. It explains the presence of academic rigour and integrity in all aspects of the company’s work’. 

[00:01:06] David: Yes. 

[00:01:07] Santiago: Okay, so it highlights the importance the company attaches to rigorous evidence, but… 

[00:01:15] David: Well, one of the things which is so interesting about this one as a principle is that a lot of our work really does work with world experts trying to get rigorous evidence in different ways. But it doesn’t necessarily, there’s nowhere in this where it mentions decision making. And I think that’s a really important aspect, and we really do value, and I certainly value, academic rigour and integrity in our work. And I think that’s central to how we work, to what we place as importance.

[00:01:46] Santiago: Yeah, but you started this conversation saying that it was in contradiction with… 

[00:01:52] David: Maybe, I didn’t mean in contradiction. 

[00:01:54] Santiago: I don’t think there is a contradiction at all. 

[00:01:56] David: You’re absolutely right. 

[00:01:58] Santiago: You know, Informed Decision Making is about measuring uncertainty and that is a very important part of academic research as well.

[00:02:05] David: Well, no, that’s different. I mean, the academic rigour, you know, evidence, most decisions that I have to make now as a company director, I don’t have the evidence. I can’t point to research studies which actually will give me the right answer.

What I can do is be really informed about what there is evidence for and what there isn’t evidence for. This is where the uncertainty is so important. I am aware of academic studies around the values of different management structures. That academic rigour that has led to people actually studying and gaining that learning is something which I value.

However, it doesn’t help me with my decision making. How it applies in particular cases, that’s part of being informed, part of recognising that there’s complexity, that there’s different evidence for different things, trying to stay up to date with what is known and what is not known. That’s important in terms of being informed. 

The evidence base is not about my day to day decisions, it’s about how we work. I would love, for example, for people to study our principles, to actually get academic evidence related to when these principles are applied, what value do they bring, what are the compromises that are being made by choosing a given approach over another one? 

[00:03:33] Santiago: What they enable and what difficulties they create? 

[00:03:38] David: Absolutely. Evidence Based is slow. The way I like to see it is mathematicians are fantastic at this. What mathematicians are able to do is they’re able to take all usefulness out of particular sort of questions until you’ve got something you can actually answer. 

As a mathematician, I used to love abstraction. And what I was able to do through abstraction is to get to something I could answer. Well, I knew the difference between right and wrong, and I could answer with certainty. But almost certainly by taking it to that point, I have removed its usefulness. 

[00:04:13] Santiago: But in some cases you add a usefulness by doing so. You narrow it down, but it still remains useful. What comes to mind is the conversation we had yesterday where I had to rephrase a question three times because it was not carefully thought through at all the first time I asked it, the second time I removed a couple of words that were completely unnecessary, and the third time I actually was very specific in the question, and only then you were willing to answer it.

[00:04:46] David: Yes, well I think absolutely, I can be a bit obsessive on these things. But actually, there’s certain things I know and there’s certain things I don’t know, and actually recognising that and being able to say what I have evidence for and therefore I can answer, this is important to me. And I think Evidence Based, it’s a difficult one because it is central to how we work, it’s central to who we collaborate with. We do work with serious researchers who value evidence.

Well, which evidence? Whose evidence? It’s hard, rigorous research with integrity is expensive. You know, we can’t use this for all our decision making. And there was a very deliberate effort to separate these two things out. We recognised the importance of Evidence Based. There was a time when evidence based was what we felt would be the more central pivot principle, but recognising that it falls under the Informed Decision Making, but there’s more to making informed decisions than just evidence, because we can’t have the evidence for all the decisions that we need to make. You can’t do a trial for everything. 

[00:05:57] Santiago: Sometimes a randomised control trial could take a year or two. 

[00:06:02] David: That’s an underestimation. Sometimes it can take five or ten. But yes, yes, you can take a long time to get that evidence. And even if you got that evidence, will it help you in your decision making later? Maybe not. 

But you would know something with certainty. Now, how useful that knowledge will be is a whole different question. 

[00:06:22] Santiago: Well, are you suggesting that every trial will lead to a certain answer? 

[00:06:26] David: Yes, even if that certain answer is that we don’t know, we don’t have enough information. So a trial gives you evidence. And having a negative result to an important trial, it’s important information. You get publication bias if you don’t publish such negative results. 

My favourite example of this is that, you know, there was this study of studies on publication bias, which demonstrated the publication bias in studies on publication bias. 

[00:06:51] Santiago: You’re getting incredibly close to Russell’s Paradox in a wonderful way.

[00:06:56] David: But most importantly on this, the processes to do rigorous scientific research, to really have the integrity to understand what there is evidence for, what there is not yet evidence for, what the limits of that evidence are, that’s something which we value deeply and we accept that it comes at a cost. 

[00:07:17] Santiago: Well I was going to say, it must be one of the principles that creates more barriers than opportunities. 

[00:07:23] David: Well, yes and no. In terms of scaling IDEMS, yes, it creates a huge barrier. In terms of opportunities to work with academics and serve academics, this is, of course, one of our great strengths, and so it creates opportunities as well. 

Are they the opportunities that are going to lead to scaling our impact? No, because actually researchers are not going to be the answers in terms of scaling impact. We are making our core business harder and making essentially peripheral business easier. 

This is a challenge. I don’t want people to undervalue the work with researchers in academics. We value that work. But it is secondary to our social impact. Social impact is our primary focus. You know, we do research as an organisation, but we are a social impact organisation. 

[00:08:16] Santiago: But the social impact can be evidence based as well. 

[00:08:20] David: It has to be evidence based for us. You know, actually, I don’t believe we can do good social impact without elements of Evidence Based. And this is one of the big challenges we face because we could get a lot of our ideas out at scale much faster if we weren’t so focused on Evidence Based. But I’m scared. 

[00:08:41] Santiago: I’d like to highlight one example of that, which is Early Family Math app, which we got fantastic resources from our colleague Chris Wright in the US, we transformed them into an app.

[00:08:54] David: He did. We supported him to do so. We cannot take any credit away from him. He did all the work. He’s an amazing guy. 

[00:09:02] Santiago: He did it. I think Esmee supported him in the process, but never mind. Yes, he did it. Then the question is, okay, let’s take it out. But before taking it out, we wanted to test it properly. And that slowed the deployment. And we did an informal pilot in Kenya. And then we did a much more formal…

[00:09:26] David: We’re in the process. That’s only just starting now. That more formal pilot, there’s a little bit of funding behind it and it’s actually now got a bit of substance. It’s a slow process. It slowed the whole thing down.

Now, I think in this particular case, we’re maybe being overcautious. No, there could be value in getting things out more widely, but our nature is to want to be cautious about understanding how this adapts into different contexts, how it brings value. 

[00:09:53] Santiago: But we’re also opportunistic.

[00:09:55] David: Absolutely. 

[00:09:55] Santiago: I mean, not having that evidence for the app, if we had had a clear opportunity for deployment, we would have taken it probably. 

[00:10:03] David: And we’d have included elements to collect the evidence. But this is the thing, this is that balance between these, because actually doing so without collecting the evidence, and to be able to recognise and say, I’m scared of getting things wrong and not knowing about it. I’m not scared of getting things wrong and learning that we’ve got things wrong. 

[00:10:28] Santiago: Can you highlight that difference again? Because it’s quite subtle. 

[00:10:31] David: I am afraid of getting things wrong and not knowing that we’ve got things wrong and learning from it. I am not afraid of getting things wrong if we’re going to learn from it, so we’re going to improve. You learn the most when you get things wrong. That distinction is really subtle, as you say. It’s really important. But it’s part of the evidence base, you know. 

[00:10:53] Santiago: And that fear of mistakes can be paralysing to many people.

[00:10:58] David: Absolutely. And we mustn’t be paralysed by being afraid of getting things wrong. We have to recognise that to make progress, we have to have the courage to get things wrong and to admit we’ve got things wrong. And not only that, to be able to transparently get things wrong. 

[00:11:12] Santiago: And if we don’t get things wrong and we don’t learn from the experiences we cannot innovate. And innovation is central. 

[00:11:21] David: So, that subtle difference, you know, I really want to emphasize this element that getting things wrong, but being ignorant of them going wrong, I don’t want to find myself in that position. Because I’ve seen it too often, where I’ve seen people getting things wrong, and just because they have to sell their product, they have to sell what they’re doing, they brush over, or they ignore the red flags, the signs of things going wrong.

[00:11:49] Santiago: And sometimes knowingly. I don’t want to mention the name of a particular foundation that you were in a talk about. 

[00:11:57] David: Well, it’s, again, it’s often a question, in that particular case, who knows? And do they have the power to change? Or do they have the influence? There are people who because of the structures that they work in, cannot openly get things wrong. And certain funders don’t encourage that, which to me is just so wrong, especially if you’re looking for impact funding. You need to encourage that learning.

[00:12:23] Santiago: But not just in funding, we are driven a lot by the corporate world, where that secrecy is ingrained. 

[00:12:33] David: Well, secrecy, yes, but the corporate world is very open to getting things wrong. Actually, I would argue that’s one of the things they do very well. The big Silicon Valley sort of processes on this is fair and fast.

[00:12:45] Santiago: Yeah, we have in the Responsible AI series of the podcast, there’s cases of Amazon recruitment. 

[00:12:52] David: And they dropped it, and they were willing to drop it, and that was very responsible of them. So, you know, this is something, that particular element of getting things wrong, I would argue the corporate world is much, much better at that. And that’s why they’re deemed good at innovation, because they’re willing to fail. I’d argue they’re not necessarily great at the evidence base, using evidence base. 

[00:13:15] Santiago: I perhaps was referring more to individuals within corporations that in order to progress within a corporation, sometimes you cannot highlight your failures.

[00:13:26] David: That’s not always true, I believe. I know enough corporates to know that some corporates succeed by being open about that, so it’s not as simple as that. And as I say, there is an appreciation of failure. If it was the case that in the corporate world, failure wasn’t appreciated, then if you ever went bankrupt, you’d be a failure forever, but it’s not the case.

Most good entrepreneurs are serial entrepreneurs who have, at some point or another, failed. It’s all part of that learning process and having gone through that and recognising then, you know, that’s part of the experience building in certain contexts.

[00:13:57] Santiago: And you learn so much from those experiences. 

[00:13:59] David: Yeah, and my hope is to never learn quite from that particular experience. This is where I don’t see myself as a full entrepreneur, I have accepted the mantle of a social entrepreneur, but my obsession with evidence is part of where I think that’s my academic side still coming through.

And so I think that failing, where it builds the evidence base, where there is deep learning which is able to come out of it, that’s not a failure, that’s part of a learning cycle. And this is something which is really important, actually being willing to generate evidence, to understand it, to be able to be honest about that, to be able to try and dig into elements of the evidence base when they’re uncomfortable.

I like to say if you have a hypothesis and you do research and the research shows, just supports your hypothesis, it’s not very interesting research. This is the thing, you know. 

[00:14:56] Santiago: We had a member of our team doing a collaborative piece of research, particularly on feedback of online assessment of a specific topic in mathematics, integration. 

[00:15:07] David: Yes. 

[00:15:08] Santiago: And they did a study in Kenya, in Maseno University, and in Loughborough. 

[00:15:13] David: And New Zealand. 

[00:15:14] Santiago: My understanding was that it was going to be replicated in New Zealand. 

[00:15:16] David: Okay, maybe, yeah.

[00:15:17] Santiago: But the conclusion was there is no impact from that specific experiment. 

[00:15:23] David: On that particular misconception. And that’s so powerful and so important. 

[00:15:28] Santiago: And they presented it at conferences, and they published a paper, we didn’t find any results. 

[00:15:35] David: And that’s such an important result, because there’s a huge cost to building that sort of feedback and to improve it, trying to address those misconceptions. And the fact that it didn’t have an impact is an important result. 

However, what’s the learning to take away from that? And this is what’s so important. Is the learning to take away that, therefore, feedback from misconceptions is not useful? That’s one possible learning. Or is the learning that the feedback that was given in the way it was given was not effective? This is what’s so important about the evidence base. 

[00:16:08] Santiago: In this case, that can certainly inform how to replicate a similar experiment in order to narrow down the reason for that lack of evidence. 

[00:16:18] David: Absolutely. And maybe whether, you know, whether there’s elements where actually time could be more effectively spent by doing things differently. So recognising the value of results which are contrary to expectation, the most valuable research is research which either generates new hypotheses, which you didn’t have before, or results which challenges old hypotheses.

That’s where the learning really comes in. And so if you challenge deeply held beliefs, that can be really valuable research to explain, you know, this is the limits of what that understanding and what that context is. And similarly, if you generate new ideas through the research, you know, that’s really interesting and exciting research. 

So Evidence Based, for me, within IDEMS, it’s a strength we have, that we are deeply embedded in evidence approaches, but it is, as you’ve said, one of our greatest barriers to impact at scale. Because to have impact at scale with strong evidence is so much slower, so much more, so much harder. 

[00:17:39] Santiago: It is, but I’m thinking of the conversation we had about Viral Scaling. Imagine if that viral scaling, even if it starts slow, that exponential growth, even if it starts slow, is built on evidence based interventions. 

[00:17:56] David: And you are continually evolving, if your evolution is based on evidence, if you are actually able to direct the evolution based on evidence over time, which grows and so on, now suddenly, you can get viral scaling, which takes you down a road you don’t want to go. It takes you places where it’s out control and it’s not where you want to be, and there’s no way back.

Again, this is where the evidence, being able to use evidence solidly to be able to say no, this is what we know, this is how we’re trying to sort of gain evidence and gather evidence, it’s central to what we want to do. 

[00:18:30] Santiago: And it’s central to what IDEMS is. So many conversations we’ve had, both recorded as podcasts and not, were about IDEMS as an organisation, where, for example, the aspect of being fundamentally profitable, or the staffing models, the enabling staffing models, we always come back to yes, there was a publication, there was a study done about this, there was academic research done.

[00:18:56] David: That has informed this in different ways. This is again, it comes back to the Informed Decision Making. Part of that information is about using evidence, using information. Evidence becomes information, which is really important. It’s a deeply ingrained component of what we’re trying to do.

I don’t know whether it will hold us back too much, I don’t know what the price to pay is for this particular principle, but I do know the risk of not following it. And for me that risk is too great. 

And I want to be clear, there’s two elements to this. There’s one which is using evidence, which exists. And there’s the other, which is generating evidence. And it’s both sides of that. Evidence Based does include both of these. It’s maybe not as explicit as it could be but it is central, this is both sides. 

[00:19:51] Santiago: Can it stifle innovation, perhaps? 

[00:19:54] David: Yes, but it shouldn’t, and I don’t think it’s in contradiction in our context with innovation. What I think it does do is stifle the scaling of innovation. Really, innovation doesn’t need to be stifled, but, you know, that rapid, explosive switch from innovation to scaling, that’s what I would argue Evidence Based as a principle does slow that down. 

And I don’t know that that’s a good thing. But it is central to our approaches. And as all good principles, if a principle is only good, it doesn’t belong, there should be a cost. Every good principle is a compromise. 

[00:20:41] Santiago: I believe that this compromise is worth it. 

[00:20:44] David: For us, I agree. For others, I don’t know. 

[00:20:48] Santiago: Of course, I don’t think it suits every business. 

[00:20:50] David: No, it certainly doesn’t suit every business. Again, it’s one of those rare ones, there are some of the others, our Collaborative by Nature is another principle like this, where the cost of that as a business is maybe too high for most businesses to carry.

This is one where I think the cost is high for us, potentially, as a business. But the challenge of succeeding with this, if we can succeed with this, if we can succeed by being collaborative by nature rather than competitive by nature, there’s a payoff to be had in the long term if we can carry this.

[00:21:23] Santiago: What a case study it would be. 

[00:21:24] David: This is where these principles are to be studied. I would love to actually try and have people digging in to… 

[00:21:34] Santiago: It will happen eventually, David. 

[00:21:35] David: I think one day, we will have, we’ll be collaborating with business schools across the land, so to speak.

[00:21:43] Santiago: On that note…

[00:21:45] David: Thank you for this, this has been interesting. 

[00:21:47] Santiago: It has.