25 – The Informed Decision Making Principles Set

[00:00:07] David: Hello and welcome to the IDEMS Principle. I’m David Stern and I’m here with my co founder Danny Parsons and we are going to discuss Informed Decision Making and the associated principles. 

[00:00:21] Danny: Yeah, our last set of principles that we’re doing, the last set that we developed.

[00:00:27] David: We didn’t have these when we started. These are the only four principles which as a whole set didn’t exist when we started. And it was at our first full team meeting in 2021, when one of the team sort of pointed out that a big part of what we do around evidence, around decision making, around critical review, was not captured in our principles. And this was an important part of her perspective of IDEMS and what she felt and the principles she felt that were behind IDEMS, but it wasn’t captured in our principles. 

[00:01:09] Danny: Yeah, and nice how it kind of evolved from the meeting and from other members of the team, sort of getting others input on the principles and that they can change and be added to and adapted.

[00:01:20] David: Absolutely. And I suppose from our perspective, it’s not that these things weren’t important beforehand, it’s that we didn’t consider them even as principles. There’s always lots of things you care about, but the things you choose as principles, especially guiding principles, they’re about guiding how you act.

[00:01:40] Danny: Yeah. It is a tough one to know what’s principles and what is just, yeah, things that you do and things that you believe in. But I think this set was felt to be important. 

[00:01:50] David: And I think we’ll see that as we go through them. Should I quickly read through? So Informed Decision Making is the sort of overarching one and within that we have Evidence Based, Developing Research, and Critically Assessed. 

[00:02:06] Danny: Yeah. One comment I kind of feel from them taking them as a whole is that they’re very much linked to academic ideas in a sense. But I quite like also how we’re not using really those academic terms. 

So particularly Critically Assessed, the last one, which is sort of like peer review in academia, but I quite like that we’re not sort of, we’re showing the importance of that academic side and the research side, which comes from academia. I think that is important because that’s part of our background, but not just using those academic terms, which might seem more narrow, I think is really nice.

I really like Critically Assessed because peer review is a type of being critically assessed. But there are lots of other ways to be critically assessed, which doesn’t exclude people who are not academics. 

[00:02:57] David: And I think very explicitly with the Critically Assessed, this is this desire and appreciation for criticism. And it’s that recognition that that makes us better. 

[00:03:09] Danny: Yeah. And it links, of course, to lots of other principles, but this, you know, links to the openness for me as well, that, you know, we can’t be critically assessed unless we’re opening ourselves up for assessment. And the things that we develop and do, people can look at them and see them and assess them critically. 

[00:03:27] David: And a big part of that sort of critical assessment is what will enable us to make informed decisions as we’ve tried to frame that, because one of the things for us with Informed Decision Making it really is about having this diverse set of forms of information that sort of come in.

[00:03:48] Danny: Yeah, yeah. And I think that the sort of Developing the Research part is an interesting one there as well, because that’s really showing that importance of the academic side of things and then the kind of research side of things, where we’ve talked about sort of our separation a bit or us being outside of that academic sphere, in a sense, with IDEMS being a social enterprise.

But, we’re very connected to researchers and research, and this shows, I think, how important we feel, we feel that is. 

[00:04:22] David: But developing research is a rather tricky one in a sense, because we’re really taking the research from the development angle and we’re recognising that traditional research methods are not always what’s needed if you’re looking for impact studies. And within the UK, the Research Excellence Framework is giving more and more important to impact case studies. 

But if you really want to do impactful research, we need new forms of research. We need to develop the way research works with the way that data is evolved. So this element also within Developing Research, you’ve been very closely tied to research, as you say, and intertwined with it, but also recognising that there’s need for research to evolve.

[00:05:15] Danny: And I think this is the one where we explicitly got data in there. I was mentioning that also, I remember people thinking that that’s such an important part of the world now, but also what we do and our work, you know, not all of our work is statistics and data science, but there’s data and that kind of research and evidence at the sort of heart of it.

And yeah, and you’re talking about these new ways of using data. That was something that I think other people also felt really important that isn’t really mentioned anywhere else. 

[00:05:44] David: Yes, and I think maybe very explicitly on the evidence point you said, you know, evidence based is the most maybe common of these. Why would you want to do anything which isn’t evidence based? And one of the things that I think is really important is that, well, as a company, companies do all sorts of things which aren’t evidence based because evidence is really expensive. It’s really demanding to require those levels of scrutiny, of academic rigour, if you want.

So evidence based and searching for evidence based is actually, in some ways, there’s sort of different ways to interpret this, and people use this in different ways. 

[00:06:28] Danny: Yeah, and I think there’s also a bit of a tension there in this with things like innovation, new things and trying something new and trying to be, you know, do things outside the box, which people may see as sort of… 

[00:06:40] David: In contradiction. 

[00:06:41] Danny: And this is a bit restricting in a sense that having things evidence based. 

[00:06:45] David: But it’s not evidence based decision making. This is really important. It’s Informed Decision Making. 

[00:06:51] Danny: Yeah. 

[00:06:52] David: There’s this element that underlying things that we do, we want an evidence base and we want to be contributing to the evidence base. But our decision making is simply informed. These were very interesting discussions when we had it. 

[00:07:05] Danny: Yeah. When I see this, I also think about this kind of slogan from the tech world, which, you know, go fast and break things, it’s sort of, one of the slogans that came out of the tech entrepreneurship, and that’s sort of, you know, try things and just go and worry about the sort of consequences later.

I think something we discussed in a previous podcast about actually holding back and even sometimes sort of stepping away from work or things where we’re not sure that there’s going to be a positive impact. I think this sort of, I feel is a really good aspect to have, which shows us maybe different from other types of companies and businesses.

[00:07:43] David: And particularly in tech. Tech is very fast moving and tech does not look for evidence, it doesn’t look for the best. Often it’s very much about good enough. 

[00:07:53] Danny: And sort of trying stuff and worrying about the effects later. You know, one thing might work and be the big breakthrough, other things, you know, we’ll just try them and see what happens. And especially, you know, with our work in a development space as well, that can be pretty dangerous. 

[00:08:10] David: Absolutely. And it’s this balance between supporting innovation while having that evidence base underlying it. 

I guess things we haven’t dug into, which would be nice to just do briefly now, is to sort of say, well, okay, what are these in contradiction with? Maybe not contradiction. When we’re thinking of principles, we’re always wanting to be able to say, if this is guiding us towards evidence base, then I think you’ve articulated this is sort of guiding us, in some sense, away from that do it fast and break it approach. Which is not bad, it can be very good at sort of scaling, you know…

[00:08:51] Danny: Yeah, and in maybe areas where, you know, there’s less risk of things going wrong. 

[00:08:57] David: Exactly. 

[00:08:58] Danny: In terms of real things going wrong, other than the efforts you’re putting in, then, you know, you’re trying to get ahead and you’re trying to always beat your competitors.

[00:09:06] David: Absolutely. So it is absolutely clear we understand why many people would not want evidence based as one of their principles. It does slow us down in certain ways because we’re looking for this, but it’s not so much that it slows us down, I would argue, because it doesn’t exclude innovation. It’s just that we’re wanting to, I guess, we don’t want to be going beyond, so far outside into the unknown. 

[00:09:32] Danny: Maybe not cautious, but a sort of careful consideration in decision making and, you know, we’re not having to be so aggressive in, you know, making huge profits and things like this. 

[00:09:44] David: And this comes back to our Informed Decision Making about looking at these diverse forms of information and looking at it from multiple perspectives, understanding, and even measuring uncertainty. 

[00:09:54] Danny: I suppose, you know, looking at that Critically Assessed one, I guess, people’s argument against this, you know, this opens you up to unfair criticism, this opens you up to being attacked, your reputation being damaged. You’re exposing yourself and even maybe criticism, which is fair and constructive can give you a bad reputation or some people may see this in a more negative way. 

So this is certainly not common to be so open about so many things that you’re doing and things like this, you know, if our code bases for different software that we’re developing are open and, you know, people could come in and talk about the quality of our work and things like this and, yeah, whether that’s right or wrong, most, or many people, would not want that kind of exposure and not really see maybe any possible benefits versus the risks.

[00:10:48] David: And in some sense, it is our confidence, not that we are above criticism, on the contrary, we would be very critical of ourselves, we know that we’ve had to do a lot of things faster and we can do them better or we’d want to put time in. It’s that appreciation of if other people are putting that time into critique, well that could enable us to do better in the future.

[00:11:09] Danny: Yeah, and I think if there is that criticism which could be damaging or where we’ve done something that isn’t of good quality, I think we have to believe that we would want to know that and we want to improve and we wouldn’t want things out there which are not as good as they should be, or what people expect from us, or, you know, and they’re not working as well as they should be.

[00:11:31] David: Exactly. 

[00:11:31] Danny: And yeah, we have to have that sort of confidence a bit in ourselves that we can stand up to that. 

[00:11:37] David: And where that criticism is justified, we can hold our hands up and say, yes, we will deal with it. 

[00:11:42] Danny: Yeah. And I think the people that we sort of have relationships with, and collaborations with, I think, hope that that would be understood as well. 

[00:11:52] David: And I suppose that’s an element of our reputation, where we’re less worried about the general public’s reputation, how they feel about us, they don’t know about us really, at this point. But what we are more worried about, is, you know, our colleagues who we work with, who we build the long collaborations with, our long term collaborators.

And most of them, they have an academic component to them. And so if we were to find we were really critiqued on something and we were to go to them and say, you know, yes, we found they criticised us on this, we’re going to work on it. This is not going to be negative for those long term collaborations.

That reinforces the trust, that reinforces the fact that, okay, yes, we may not get everything right, but they can now go out and search out other experts who could criticise us. And we can take that on board and we could build from there. 

[00:12:45] Danny: Yeah. And I think also, with the way maybe some trends are going, there’s a lot of bodies and organisations which are embracing this kind of thing that you can, you know, submit your work and projects, software, to get on, you know, these lists of kind of… 

[00:13:01] David: Well, the one we’re aiming for over the next year or so is this public… digital public good. 

[00:13:05] Danny: Yeah, exactly. And those kinds of things where you know that it is going to be critically assessed, then you need to have that openness.

[00:13:11] David: And you want that, you know, this is positive. We’re really, I wasn’t really up to speed with how that could actually be beneficial to a lot of the projects we’re working on, but now we’re going through this process ourselves and with our collaborators. 

[00:13:26] Danny: Yeah, so I think it’s not just us sort of saying, this is a sort of trend, not just in academia now but in other areas.

[00:13:33] David: And it’s a way as well, I believe, of building sustainability into some of the tech sphere that we’re working in. 

So, I guess, Developing Research is a difficult one for us to sort of say what the opposite would be in a sense, because this is less clear, I think, what, well, for others to do. We’re in the privileged position that we have high levels of academic expertise within the organisation and we have the experiences across different disciplines to be able to be looking and sort of critically evaluating research as it’s happening.

Most non academic organisations you would expect to be fitting into and accepting research as it happens. 

[00:14:27] Danny: Yeah and using the tools that are established and available. That’s not really your job to develop new ones, you’re sort of relying on others to develop, and you follow best practice.

[00:14:37] David: So this is a very unusual one in all sorts of different ways, and it’s one which emerged, I would argue, because we are doing it. It’s not that we’re doing it because the principle exists, because we’ve taken the principle. The principle exists because we were doing this, and you and I were involved in elements of doing this before IDEMS. 

[00:15:01] Danny: Yeah. 

[00:15:01] David: And we’ve got sucked into that area where this need, and the need, which is emerging, I suppose, from the world of data science, it’s this world of big data, it’s the new data sources, and the way data is emerging, transforming the world we live in.

And the fact that sitting between, if you want, the words of research and the words of data science, we recognise that there are elements of this where research needs to take advantage of some of this and we need to be bringing in some of these new data sources into the way knowledge is developed. And we have a unique skill set, maybe not unique, but we have an unusual skill set which is enabling us to help partners and collaborators use different data sources to build knowledge. 

[00:15:58] Danny: Yeah, I think, as you say, it is that we have that, unique sort of intersection of skills, as you say, that means that we are in the position of being able to do this and we should be supporting sort of the groups on both sides. 

[00:16:14] David: Yes.

[00:16:14] Danny: As you say, the sort of development implementation sort of side and the data science and academic side. 

[00:16:22] David: Interesting you put it that way round because I was thinking of it more as the sort of data science versus traditional research, whereas you’re putting the data science with the academic. 

[00:16:33] Danny: Yes, I suppose academic is probably on both, I’m thinking of academic data science, but then you’ve also got sort of academic researchers. 

[00:16:41] David: Maybe this is too abstract. Maybe we should get really concrete on this. In the data science world, when you look at big tech, there’s a concept of A/B testing, which has been used, which has come out, which has emerged as part of market research in fantastic ways, where you can take a product and have some of your users who come and visit your website see it in one way, and some people see it in a different way, and based on the time they spend on the website and the rest of it, you can make decisions as to what the best way is to display your website. 

Oversimplify it a little bit, but that’s really this concept of A/B testing which has emerged in the sort of data science world of big data related to tech, and so on. On the other hand, you have Esther Duflo, who won the Nobel Prize in Economics because of randomised control trials. And actually when you dig into it, it’s actually not so different. You’re wanting to have comparable things where you can compare between your control, what you had before, what you now have. And applying that to development has become an incredibly big part of public health and many, many academic, if you want, studies.

[00:18:01] Danny: And medicine. 

[00:18:02] David: Well, it was already part of medicine. Esther Duflo brought it to development, or international development and social sciences in general. But the point is that these two, there’s a whole spectrum between them, where if you’re, as academic researchers, you’re focused on the perception of what a randomised control trial was, then you’re missing the opportunities that come once your interventions have tech embedded and where you could be bringing in some of this A/B testing but making it more rigorous so that you’re actually not just getting preferences in that sort of way, but you’re getting towards something which is really academically rigorous and relates to deeper understanding.

So that spectrum between these two, this is where a lot of innovation is now possible and happening. And this is what we mean in some sense by having knowledge of both sides and being able to bring people together. 

[00:18:59] Danny: Yeah, and I think it fits with other, you know, we’re often playing that kind of bridging role and this collaboration and things like this that we’ve discussed before. We do seem to have this sort of uncommon intersection of skills and experiences in a lot of different dimensions. And I think that’s often where we find ourselves adding a lot of value. 

[00:19:18] David: I think that sort of maybe comes back to this first point, the overriding point, which is probably the right place to finish, about Informed Decision Making.

When you say that we add value, quite often the way we’re adding value is actually helping our partners with informed decision making by considering other sources of information, by thinking about things which may be outside of their domain expertise and bringing in knowledge from elsewhere. 

So this idea of Informed Decision Making not just for us but helping our partners make informed decision making is I think maybe really what that leads to and why we act like this.

[00:19:58] Danny: Yeah, yeah, I think that’s right. I was sort of going to mention uninformed decision making. I suppose the sort of counterside of that is your kind of gut instinct and your intuition, which you maybe can’t articulate exactly where that decision making comes from.

But I suppose that’s a sort of common thing in business that people want to act on their gut instincts and things like this, where they might see informed decision making sort of holding you back. 

[00:20:25] David: And I think there’s a, there’s a really interesting tension there, I feel, because actually I don’t believe that, you know, gut instinct isn’t necessarily informed. I’ve had experiences personally where I have had a gut instinct which I didn’t understand and it’s taken me a long time to understand why my gut instinct was more informed than my intellectual decision. And that’s, to me, part of this diverse sources of information. 

I would argue going against your gut instinct, that is not going to be because of the information you have, that wouldn’t be informed decision. You want to understand your gut instinct, you want to dig into it. 

[00:21:07] Danny: No, I agree. Yeah, I think your gut instinct, you know, comes often from your experiences, it’s just, you maybe struggle to articulate why. But it’s quite hard then to know when you’re following the principle, sort of understanding your gut instinct, I think… 

[00:21:20] David: Exactly. And this is where we have tools to do this. I mean, not everybody likes our pathway approach, but our pathways are exactly an ability as a group to dig into and to try and get people to make explicit those gut instincts, that understanding what they think will happen.

[00:21:39] Danny: This is your pathways of change that’s leading to theories of theory of change. 

[00:21:43] David: Within a theory of change concept, we have a particular rigorous methodology for pathways, which is really about making and embracing people’s gut instinct and encouraging them to be really critical of what they’re seeing to be able to actually communicate exactly that.

So that to me would be a really nice example of where we’re not doing informed decision making in contradiction with sort of gut instinct decision making. 

[00:22:12] Danny: Yeah. 

[00:22:12] David: What we are saying is if you have gut instincts and you want to work as part of a team, you probably need tools to help you communicate and explain to others where that gut instinct is coming from.

[00:22:24] Danny: Yeah, and I mean, I feel we often see, or you often see bad decision making being led by data or information, which hasn’t had a lot of thought behind it. And actually, you sort of feel sometimes people use that as an excuse not to have to think about, you know, I’ve got some data, which says this, so that’s the best. This number is bigger than this number. And they generally simplified problems down to a level which isn’t helpful. And you’re still taking out the complexity of the problem. That looks like informed decision making and evidence based, but it’s… 

[00:23:00] David: Well, no, this is, I think, what’s so important. And this is where we don’t have evidence based decision making, because that’s what often happens in that context. We have Evidence Based, so we like evidence, we appreciate consideration of evidence. But the informed decision making is not based on, it’s not exclusively data, it’s not data based, data driven decision making, this sort of thing.

[00:23:22] Danny: Yes, those kind of terms, yeah, always make me a bit nervous. 

[00:23:24] David: Exactly. It is this idea of considering and appreciating multiple sources of information. 

[00:23:30] Danny: And you often then lose a lot of information just by, you know, going by data. 

[00:23:35] David: What data is available? 

[00:23:36] Danny: Yes, exactly. You rarely have data that’s actually necessary, and so on. 

[00:23:40] David: You’re absolutely right. And I’m really glad you brought that out and highlighted that as a way to sort of finish this episode. Because I think that that point that our informed decision making is somehow in contradiction with gut instinct decision making, which you don’t investigate. We would encourage you to embrace your gut instinct, to turn that into information… 

[00:24:07] Danny: And you need to know when that might be wrong and when there’s information you haven’t considered, which is relevant, and so on. 

[00:24:13] David: But it’s also in contradiction with data driven decision making, or with evidence based decision making, where you are excluding those elements of gut instinct. Somehow it’s in between these two, is where we position this. That we want to value those gut instincts, but we want to investigate them. We value evidence based, data driven, but not for the decision making. You’ve got to be more holistic in what you’re looking at in the source of information you take.

So I think there’s a sort of, there’s complexity. 

[00:24:44] Danny: Yes, that was sort of what I was thinking of in terms of, yeah, finishing off that I think we’ve really dug into quite a bit of complexity, this is quite layered and complex. It’s nice that we’ve talked this out.

[00:24:58] David: And it’s subject to misinterpretation, and we accept that. And we’re happy for people to critically assess us and critique us for our interpretation of some of these. And it will maybe help us to articulate these better. 

[00:25:10] Danny: Absolutely. 

[00:25:10] David: And we want that critical assessment to help protect us. And this is where I think the final thing on this Informed Decision Making is: the informed decision making is not about making the optimal or the best decision for us. One of the things is we’re wanting to reduce our susceptibility to misinformation. And that’s something which I think is key to maybe finish on, that accepting this complexity, embracing the critical assessment, you know, having new forms of evidence emerging from research, which takes in alternative sources of data and so on, and really using that as a basis. But allowing our decision making to be informed by this whole variety of factors, we hope will enable us to deal with the complex world we live in.

[00:25:57] Danny: Nice summary. 

[00:25:59] David: Thank you. 

[00:25:59] Danny: Great. Good way to finish. 

[00:26:01] David: Great. Thank you.