26 – The Informed Decision Making Principle

[00:00:06] Santiago: Hi, and welcome to the IDEMS Principle. I am Santiago Borio, an Impact Activation Fellow, and I’m here with David Stern, Founding Director of IDEMS. Hi, David. 

[00:00:16] David: Hi, Santiago. Looking forward to another principle. What have we got today? 

[00:00:21] Santiago: Today is the first one in the last group of four, it’s Informed Decision Making.

[00:00:28] David: Oh yes. This whole group we didn’t have for our first few years. This came later. 

[00:00:34] Santiago: And it came out in the first IDEMS retreat. 

[00:00:37] David: Exactly, the first full team meeting, where we actually started having a bigger team and we brought everyone together and one of the things we did, of course, was to go through the principles. And it was Chiara who was one of our Impact Activation Fellows at the time, now a Mathematical Scientist on the team, who said, it’s a big part of what IDEMS does and how it works, which I don’t feel is representing the principles. And that led to these four extra principles. 

[00:01:04] Santiago: And we discussed in a previous episode how transdisciplinary is potentially one of the most academic principles, but this group of four, it is about academia in some ways.

[00:01:16] David: No, it’s not. And this is important, where the four are distinct, and some of those distinctions are going beyond just academia. Of course, Evidence Based, which is one of the four, is very associated to research. But this isn’t really about academia, it is maybe more about research.

[00:01:34] Santiago: But it could be shared principles with a lot of academics. 

[00:01:38] David: Absolutely. Let’s dig into Informed Decision Making. 

[00:01:41] Santiago: Okay, so as I like to do, let me read out what the website says about this principle. This principle indicates the company’s deliberate efforts to include diverse forms of information while measuring their uncertainty as part of decision making processes. It intends to support balanced consideration of options while reducing our susceptibility to misinformation. 

[00:02:09] David: It’s a difficult one. This one’s a mouthful, isn’t it? 

[00:02:11] Santiago: I struggled to read it. 

[00:02:12] David: It’s one we struggled with in a lot of different ways. People were, you know, data driven is one of the big things. And we thought, no, that’s not who we are, that’s not how we are. This important distinction here about data versus information, and a lot of data people then undervalue qualitative. We’ve got an anthropologist on our team and this is something we value very deeply. 

[00:02:36] Santiago: Yeah, and that is reflected in the diverse forms of information.

[00:02:39] David: Exactly, yes. And it’s sort of understanding the different types of data, different types of information, serve different purposes and come with different amounts of information. Sometimes you want depth and often qualitative methods are better to go deep into an understanding. Sometimes you’re wanting to get just a sense of how a system works. And again, you need different information for that. 

[00:02:59] Santiago: We discuss in one of our first episodes that we recorded, I think, for another series, the different types of intelligence as well. 

[00:03:07] David: Absolutely. 

[00:03:07] Santiago: And different intelligences work differently with different forms of information as well.

[00:03:11] David: Absolutely. And one of the key words in this one, of course, is uncertainty, which is a really important part of how we work and what we do. And the fact here it’s about measuring the uncertainty. It’s about the fact that, you don’t just work with averages. 

[00:03:25] Santiago: Sorry, let’s take a step back. Because measuring uncertainty is quite a complex term. 

[00:03:31] David: Yes. 

[00:03:32] Santiago: So can you tell me a bit more about what that means? 

[00:03:35] David: A very simple way to explain this difference is this is the difference between mathematics and statistics. In some sense, if you think about mathematical models, they model how systems work on average. Whereas actually, if you think about a statistician, statisticians really worry about the uncertainty and the variability. 

And there’s a difference between those two, variability and uncertainty are playing different roles slightly. But the idea that when you have information your conclusions are still uncertain. That’s information, how certain are you about the validity of that information? How confident are you in it? Being comfortable with the fact that information you have may be the best information available, but it may still lead to misinformation, to misunderstandings. 

[00:04:24] Santiago: But you mentioned that qualitative information is as important as quantitative. In many contexts. Is it possible to measure uncertainty with qualitative? 

[00:04:37] David: In some ways, qualitative is a way of measuring uncertainty. And I still remember my wife’s PhD where she had her quantitative component and then she had case studies, which were some of the same people. And I still remember one evening she came back having had her hair done by one of the people she was interviewing who did people’s hair as her job.

And she came back and was furiously looking for the questionnaire that lady had filled in. And, of course, on her questionnaire was saying, have you ever experienced any gender violence? No. And all she was talking about was the problems in her relationship related to gender violence.

And so the case study really highlighted the uncertainty in the data that she had for all of the rest of the data, that she knew that there were participants who asked in one form responded differently to when discussing in another form. 

[00:05:31] Santiago: In that case, perhaps as well, some things could be considered gender violence by one person, not by another.

[00:05:39] David: Exactly. It’s the way you frame the question. But that’s exactly the uncertainty that you need to place on where your knowledge is coming from and how it’s obtained. I know a lot of people who overvalue, it’s not overvalue data, it’s sort of mistake data for certainty.

And actually, there’s a lot of complexity that comes into this. And so recognising uncertainty and living with uncertainty, accepting uncertainty in your knowledge, in what you know, and what you understand is something really important. 

And that’s where trying to measure that, one of the ways to measure certainty related to quantitative measures, is to do a small number of qualitative studies to try and dig in to understand how certain can you be about those results and how people interpreted them and so on. So that’s one of these diverse forms, helps you to get a sense of certainty. 

[00:06:29] Santiago: And what you’re saying resonates very much, at least in my head with the scientific method. In general, the data, the knowledge, tends to lead to a hypothesis that you are taking as correct, but accepting that there’s a possibility that it could be improved.

[00:06:45] David: I would frame the scientific hypothesis slightly differently to that, a good scientist never has a hypothesis that they believe is correct. This is the difference I would argue between science and pseudoscience. In pseudoscience you have a hypothesis and you try to prove it’s correct. A good scientist has a hypothesis and says, I can’t yet prove how this is wrong. And you’re trying your best to show where it is wrong. 

[00:07:15] Santiago: That’s what I was trying to say.

Okay. 

[00:07:19] David: But it’s this essence of recognising and, you know, a good scientist recognises that all hypotheses are wrong, but you might not yet know why or how or where. My favourite example of this, which a lot of people understand, is Newton’s mechanics. For hundreds of years people couldn’t show how it was wrong. And then Einstein is one of the best examples of where relativity came out to show that actually if things go very fast, if you’re at the extremes, then what you take as your rules don’t apply in the same way. And the context within which the rules don’t apply was well understood, and there was a hypothesis which was better. 

You don’t feel this is a common example? 

[00:08:05] Santiago: No, I I would say it’s not my favourite example. My favourite example would be Euclidean versus non Euclidean geometry.

[00:08:12] David: Okay. 

[00:08:12] Santiago: Which is very much related. 

[00:08:14] David: But I would argue, and much as I love thinking about geometry, and it should be really well understood, Euclidean versus non Euclidean, let me just explain why this is so important. The word is not flat, and so actually, think about straight lines in the world we live in, they do meet twice. Parallel lines meet twice on a sphere. 

[00:08:36] Santiago: And we’re digressing. So let’s get back to Informed Decision Making. There’s two sentences in the description. The first one is about how to use information with the measure of uncertainty as part of decision making. 

[00:08:54] David: Yeah, and I suppose maybe, I’m not sure, the wording of this one, this came later, it’s not gone through as much constructive criticism as maybe some of the others. And so I don’t feel the wording is quite right on this yet. It’s not bad. I don’t like measuring the uncertainty necessarily, because the importance isn’t measuring it as much as recognising the uncertainty. 

So I think there’s elements of the wording which I think will probably evolve and it’ll probably improve. But the importance of uncertainty is central to this and that’s where that has come out clear. 

[00:09:25] Santiago: Maybe I’m completely wrong, but I’m reading that it’s as important to know what you know and know what you don’t know. 

[00:09:32] David: Absolutely, being aware of what you don’t know, the limits of your knowledge, and I always say, the biggest thing education has given me is a sort of awareness of all the many things I don’t know. I’ve learned many more things that I don’t know than I have gained some knowledge along the way. But it’s that awareness of how much there is out there that I don’t know, and that respect for those things that I know I don’t know is really one of the things that comes. 

And this is the uncertainty, being comfortable with things that you don’t know and maybe you can’t know. That’s difficult, some people look for certainty when actually accepting uncertainty is important.

[00:10:10] Santiago: Yeah, and we were having a couple of days ago a conversation about, what was it about? Was it the one about solar panels? I realised that there were a lot of uncertainties that I wasn’t aware of. And I hadn’t considered the benefits and disadvantages of domestic solar power. I had made a judgment without considering a very important point that brings my whole judgment into question. 

[00:10:38] David: One of the things is, at some point, you need to make a decision. And recognising that that decision is going to be made with uncertainty, rather than the fact that actually, quite often, once people have made a decision, they feel very strongly they made the right decision, they knew everything there was to know about it. 

Whereas actually, I find, one of my strengths is recognising that quite often I have to make decisions where I know I don’t know enough but I’ve done the best I can with what I know and I have an awareness of the uncertainty, and I’m pretty confident that I haven’t missed out anything really big. So it’s not a terrible decision. 

But maybe it could have been a better decision if I had other information that I don’t have. And this is where an Informed Decision Making is important. It doesn’t need to be the best decision. This is where, again, we are very privileged, we don’t need to maximise.

[00:11:29] Santiago: Yeah. And that’s very important, what you were just saying, it’s about the Informed Decision Making, because in my judgment of domestic solar power I had fallen into misinformation. 

[00:11:43] David: And this is one of the key things that we are aware of in our society right now, there’s… 

[00:11:49] Santiago: Huge amounts of misinformation, particularly fuelled by social media. 

[00:11:54] David: Social media is part of it, yes, but it’s also fuelled, and one of the things that worries me, is that there’s whole industries of misinformation because of particular interest groups. There’s misinformation, which is quite often, benign’s the wrong word, it’s come naturally just because some people don’t understand things as fully or they haven’t thought things through fully, but it’s the best of their current knowledge. But there’s also misinformation which is designed to misinform. 

Both of those are things that I’m worried about in our current society. I am genuinely aware how easily I could be misinformed, and try to protect myself against that by sometimes not taking a stronger position as maybe I would like to take. And part of that is making sure that I’m as informed as I can be.

[00:12:48] Santiago: And even reliable sources, or what we consider reliable sources of information, are not as reliable anymore. I mean, I’m from Argentina, and you watch one news channel, you watch a piece of news presented by one news channel and then you watch the same piece of news presented by different news channel, and it can be completely different news. What do you take? Which one is correct? Which one is right? 

[00:13:18] David: If you want to take statistical methods on this, you know, a sample of size two is too small. Watch some more news channels. Now the problem is that other news channels randomly distributed. And so, you know, if they were biased within certain ones, and so on, so… 

[00:13:31] Santiago: They’re biased in every one of them. And they are fuelled by specific interest groups. 

[00:13:36] David: Exactly. And understanding those specific interest groups and how they fuel the different news channels, that’s really hard. This is one of the reasons that people turn to social media. But social media is even more fuelled by special interest groups of much smaller magnitude. You’ve got to be pretty powerful to fuel a news channel, whereas actually within social media there’s a different process at play.

[00:13:58] Santiago: And there’s technological processes at play as well. 

[00:14:01] David: Absolutely. So I’m very worried and conscious about susceptibility to misinformation. And I don’t have any particular silver bullets to solve this. There aren’t any. 

I guess one of the things which is important is that if we’re looking at Informed Decision Making as being an IDEMS principle, which it is, what’s this in opposition to? What else is a sensible way to make decisions? Is Informed Decision Making the only sensible way to make decisions? 

[00:14:30] Santiago: Several times I heard you make decisions by instinct. 

[00:14:33] David: Absolutely. And some of those instinctual decisions, and this is one of the things that I would argue for myself, these are informed, and part of that information is the diverse sources of information, and recognising when your instinctual approach can actually give you solid. And I think one of the things which is really important here is if we take some of our other principles, Consciously Ethical. 

So what about conscious decision making? Explainable decision making? Justifiable decision making? All of these could be sensible as well, and they’re not what we’re talking about. Data-driven decision making, should you be using data? That’s not what we’re saying. I’m not saying other people shouldn’t do this. One of the things with Informed Decision Making, justified decision making, should we be able to accurately justify all the decisions we’ve made? No. 

One of the things with Informed Decision Making is it’s actually quite weak compared to justified, for example. And I would argue in other contexts, I would really want people to be making justified, at a governmental level or international level, I want them to be able to justify the decisions they’re making. I want them to be doing more than Informed Decision Making. 

[00:15:49] Santiago: But we often, we have the weak principle but we often go beyond it. 

[00:15:54] David: Absolutely. 

[00:15:55] Santiago: Try to get to justified decision making. 

[00:15:57] David: I don’t know. I mean, there’s a cost to it, there is a real cost to that. What we do want to make sure we avoid is decision making which is ill informed, or which is through misinformation or miscommunication and so on.

There is an element of consciousness to it, which is important. That’s the informed part. But it is deliberately at the stage where there is a recognition that informed is enough for us at this point in time. Maybe this will change in the future, maybe as a bigger organisation, maybe as an organisation which has more influence, this is a principle that might need to change. 

[00:16:40] Santiago: And as we grow, we have set up some processes in place that don’t require to measure uncertainty necessarily for decisions. 

[00:16:50] David: Absolutely. And this is where I feel measuring uncertainty is something which I’m not that comfortable with. Maybe while being conscious of uncertainty or while reflecting on uncertainty might be a better wording of that. 

One of the things we’re trying to do with the Informed Decision Making, or making this weak, is allowing more people to make decisions, and being able to share that decision making load. And that’s one of the reasons this isn’t maybe as strong as it might have been otherwise, but we need people to be informed and to be gathering information about this from diverse sources. 

[00:17:22] Santiago: Yeah, and now that you mention it, sorry, I’m going back to your previous point about measuring uncertainty, have we actually gone through a measuring uncertainty for any decision that we’ve made?

[00:17:34] David: It’s not measuring the uncertainty of the decision. If you look carefully at this… 

[00:17:38] Santiago: Measuring uncertainty for any decisions. 

[00:17:42] David: As part of the decision making process, we’re measuring uncertainty of the information. And this is something that we do generally try and do. Pretty much, when we look at different forms of information, systematically, I would be trying to do this. Now, how conscious that measure is, how explicit, how formal it is, that I’m not sure, and that’s why I’m not that happy with the word measure. 

Assessing might be a better word. We’ve gone through this, we’ve got a new full team meeting coming up soon and we might want to change that word measuring because it doesn’t capture what I think is important. What I think is really important is the consideration of uncertainty. 

And the measuring of uncertainty is distracting us, because it’s not about how you measure it, it’s about how you consider it. So this would be an example of a process where going through these discussions, going through these presentations, helps us to improve our wording. Maybe by the time the podcast actually gets released… 

[00:18:37] Santiago: The principle wording has changed. 

[00:18:38] David: The principle wording might have evolved. One of our principles, Continually Evolving. And this is the sort of thing where I feel that in this discussion, we’re really highlighting the fact that uncertainty, the consciousness of the uncertainty is important, but the measurement of it is maybe not what we’re wanting to get people to focus on. 

[00:18:57] Santiago: And let’s not get distracted that much. It’s about making a deliberate effort to include diverse information, measuring uncertainty. It doesn’t mean that every single decision has to… 

[00:19:10] David: Let me be clear on what I think, what I understand it as saying. It is we definitely need to make the effort to have the diverse forms of information. So you don’t want information from a single source, from a single approach, from a single method. 

And whenever you have information, you want to have an idea about the uncertainty of that information. You don’t want to take that information as fact. That’s the important aspect. You want to know how reliable is a particular piece of information you’re considering.

There’s different elements of that which you can measure which relate to the uncertainty. But being conscious that any information you have has an element of uncertainty and some informations are more certain than others, and how you get that data really matters. 

This is one of the elements where people actually often forget this when they get into artificial intelligence, it’s really scary to me. Artificial intelligence is a wonderful tool, incredibly powerful, but people take what’s coming out assuming that it doesn’t have uncertainty, it’s right. 

[00:20:14] Santiago: So many times I asked a particular generative AI system, to write some code for me and it’s full of bugs. So if I don’t read and analyse the results, then I’m likely to not get the use that I intended to get. 

[00:20:32] David: Absolutely, and being able to deal with uncertainty in what comes out, okay, doing that for code, as you’re suggesting, that’s not quite related to what we’re doing. But you could, as part of your information gathering processes, related to a decision we’re wanting to make, it’s absolutely sensible to ask AI in different ways as a component of that. But you have to really be careful of recognising the uncertainty. 

And one of the problems with AI as a source of information for that, compared to other sources of information, is sometimes measuring the uncertainty of what comes out of AI is hard. It’s not made as explicit, it’s not as easy to understand that. And that means that in our context, if we can’t consider that uncertainty, if we can’t measure it, then it’s harder for us to use it in the decision making process. We’d need to trace it back, maybe, to somewhere where we could. 

[00:21:21] Santiago: Can I ask you a final question before we close?

[00:21:23] David: Of course. 

[00:21:24] Santiago: Does this apply potentially more to higher level decision making than the lower level, more strategy or big thinking? 

[00:21:35] David: No, I would argue that the ideas behind Informed Decision Making are universal within our current structures. Maybe this is where the measuring of uncertainty is confusing in this, but I would argue that this is what we want to instill at all levels of decision making because the whole point of the processes and the practices we try and set up within IDEMS is to enable decision making to be made by many people within the organisation. We’re wanting this sort of distributed decision making structures.

And so part of what this is, we hope, doing is helping people to recognise what they need to do to make decisions. Now to make decisions we need to be well informed. We don’t need to be perfect. We don’t need to have everything. We don’t need to justify everything. But we need to be able to be well informed and we want to be able to go through processes to be well informed and part of being well informed is accepting uncertainty and being able to have a conscious, in this case, measure, of how certain we are about what we’re using as information in our decision making processes so that we can actually then discuss with others what informed the decision making process.

So Informed Decision Making, as a protection against decisions related to misinformation is important and it’s important whether it’s a micro decision or a macro decision. It’s a really practical and constructive principle. It’s one of the weaker principles because informed is a weak word for decision making, and that’s very deliberate.

It’s still powerful. 

[00:23:20] Santiago: It is indeed. Once again, what a fascinating discussion. 

[00:23:24] David: Thank you. 

[00:23:25] Santiago: Take care.