29 – The Critically Assessed Principle

[00:00:06] Lily: Hello and welcome to the IDEMS Principle. I’m Lily Clements, an Impact Activation Fellow. I’m here with David Stern, a Founding Director of IDEMS. Hi David. 

[00:00:15] David: Hi Lily. Which principle are we discussing? 

[00:00:19] Lily: Critically Assessed. 

[00:00:20] David: Oh yes, okay. This is one of the new ones. 

[00:00:25] Lily: Well, you know, a few years old now, nearly coming up to its third birthday. As I remember, this came out of a team meeting and Chiara, one of the colleagues asking, we don’t have any about… 

[00:00:35] David: It was the Evidence Based, which was really the one that she said, you know, this is central to her work, she believes it’s so important, why don’t we have a principle related to this? And actually, then that got fleshed out into four different principles. So we went from 16 to 20 principles. 

[00:00:52] Lily: Yes, let’s hope that we don’t keep fleshing it up to more and more. 

[00:00:56] David: Well, I don’t know. I mean, this is the thing, I think this Critically Assessed one is so important to me now. It’s one of the new ones, but it’s one where it’s complex, it has complexity, which I always value, and it was missing, and it’s central to sort of a lot of things we value. 

[00:01:14] Lily: How about you read out or explain what this principle is?

[00:01:18] David: Okay. ‘This principle conveys the company’s positive attitude towards being peer reviewed. It expresses an appreciation for criticism, which enables reflection, leading to opportunities for learning.’ 

[00:01:34] Lily: Well, I remember when I started working with you, which is just after my Bachelor’s. And I was writing an application to do my Master’s and you said about that you could read it over and I was terrified. I didn’t want anyone else to read it because I didn’t want that criticism. 

[00:01:49] David: Yes. 

[00:01:50] Lily: And this was obviously before IDEMS was built up, but then it became this whole, you know, well, we need to have that kind of critiquing. This is an important part of being able to grow. 

[00:02:02] David: And I know you well enough to know how much that’s changed. 

[00:02:06] Lily: I did not like that experience.

[00:02:10] David: But the appreciation of, you know, I love this and this is probably one of the more academic principles in many ways, but it is not just applied to academia. I’ve got lots of PhD students who have come through, all research students at different levels who have come through. And one of the things which they’ve always struggled, I think, as you’ve mentioned, to sort of reconcile in their mind, especially the first time they write journal articles.

You know, they write a journal article and they hope the reviewer has no comments for them. And I said, what? If a reviewer takes the time to give you lots and lots of comments you should be saying thank you. Whether or not you agree with them, they have put time and effort into helping you make your paper better.

This is sort of where this approach comes from. So, if you want to be the best you can be, you will only get there by being critiqued, it doesn’t matter how good you are. And this is, I think, one of the things which really distinguishes top academics, they appreciate and value peer review and the critique of others. Not because necessarily they agree with the critique, that’s a whole different thing. But if somebody’s taken the time to critique it, it means that you should be able to improve it. 

[00:03:34] Lily: Really nice sentiment. 

[00:03:36] David: Exactly. Yes. And if you can improve it, then you should be grateful for them. And it might be because they misunderstood it, but if they misunderstood it, others will misunderstand it in the same way. And so even if you’re not trying to address the comments as they feel it should be addressed, you can make improvements to make sure that your perspective is better presented or represented on the page. 

Now, this is a general approach. It’s really hard sometimes as individuals not to take things personally, to sort of say, ‘oh, the reviewer doesn’t like me’, or whatever it may be. It’s not that at all. Critique, constructive criticism is a fantastic opportunity for learning. 

Now there are possibilities and there are times where people are not constructive in their criticism. And I must admit, that’s one of the reasons I probably never want to be a politician, because I have observed and I have seen many, many forms of criticism, which are nothing to do with what you’re saying, but just about actually finding a way to change the narrative.

There is difference there. But this is where we’ve framed this in terms of peer review, and the reason we’ve done that is exactly to hone in on this idea of constructive criticism. That’s what peer review is. 

[00:04:57] Lily: And so then, so in IDEMS, it’s not associated with the university, it’s not kind of academic institution, but I feel that we’ve spoken about Critically Assessed in, well, the context that I know it best, which is academia. So where in IDEMS does it feed in that we want to be or that we, where does this principle feed in? 

[00:05:17] David: Open source software. So, you know, by making the software open source and making our approaches to this open source, it opens us up to criticism. And we do get criticism, you know, why aren’t you doing this? Why aren’t you doing that? We’re a small team. We don’t have time. We’d love to do some of these other things.

But, it does help us to do this. One of our big learnings related to R-Instat, this sort of software on analysing data, there was an academic who does reviews of front end to R, which is what R-Instat is, and when he did it, the feedback and the way it helped us to learn and improve was just immense. It was such a valuable process for us. It actually received that critical review. And how we were able to learn from there to draw out and to improve, that’s what it’s all about. 

Now, again, that still remains relatively academic. So let me go into a less academic process. We’ve been developing apps and chatbots for Parenting for Lifelong Health, related to reducing violence against children through parenting programs. And the first partners who have implemented them have often been quite critical in certain ways about elements of them being too wordy, being too long, and so on. 

Now, it doesn’t matter whether this is a criticism of our work or our partner’s work, it’s a criticism of the product that we together have been putting forward. And it’s helped us go through processes which improve it, which iterate on it, which make it better. And that’s a natural process, and it happens in all sorts of different areas. 

If you think about this from a pure commercial standpoint, this is why organisations want customer feedback, that is a form of critical review. Getting criticism from your customer, you can agree with it or you can disagree with it, you can act on it or not, but you want to hear it because it might be that there’s something you can do, which makes it better. 

[00:07:17] Lily: And there’s always going to be someone that disagrees or, you know, you can’t cater to everyone, but it’s good to know how it’s being perceived, that kind of general, to know if lots of people are saying a certain thing, I suppose. 

[00:07:31] David: And so I guess the key thing there is, as you say, customer feedback, this is a form of critical review, which is rather different, but it’s a form of critical assessment, which we would welcome and which is welcomed across the sort of business sector.

But what I think is really important about maybe our approach and our reflection on this, you know, we’re not saying critical review, we are saying critically assess. And part of that is, there is an element of the assessment being… How do I differentiate between critical review and critical assessment? I guess, part of the difference that I would have in mind on that is this element of value attached to it. 

A critical review, you can have people who critique it for various different reasons. Critical assessment is deeper. It sort of means you’re asked to also provide in some sense of value judgment. And so actually having people going beyond just getting reviews, but actually having people go through the more rigorous process of assessing it, maybe comparing it to others as opposed to just reviewing it. 

I think it’s a deliberate choice to put ourselves under scrutiny. We want to be scrutinised, and that assessment process. So, you know, regulators assess you, customers review you. We want, we’re happy to have regulators assess us. And if we don’t meet the standards, we want to know about it, so we can act on it and we can make it better. 

This is a choice which has been made at the institutional level, we are a community interest company, which means broadly that we have a regulator, but we have no other benefits. Not quite, but broadly that is a component of this. So we embrace that form of assessment, we want others to critique us, to give us the assessment. Are we doing enough in terms of serving our community? That’s part of the choice we’re making. 

So putting ourselves forward for assessment is something we’re keen to do and we’re keen to make this part of our structures. As you know, often that element of review of our work in different ways is important. It is not undermining you, when you recognise now that quite a lot of your work is not undermining you when you put something to me from your review and you know it’s probably going to change. But that’s exactly this sort of critical assessment in a sense. It’s being open to that and it changes, I think, the way you write the first draft, is that right? 

[00:10:09] Lily: Oh yes, yes, yes. Well, yeah, I know that this is a draft, to obviously put lots in it, but to be open to the fact that this is going to change, so don’t fall in love with it. 

[00:10:23] David: Yes, don’t become too attached, don’t make this personal, you want to be able to put it down and say what’s there is there, yes, I can recognise there’s ways that it could be changed, but I want it to be there to be able to go through that process. And I think this leads to efficiencies in a lot of different ways. 

[00:10:40] Lily: Sure, because I’m not working on those perfections. Well, I guess as you said, it’s the first draft, but you need to know that it needs to be open to have other people looking at it, giving their input, that diversity. 

[00:10:51] David: No, having that diversity of perspectives look at something is something else which comes through in our structures, that we do try to have things which sort of, you know, we don’t have formalised structures around this, but we do have informal approaches where we try to make sure there’s multiple people sort of looking at things, going through and assessing them in different ways.

And that critical assessment is central to this, we’re embracing that, we’re really wanting that to be part of our decision making, we’re wanting to get those diverse perspectives before we make an Informed Decision, which is another one of our principles, based on the critical assessment of what we’re doing and what we’re thinking.

And this is also important, related to some of our other principles, like Systems Thinking. That, you know, one of the main reasons I find, recently, I’m really changing things that come from you or from Lucie or from others within the team, tends to be that you’re thinking from a singular perspective, you’re not thinking about the system.

And that’s something where, that’s fine, that’s normal, that’s not your role. You are a specialist within what we do, in certain ways, and, you know, you are evolving into a wider set of skill sets, but it is okay to have that specialist view. In fact, that’s encouraged, that’s important, that’s valuable.

But we also need to be thinking in terms of the systems. And so quite often that requires a different perspective, and it requires people to be open to that. And this I think comes back to this sort of, some of our other principles like Transdisciplinary, where by and large, when you engage in that, I’ve noticed that you often take the position of the others to understand that perspective.

And so over time, you’re actually gaining experience in many other domains where you’re not an expert. You have your expertise, but you then have experience and skills in other domains. That’s really hugely valuable for how we work and what we do. But a lot of that comes from embracing that critical assessment of your work by people in other areas, from other perspectives. And it’s hard, it is really hard. 

[00:13:00] Lily: Yeah. So this kind of critical assessment, or critically assessed, what are the kind of downsides to this as a kind of organisation? 

[00:13:09] David: You’re absolutely right, I mean, it’s really important, again, I’ve said this, I think, in almost every principle episode, you know, a principle is only really valuable if there is a viable alternative, which could be better for other people. So that’s central to what I see in terms of the principles. 

So, Critically Assessed for us, it does, I suppose it really comes down to the fact that it’s a rather academic principle. We’re looking for learning, we’re looking to understand, we’re looking for people to critique us and to question us. But with that we need to be thick skinned, you know, you need to have, you need to be able to take what people are saying and not be led by it too much.

So, if we were just trying to do our thing, then in some sense, you don’t want everybody and anybody’s perspective. You want to be able to sort of have, I would argue, rather than everyone and anyone critically assessing what you’re doing and always having to address that, that’s slow. 

This is related to our Systems Thinking, it’s related to many of these other principles that slow us down, but slow us down and build stability. If you’re wanting to really run, to really go far, go fast, you kind of want to be minimally assessed, would be another sensible principle.

Yeah? That you want to have the minimum assessments to allow you to be as efficient and effective, getting as far as you can, as quickly as you can, without all the processes that slow you down through critical assessment, having to deal with everybody else’s perspectives and so on. You have a vision, you want to run towards it.

You don’t want people holding you back. You don’t want people sort of trying to divert you into a different direction because that’s their opinion. You know, you want to be minimally assessed and that would be absolutely, that would be a wonderful position to be if we were trying to achieve something different.

And so I would argue minimally assessed would be a very good principle for a more focused organisation which wants to just get on and run at this point in time with where they are. 

[00:15:25] Lily: And I suppose, well, it just makes me then think of, there’s that kind of thinking hard aspect that IDEMS really kind of adheres to, and I suppose that critical assessment allows you to do that thinking hard, because you’re getting these other perspectives in, you can’t just run away with an idea, which I wish we could sometimes. 

[00:15:45] David: Exactly, thinking hard is a really, you know, thinking fast, thinking slow. Thinking hard is thinking slow. You know, sometimes it’s so good to just think fast and to just get on and get things done. And so being minimally assessed would be absolutely the right thing for many who are wanting to think fast, they’re wanting to think on their feet, they’re wanting to make that quick progress, they’re wanting to have a process which really goes through in that way. And it has a narrow focus so that you can keep your eye on the prize. 

That’s not us. That’s not how we work. And it’s something where I think the principles help us to sort of frame out what we’re trying to do and why. And I think one of the things which really comes out of them, and this is where, you know, going through all the principles, a lot of them, like critically assessed, very much, when we’re small, as we are still, and we’ve been growing fast, they are burdens for us to carry. But they are burdens which we carry willingly, because as we grow, it brings stability, it brings elements with us, elements of sustainability. 

So if we can grow, bringing those burdens with us, we’re going to grow much better, whatever that means. We’re going to grow in ways which have been critiqued along the way, where we’ve been able to take in and consider other people’s perspectives, where we’ve thought hard about where we’re getting, where we’re going, what we’re doing.

It’s a very different approach to growth to the approaches which have been successful in recent times. Can we succeed with this approach to growth? Time will tell. We’re not afraid of a challenge. You’re not afraid of a challenge. You know, we employ people who aren’t afraid of a challenge. What we’re trying to do is hard.

But if we can succeed, then I believe what we can do is grow different, is actually demonstrate that there are other ways to succeed. And I think that critical assessment is really an important part of that, because if people aren’t assessing us critically, we could lose that. We could actually go off in a direction where we’re growing, maybe we feel we’re growing well, but we’re losing at what we were trying to do. What we’re trying to do is maybe not as important, we’re losing the opportunities for impact in ways that we have if we carry these burdens with us. 

And I really believe that having and taking on board that criticism, addressing it, receiving it positively, and working on it, thinking hard about it, it’s the right approach for us.

[00:18:36] Lily: Excellent. I think, well, obviously I agree and it’s always useful to have these conversations and to hear, hear about these kind of, there’s so much more depth. There’s always so much more depth to these different principles, which I’m sure I’ve said many times that it is something that strikes me.

Is there any kind of final points you wanted to say? 

[00:18:56] David: I guess maybe the final point for me is sort of why did we feel this principle was needed, that it wasn’t covered by being Evidence Based or Informed Decision Making? What is it about this principle which really is different and which leads to a different way of thinking within the company.

And I think the key point there is this aspect towards critique, constructive criticism, which is way beyond academia. And it’s something which is not just about evidence, the evidence that comes out, it’s not just about improving decision making. It is about welcoming critique because we will learn, and welcoming criticism, constructive criticism, as a form of engaging in internal learning.

And that to me is really the thing I want people to take out of this. This is why it’s so important for us to have this principle. 

[00:20:07] Lily: Yeah. Thank you very much, David. That’s been, uh, very interesting and valuable as always. 

[00:20:13] David: Thank you.