255 – Authoring STACK Questions as a Service

[00:00:07] Santiago: Hi, and welcome to the IDEMS podcast. I am Santiago Borio, an impact activation fellow, and I’m here with David Stern, one of the founding directors of IDEMS. 

Hi, David. 

[00:00:16] David: Hi, Santiago. Looking forward to discussing, I guess, what has been a big chunk of your recent work?

[00:00:23] Santiago: Well, it’s not just a big chunk of my recent work, it’s what we were hoping to get to years and years ago. I joined IDEMS through STACK authoring as an impact project for Maseno University, particularly for Mike Obiero, who has been in several episodes and we’ve mentioned in many others.

That was back in 2019, and I joined freelance and then I joined full-time doing other things, but STACK authoring was probably the only constant I had throughout my time in IDEMS. We had small contracts before on and off, but we are finally getting to a position where we got a couple of bigger contracts.

We talked in one of our earliest episodes about being fundamentally profitable, which is bringing money into the company to pay for my salary, in a very simplistic way, being cost recovery at least. And I haven’t been since I joined, probably until now. I’m not sure if I’m quite there, but through these contracts, if I’m not there, I’m very close.

[00:01:48] David: Yeah. And it’s not just you, Michele and Georg are also involved in STACK authoring. So it is something where it is now a solid stream of our work, it is something which is likely to continue to increase and expand as other institutions hopefully take us up on this, because it is different to the standard model that most institutions are taking. And I’d like to highlight some of what we’re learning about those differences.

[00:02:27] Santiago: Go on.

[00:02:28] David: In many institutions, they recruit, employ or assign people who almost become specialist STACK authors. A lot of the STACK professionals network is people who have that as part of what they’re doing, a big chunk of what their work is is to be involved in STACKifying courses.

[00:02:53] Santiago: Yeah, as one of the original members of the professionals network, the description to become a member is having STACK authoring as a substantial part of your work.

[00:03:05] David: Yes. 

[00:03:06] Santiago: There is an active network of over 20 people, 30 maybe, who fit the description, and they’re all based in specific institutions except for IDEMS.

[00:03:23] David: Yeah, absolutely, and that’s specific academic institutions, except for IDEMS. And what I find is so interesting with this is the value that we have or that we bring by working across institutions. As part of our discussions is the nature of what is the licensing of the questions that are created at a given contract? These are things where I feel the added value of that cross-institution work is. I’m not saying there isn’t also added value in being within institutions, but I do believe that we’re seeing more and more that the value of this cross-institution work is immense.

[00:04:10] Santiago: And some institutions, one of the ones that we are working with in Italy, they do have people not fully employed to author STACK questions, but as part of their doctorate, they’re very much involved in authoring questions. 

[00:04:31] David: And they’re part of the STACK professionals network. 

[00:04:34] Santiago: And they are a part of the network, yes, but we’re finding that the experience that we have from working across institutions, both with contracts and as impact work in African universities, we’ve managed to develop a level of expertise where the quality of the questions that we can create tends to be better than the in-house people in some instances. I must admit in some instances not, but in this particular instance yes. Because it’s not their main focus and they haven’t been doing it for as long, we are in a position where we can offer a better service than the in-house people.

[00:05:23] David: And more than that, because we have a team of people, we can almost always add value. Even in the context where they are authoring really exceptional questions, we can almost always add value because we can bring that diversity of skills to bear from across our team.

Yourself, Michele, Georg, myself, and others in our team have different skills which relate to this. A simple one related to the University of Edinburgh, where we haven’t had a contract but we have had discussions, is that they have a lot of experience, of course, because this is where Chris Sangwin is, the key STACK developer, and Edinburgh has really embraced STACK in wonderful ways in the maths department in particular. But statistics is not what Chris Sangwin has thought deeply about, and it is not the specialist area of any of the team of the STACK authors in Edinburgh, and so it is just a lecturer who has been working on that. There’s a lot of added value we can bring there and we could bring in those discussions, because we have people who have been thinking across the statistics domain as well.

And I think this is something where, if we can build out our role, this will only increase the body of people that we have who can work across these domains and offer this support. And I believe that this is not going to replace the STACK professionals within institutions , but really to compliment them and bring added value, which is what we’re aiming for.

[00:07:03] Santiago: And in one of the institutions that we’re working with, part of our contract is to train local people, but our hypothesis is that even after training people, even if they become really good authors, we can still add value because of the structures of the institution, where those potential specialists won’t have the time to author more than for their courses.

So we have a contract that is until the end of next year and part of that is training, but we see the potential of expanding or having a new contract for other courses, for other lecturers, for other professors who are starting to embrace STACK because they see the value of it in the use in the current course where it’s being used.

[00:08:03] David: Absolutely. And I do want to come back to the fact that at the heart of this is this idea that really what we’re developing is an assessment specialist team, a team of people who are really specialists at sort of mathematics assessment, electronic assessment particularly using STACK, but also beyond. 

This is something that doesn’t generally fit within standard academic structures. This is something which all higher institutions of learning need, but it doesn’t make sense for them to replicate that in every institution. But it might make sense for there to be a group like ours that works across these institutions and that supports those who are interested in driving this forward within their institutions.

And that’s very much what I’m enjoying about the current contracts, as it seems to be setting us up into that role where we can be leveraging the small grants that are available and accessible to be able to build up our team and with it, the universities we’re supporting.

[00:09:18] Santiago: And correct me if I’m wrong, the grants that you’re speaking about are grants that universities potentially have access to, and they subcontract us as part of the implementing element. 

[00:09:28] David: We have not yet received direct grants for this, it would be amazing if we did. But we have been benefited from our partners applying to internal grants and subcontracting us to deliver on those internal grants.

[00:09:46] Santiago: Yes, and, I think it’s important to note, you mentioned the team that we’re developing, it’s interesting that two of us are secondary school teachers with just a degree in mathematics, and the third one has a PhD in mathematics, but his specialty really lies on the algebra side of things, that sort of area and not calculus. And one of the areas where we are contracted to work on is very much calculus, partial differential equations.

And we work very closely with the lecturers who are the subject specialists so that we can collaborate with them and have them define what they want us to do. And I have to say, AI helps a lot, but we don’t need to be subject specialists in order to offer good questions as long as the specifications of questions are done well by the subject experts.

And that is amazing, I’m currently authoring questions where I don’t understand the mathematics, but because I work very closely with the lecturers, I manage to author high quality questions that really meet their needs.

[00:11:15] David: And what’s interesting is, this is exactly where STACK was not just conceived for high school level mathematics. It was conceived for university level mathematics, even high levels of university and it can add value. Good assessment can add value right all the way through the university curriculum.

[00:11:34] Santiago: Yes. And in recent years, teaching and learning has become much more important in universities or much more prominent in universities than it was when I was at university, for example. 

[00:11:49] David: But, I think broadly, what you’re articulating, which is correct, is that we have a time right now, especially with the discussions around AI and education and what this is meaning, where there is interest and need to think carefully about assessment. And we are positioned to be doing so in a way which, in both high and low resource environments, we have a depth of experience which is emerging, is really there.

What I’m really enjoying about this is that it’s not different. There’s many, many cases where you have to do things very differently in low resource environments. We’re in some of the highest resource environments in the world, and we’re in some of the lowest resource environments in the world, and for STACK as an intervention at university level mathematics, there’s differences in the nature of how it’s used, but the role it’s playing is surprisingly similar. 

[00:12:48] Santiago: I have to agree and disagree with what you just said, because when we were doing impact investment by authoring questions in low resource environments, the amount of questions that we had to do and the lack of funding meant that we had to make compromises in quality. One of the things that I’m really enjoying now is that with this funded work, I can really spend the time to make each question extremely high quality with all the requirements that a question would normally have, give purposeful feedback in all different sort of ways, which we couldn’t afford to do in some instances for the low resource environments.

And if we grow the team, if we grow the contracts, we might be able to start putting that same amount of effort into impact work, and continue training the local teams to be able to author these higher quality questions locally as well.

[00:13:55] David: I see what’s interesting there is that what’s needed is the same in both. What we were able to do is different because of the amount of time, the human resources we can put in, so it’s really nice to have the high resource ones, but it’s so interesting that actually what is needed is surprisingly similar.

[00:14:14] Santiago: Oh, yes. That I agree with, I cannot question that in any shape or form. 

[00:14:18] David: And not obvious, because the role it plays is so different. Very simply, in Kenya, where one lecturer has over a thousand students the main role it plays is that it is enabling that lecturer to have the students receive feedback in a way which is accessible, achievable, regular, all of these different things. So it’s actually making a high quality education possible in a very challenging environment.

In the high resource environment, there’s tutorial assistance, there’s all sorts of people who help facilitate the course. So the manpower and the student interaction to give feedback is not the limiting factor, but what’s interesting is that what the students value is still the mastery, it’s the same mastery which is valued in both of these contexts because it’s a low stakes piece of feedback. And so, it’s very interesting that what is valued is very similar.

[00:15:19] Santiago: And let me clarify. By mastery, we mean attempting questions multiple times, getting feedback on attempts and being able to repeat different variants of the same question in order to gain the learning through the feedback that is given, through repetition.

[00:15:36] David: Yeah.

[00:15:37] Santiago: But, yes, I agree that there are different contexts and there’s different levels of support, but the nature of the work, the authoring of the questions, does not change, that’s the constant. Because the high quality question will serve the purpose in both contexts and the level of complexity of the question and the level of depth in the question, if we were able to invest the same amount of time, I don’t think it would change that much.

[00:16:12] David: But, I do want to come back to the fact that, a priori, there’s many different types of questions that we could be making. We could be making mastery, formative assessment questions, we could be making more summative assessment questions, we could be making questions which are stimulating and trying to get people to go beyond the material, extension questions, we could be making questions which are actually trying to teach content, the textbook in the quiz, as has been mentioned in other contexts.

There’s many different types of questions, and it wasn’t obvious to me a priori that the questions that would be useful across these different environments would have the same nature. Now, of course, I believe that, in an ideal scenario, you will have a mixture of all these different types of questions. But even if you don’t have that, what is coming out is that the formative assessment questions are the ones which are most valued and the best starting point in almost every context.

[00:17:13] Santiago: Yeah, and the feedback that we give through question authoring is the key component there. Quality feedback of different types at different stages is what really makes a STACK question for mastery valuable.

[00:17:35] David: Well, yes, and one of the things we’ve learned from the Kenyan context is, even when you can’t have really high quality feedback, having feedback over no feedback makes a huge difference. We had examples from Masinde Muliro University where a student redid a quiz 18 times, and they got it eventually. Now, of course, if the feedback had been better, my hope is that it shouldn’t have taken 18 times to get to that, but that is a question of, yes, we can improve the materials and that can happen over time.

[00:18:13] Santiago: Yes. And even Chris Sangwin, for his courses, the questions he authors, he says it takes three iterations to get them right, and it’s the analysis of the questions when they’re attempted for the first time that informs the type of feedback that he implements in the questions, and then a third iteration makes them more complete, let’s say.

[00:18:39] David: Exactly. And this is what we’ve never had the bandwidth to do, to get those really high quality resources in the impact context for the Kenyan universities and beyond. But that would be amazing if we could actually get to that. And maybe we can get to that through working with these higher resource institutions, and having these questions that cut across and that are open.

[00:19:03] Santiago: Yeah, and one final point is that working across universities – and usually we are lucky enough to have questions with open or almost open licenses – allows us to bring in materials from one institution to another. It’s not just the experience and expertise that we have, it’s the resources that we create that might apply to multiple contexts.

[00:19:36] David: And I want to be very careful on that because there is no problem with us working on question banks or universities’ proprietary systems, where we don’t share openly. But almost all of our partners have been happy to share openly. This is a very academic trait that actually, there are some universities, and even Edinburgh itself has had a long-term discussion of how open their question banks should be or not.

These are difficult questions for institutions, and we can fit in with whatever they’re doing, but what I’ve loved is the fact that most of the institutions we’re working with, they buy into the fact that they will benefit from other people using and iterating on the questions that are authored for them. So the open questions is not just a generosity thing, it’s also better for the institution because of the way we’re building them and the way that we’re able to do this.

[00:20:36] Santiago: Indeed it is. And I work in an academic collaboration with a group of lecturers and researchers, on what we’re hoping will become part of the STACK library. And one of the problems that we have is that one of the universities has a policy that the questions are not open. And we’ve had discussions with Chris Sangwin about what is the minimal unit of exchange, and he’s happy to share quizzes and questions, but not necessarily courses, because that is what makes them unique.

So there are subtleties in there, but in general, working across institutions has not just benefited us, but has also benefited the individual institutions.

[00:21:27] David: And I think that’s a good place to end this discussion because that’s what we really want. You know, benefiting us is not at the heart of it, it is benefiting the institutions that we work with, which is why we do this work and why we’re all so passionate about it. Improving education is in all of our hearts.

[00:21:47] Santiago: It is. Well, David, thank you very much. It’s been an interesting discussion.

[00:21:52] David: Thank you.