240 – Integrating Deep Student Assessment into Open Statistics Textbooks

[00:00:06] Lily: Hello and welcome to the IDEMS podcast. I’m Lily Clements, a data scientist, and I’m here with David Stern, a founding director of IDEMS.

Hi, David.

[00:00:14] David: Hi, Lily. We talked recently about your work on the PreTeXt textbooks. I think we finished that one by saying in a few months’ time we might have a follow up episode, but this has come a bit sooner than expected.

[00:00:26] Lily: Absolutely, it has come a lot sooner than expected, but there’s so much that we can touch on here. And one thing in particular is the integration of STACK exercises into textbooks.

[00:00:37] David: Yeah, and I should be clear, of course, STACK is a tool that we’re really behind, we believe in, I’m on the International Advisory Board, but it’s not that it is STACK itself, it is mastery assessment, it is strong electronic assessment with automated feedback, where the system has complexity behind it, so it’s not just multiple choice questions, it’s deep assessment, student assessment.

[00:01:05] Lily: Yes. And I know that we did a podcast at one point on CAST, where we talked about where a lot of the ideas for these assessments have come from, and we’ve done various podcasts on different, more specific questions that we’ve done with STACK.

[00:01:18] David: There are plenty of episodes out there talking about STACK, CAST, which is Computer Assisted Statistics Textbooks, this passion project from Doug Stirling, which is a wonderful piece of work which still inspires us. And I suppose, that’s really good to bring back here because I think the thing which is really relevant for the discussion we had last time, but related to the assessment piece that comes out of the learnings from CAST, is that in CAST, he already had a form of multiple variants. His multiple variants were a “full version”, which included interactive applets, a “brief version”, which was just the key points, and a “video version” where you could actually watch the video of it. And separate from that, he had these electronic interactive exercises. And so he already had for the same content, these multiple variants within his textbooks, which are very different to the multiple variants that we discussed last time.

But you can see that parallel, that different students will want to engage with the same material in different ways, either in the brief form, in the interactive form, in the video form, or through electronic assessment. And his multiple variants was, I have to say, a little bit clunky, I mean, it was really well implemented for when it was implemented over a decade ago, but it never really satisfied these multiple variants in the way that I think we are conceiving it now. He had a lot of the ideas for it. We discussed that we were inspired by his work, but I think we are actually executing this potentially in a different way.

But the piece which is critical – and this is other discussions and thinkings – is the fact that the learning does not come from passive consumption of textbooks. In all of these situations it is the doing, you learn by doing. And that’s where the integration of these interactive exercises is something which is central.

And that’s an additional component. And again, I think we might have discussed in an episode about the idea that there’s different ways we want people to engage in these interactive exercises.

[00:03:38] Lily: Yes, the five quiz model, we have done a podcast on that, I’m pretty certain.

[00:03:42] David: We’ve done a podcast on the five quiz model. I believe we’ve also done a podcast on the three different types of statistical questions, where you are actually getting people to engage with the concept, or with the skills, or with the application. We’ve talked a lot about these things in the past, but all of this now is able to come in.

And so what’s really exciting is that, actually now, if we look at, again, these multiple variants of the textbooks, and we say, okay, wait a second, these are all open textbooks, and many of them are just taking the same topics and presenting them and tackling them differently. But we could embed the same interactive assessment across all of them. So, these interactive learning experiences, we could embed them not in one textbook, but across textbooks. And this is really exciting.

[00:04:39] Lily: Well, to even add, or just to point out, some of the textbooks already have exercises, and so I’ve been flicking through the exercises when putting them into PreTeXt. And there’s some exercises for which we already have a STACK version of that exercise, and that means that you can do this exercise, but on a mastery level. So the student doesn’t only get to do it once, they can keep doing it and keep doing it.

And that’s very exciting that there’s these parallels there in the exercises, and then also there’s some exercises that they have, that we don’t have. Okay, well, maybe we could have an exercise like that, maybe we could build one of them in STACK.

[00:05:22] David: Sorry, let me be clear. This is a really interesting discussion I’ve been having with US educators recently, where many US top universities are getting what they call “teaching focused faculty” – I apologise to Dave Kong and others if I’m getting that term wrong. These are actually people within the universities who are being employed by maths and statistics departments to focus on the teaching of the large classes, because often you’re doing a lot of service teaching.

And one of the things that I discussed with them is: what about “assessment focused faculty”? This is actually what we’re needing, and what I love within what you are describing here, is that actually, at the moment, as a textbook author, you need to do both the textbook part, but also the assessment part. Whereas actually what we’re saying is that we can separate out the textbook author specialists from the assessment specialists, and we can have that separation in a really positive way that people who are really expert at student learning through assessment can now be contributing across textbooks rather than just within. This is really exciting.

[00:06:42] Lily: And we’ve spoken before about this kind of mini library, this little library that we have of our STACK questions. I say little library, I think that there’s over a hundred in there.

[00:06:53] David: There’s over a hundred statistics ones, and there’s thousands of questions in our open library. So no, there’s thousands and thousands of STACK questions in the open library.

[00:07:03] Lily: Yeah, a few over a hundred or many over a hundred, sorry. But, I guess what you are saying here, or what I’m hearing here, is something I’ve not thought about before, but that’s, you know, “oh, I’ll just go to the library and I’ll pick out the questions”. And, “okay, well, maybe now I want a question on this, so I’ll suggest it or make it my own way”.

Then we can start to actually have this bank of questions being just “you pick and choose what works for you in your textbook”, and add and evolve, we don’t want it just to stagnate.

[00:07:33] David: Well, more than that. I think you have said “in your textbook”, but what I love is that actually this could be in “your variant of the textbook”. This is the thing, that it might be that this is for the textbook as a whole, or it might be that this is just for you in your variant in your class, that there’s something you are coming across which you are wanting students to engage with in a different way.

And that ability to have variant authors, textbook authors, assessment authors, and for each to play a collaborative role in this mosaic, if you want, that’s being created, this is really what we hope would become really powerful, enabling this collaboration and enabling open to really become fair and care. I don’t need to go into the details of what fair and care means, but this is a lot of people thinking about how open actually changes the way we build systems, we work with systems, the way we manage data.

And I think we are getting somewhere in terms of actually envisaging an implementation, which is genuinely different.

[00:08:36] Lily: That’s very exciting. That’s a very exciting thought and not one that I’ve thought of. Like we said in our other podcast on the textbooks, I’m in phase one, which is the translating, we translate the textbooks to PreTeXt first. I’m thinking about the other phases which are yet to be kind of formally defined.

[00:08:54] David: Absolutely. And we’re now getting into phases four, five, and six, but anyway, there is a concrete, long-term process here at play, and yeah, it is exciting and I’m looking forward to it.

[00:09:08] Lily: No, me too. Me too.

[00:09:10] David: The other thing which I think is worth digging into, because I think we still have a bit of time, I don’t wanna make the episode too long, but once we started taking on this assessment piece as being something separate, you talked about the five quiz model, I want to talk about what this might mean in terms of, again, different publishing possibilities.

At the moment, there’s an interesting relationship between, let’s say lecture notes, textbooks, courses on a learning management system, and I really love envisaging that the open community could offer something which was really joined up so that I, as a lecturer or a teacher, no let’s stick with lecturer for the moment – lecturers have more independence in different ways – as a lecturer who’s going to teach my course, it’s not just that I could choose a textbook, or I could have somebody else’s lecture notes that I can adapt, or that I could take an existing course on a learning management system. It’s somehow that I have not only this package of things which are all integrated nicely so that I can integrate it into my learning management system, I can have the associated textbook, I can have the associated equivalent of lecture notes, which relate to the textbook and may be more tailored. But I have this integrated system which I can then make my own, because, of course, these things are open to some degree, the lecture notes can really become my variant of the textbook, they can be something which I really pick and choose from different textbooks, my course could refer to two or three multiple textbooks. I could be taking part of one textbook here, part of another one there, I could be choosing to have, even for my own course, for my own students, maybe multiple variants for those who want to specifically focus on R and for those who want to be software agnostic.

And I, as a lecturer, have the power to do all this and the ability to do all this, but not only do I have that, it’s easy. Because we know lectures are overworked, they’re overwhelmed. They’re able to do this, they have the intellectual ability, but, at the moment, to do all this, to take things and make it your own, it’s a lot of work. People spend a whole semester preparing for their next semester’s class. And in the context that we work, in low resource environments, they don’t have that time, and so they need something which can be put together more quickly and efficiently.

And being able to meet people where they are on that combination of flexibility and structured support, so that in their context they can build what works for them, that’s the vision. That is something where commercial companies can offer this sort of experience at the moment with their textbooks, they can offer a really supportive system because they have the money coming in from the textbook sales. They can then be going in and offering the lecturers who use their commercial textbooks an experience which, at the moment, the open source community cannot offer.

But there’s no reason why we cannot have that joined up thinking for open textbooks. To be able to offer an equivalent version to the commercial textbook company, there’s gonna have to be money involved. So we’re gonna have to have business models that sort of marry in with these textbooks, and that might mean, we always discuss the licenses, that might mean that the non-commercial textbooks are excluded. Or it might mean that the non-commercial textbooks now have commercial licenses, which means that the authors actually get money coming back in this. We know these are systems which are possible.

You know, how the business models get built on top of this so that the lecturer experience can be world class, so that your lecturers can get an experience equivalent to or better than the experience that the current top publishers can provide, that’s where we hope these pieces will come together. Who offers those services? It might be IDEMS, but my guess is it’ll be other organisations who work with us, as we’re collaborative by nature. We’re never going to become a textbook company. But offering textbook services might become part of the portfolio that we have. We might not, it might be that there’s partners who take that on, we don’t know.

But the fact that this can tie together into making the experience for individual lecturers of the open source ecosystems superior to the experience of the current publishers, that’s the dream. And of course, part of this might be that some of the current publishers actually change their business models to be more focused around the open textbook scenes. That would be exciting as well. So there’s all sorts of other possibilities that come up around this.

But, at the heart of what I think this is doing – and I think the assessment piece, which is where we started, is key to this, and you mentioned the five quiz model – if that really worked, I think we could not only enhance student learning, but also enhance lecturer experience and lecturer quality of life, because of the bundle of services that could come together to support them in this process.

[00:15:01] Lily: Yeah. Very interesting. And that’s phase, that’s a much later phase, I assume.

[00:15:05] David: Well, yeah, these are, of course, these are all, at the moment, they’re pipe dreams, but there’s so many people working on components of this, I really think what we’re doing is just bringing together these different pieces of the puzzle. Nothing we’re doing is totally revolutionary, it’s just an evolution of what’s already out there. But once you’ve evolved sufficiently, you are totally differentiated.

[00:15:35] Lily: Yes, well, and I know from what I’ve seen – obviously I’ve gone through quite a few textbooks now – there are places where certain chapters have been.. “I’ll take that chapter from you”. I’ve seen that in quite a few places. There’s definitely a market, and there’s definitely something that’s going on, but to make that easy would be very nice.

[00:15:55] David: And it might change how academic systems perceive textbook authoring. One of the things that I’ve mentioned in a previous episode – I think with Lucie – is that actually working in African contexts on this as well, there’s hope that certain types of work on variants of textbooks gets recognized and actually considered towards promotion. That’s maybe more achievable in some of these lower resource environments, where those systems are either not as well established, as in Kenya, or well established as in a lot of the West African systems, but they have the potential to value localisation in a way which it may not be valued as much elsewhere.

In the West African region, I am really certain that if we could enable lecturers to localise textbooks to their contexts, this is something that would be immensely valued because at the moment, they do feel alienated that they don’t have the textbooks which serve them, which are designed for them.

And that’s another piece of the work that we’re doing, which is part of the 20th anniversary of the West African community of practice. But this ties into that as well, of creating that opportunity for people to take these international textbooks and turn them into things which are suitable for them, relevant for them. That’s also, of course, part of my underlying motivation. And that’s an environment where I could imagine, quite quickly, that localisation of an international textbook to a particular context could be recognised in the structures that lead towards promotion.

[00:17:58] Lily: Wow, yeah, that’s incredibly exciting.

[00:18:01] David: Yeah. Anyway, there’s so many exciting aspects to this, so thank you. And again, I’ve ended up talking to you a lot, so sorry for that. I’m just loving the momentum around this at the moment.

[00:18:11] Lily: Yes. Yeah, no, me too. And there’s a lot of big ideas and a lot of ideas that seem very achievable right now, which is very exciting.

[00:18:19] David: A lot of these big ideas go back 15 years. The difference is what is achievable right now. And that’s why it’s really of the moment and really exciting. And again, your work really just motoring on, getting this expanding set of textbooks, is what’s so motivating.

[00:18:38] Lily: It’s the robots.

[00:18:39] David: It is the robots, but it’s great.

[00:18:41] Lily: Talking to the robots.

[00:18:43] David: It’s you and the robots doing an incredible job. And this is, yeah, brilliant.

[00:18:48] Lily: Excellent. Well, thank you very much.

[00:18:50] David: Thanks.