215 – The History of Computer Assisted Statistics Textbooks

[00:00:07] Lily: Hello and welcome to the IDEMS podcast. I’m Lily Clements, a Data Scientist, and I’m here with David Stern, a founding director of IDEMS. Hi David.

[00:00:14] David: Hi, Lily. What are we gonna discuss today?

[00:00:17] Lily: I thought today we would discuss CAST and the history of CAST, Computer Assisted Statistics Textbooks.

[00:00:25] David: Yes, it’s a really interesting story in different ways. I first, I suppose, got exposed to CAST in, well, it was developed by a lecturer from New Zealand, Doug Sterling, it was a labour of love, a fantastic project, and it really demonstrated to me, over a decade ago now, the power of thinking about textbooks differently.

This was when I first went to Kenya as a lecturer and started teaching on the Applied Statistics Master’s program, quite early on I started using the CAST textbooks because they were interactive, they had exercises, they were engaging with the students. And one of my students picked them up and started teaching in the Kenyan Institute of Management, and he actually ended up doing his MSc project on looking at the impact on student performance of using CAST exercises and tests as part of the teaching that he and the colleague was doing.

[00:01:37] Lily: Interesting. So how was that studied? Was that studied with an experimental design, how did he look into it?

[00:01:43] David: It was really interesting. It was, I suppose, a quasi-experimental design. He had been teaching the course previously, and so he had data from how his students did previously.

[00:01:54] Lily: I see.

[00:01:55] David: He then taught the course using CAST and CAST exercises. It’s a relatively small class, all but one student engaged with the exercises and all but one student did significantly better on the exam than his previous students.

[00:02:13] Lily: I just need to check, ’cause a statistician here, how many students were there?

[00:02:18] David: Something like six, it was a really small number in that first year, and he had his colleague who was teaching at the same time. And the one student who didn’t engage did similarly to all of her students.

[00:02:31] Lily: Okay.

[00:02:31] David: And then she sort of saw these results and said, I want to use that as well. Then the next semester they both taught using both the exercises and then we put in place the testing system so that all the students had to engage. And both of their students now did 20% better on average on this centralised exam than other centers across the country and then their students had done before.

And this was about 20 students each, so it was still relatively small samples, but it was a really meaningful difference.

[00:03:09] Lily: And out of interest, those 20 students, did any of them not want to engage with it?

[00:03:15] David: They all engaged with it ’cause they had to engage with it because not only did they have it for the exercises, which were optional, but they actually had it as part of their continuous assessment. And so they actually had this sort of summative assessment part as well. And they then performed 20% better on the centralised final exam, which was nothing to do with this.

And so, you know, this is a decade ago now. And this was part of what convinced me of the importance of good formative and summative assessment to enhance learning in these contexts. And what was so impressive about CAST, and CAST is basically dead, but what was so impressive about CAST was that Doug really was thinking ahead. He had the same content in different variants, so if you were interested in biology or management you could get a different version of the same content.

[00:04:17] Lily: Ah.

[00:04:18] David: Which is something that we’ve been talking about recently and is absolutely desirable to have these customised variants. He had that over a decade ago, he had this combination of videos, interactives, text-based, multiple views on the same content, he had these exercises, which were extremely thoroughly designed. So there were multiple variants, multiple context, good randomisation.

I still feel that we’ve taken a lot of the ideas from this, but we’ve not implemented the assessment anywhere near as well as he had done, even though the tools he was using were in many ways ill adapted. He was just building in Java, and that’s why the textbooks aren’t so easily accessible now, you can still download them and use them and run them, but they don’t run in a browser anymore because of the security issues.

But the labour of love that went into actually producing these extremely well thought out textbooks was incredible, and their impact on certain students, in context that I was aware of, on their learning, and he was doing this in New Zealand for many years, but we used it in Kenya in particular, and it totally changed my thinking about what a textbook could look like and should look like.

And so a lot of the work that we’ve got happening now related to textbooks and assessment was inspired by Doug Sterling’s work and CAST and what he built, and trying to get to the stage where that approach to building educational resources is just more accessible, that it’s not a single individual producing this. He tried to get his books so that others could author, but, that wasn’t his strength, his strength was to author the books himself and he authored amazing books.

I guess the thing which is relevant to a discussion we were having previously, is that he had these questions around estimation that led to this understanding. We’ve had episodes before where you’ve mentioned how mean I was to give people these questions to estimate the standard deviation, but this wasn’t me, this was Doug who came up with these questions, who had designed them and I used them and just saw the power of them.

And after having people who had gone through this and who had sort of now got a better understanding, what I realised was that that wasn’t enough because it didn’t enable people to actually use the statistics software to analyse data. And so I discussed this with Doug and he then designed these questions where you’d get the same sort of question that he’d have on an interactive, where you could move things around, but now instead of getting it in interactive, you just got the data. You just download the data and then you’d be asked to analyse it and you’d be asked to put the feedback back in. You’ve designed questions like this since.

This wasn’t my idea, this was Doug’s, I challenged Doug to say, I need students to be able to learn the skills of how to actually work with data and be tested on that and for that to be part of the assignment and the assessment processes. And he designed these questions where you’d download the data and then he would give feedback based on what the response you put in, which is exactly the sort of questions you’ve worked quite a lot on designing.

[00:08:36] Lily: Yes. Yes. And I believe we have previous podcasts in more detail on these sorts of questions.

[00:08:42] David: And really, I want to give credit to Doug for this because he had a system, I challenged him to say, look, I have students who have this problem, they’re learning the concepts, but they’re not then gaining the skills of how to actually do this for themselves. We need a further set of exercises to be able to build those skills as well.

And this is, I think, the heart of where my learning came about, well, these different types of assessment and what skills you can get. When I think about the understanding that people gain from these estimation questions, this is totally different from standard questions where you’d be following a process and have you been able to follow the process correctly, have you been able to calculate this correctly, can you produce a box plot or whatever the visualisation is, which corresponds to the data you are seeing?

These are all things that he had as well, but these estimation questions were totally different because they forced you to actually assimilate the concept and apply it in a way which is a higher order skill. Using those, I found I was able to take students further than I could do without those sorts of questions, but it wasn’t far enough for them to be useful because they then needed to learn how to use software to actually analyse data. And to then add the questions to do that was just a revelation.

Now, of course, Doug actually created those questions just as I stopped teaching at Maseno. And so, although other people have used them in that way, I’ve not had that same experience in the lecture hall doing this with students, as I had with some of the other questions. But we have seen this be used, James, who’s been on a number of different episodes, he’s done this with his students, and a number of others have done similar tasks with theirs. And we’ve now then brought these ideas into these trainings that you’ve been co-designing with others, where this has become sort of central to what we do.

And the fact that you get the data out, it’s now totally software agnostic, we’ve had discussions around that as well. But a lot of this started with the textbooks that Doug was building, which were a decade ago, very future looking, and I still don’t find textbooks as well thought out as the textbooks he created.

[00:11:31] Lily: Wow. That sounds really interesting.

I wanna come back to a comment that you made, you’re talking about how transformative it was and is. But then you said that it’s essentially dead now.

[00:11:43] David: Well, I mean, it died as a number of these really powerful initiatives have died because the technology’s moved on. This was written in Java at a time when Java was what you had in your browsers. Then that caused security risks and so everybody moved to Java Script and Java was no longer supported by browsers, and so now you have to download it as a separate textbook and it still exists, we host this for anyone who wants it and wants to download it, but it is not in active use because the technology it’s built in was redundant.

This is another of the deep learnings that we had, it’s not enough to build something which is really powerful and good, it does matter how the technologies it’s built in, or whether it can be transferred from one technology to another. All of his efforts were impossible to transfer, many of the exercises we’ve been able to translate into STACK, and that’s been a really interesting piece of work. The textbooks themselves, it would be amazing to reproduce these textbooks in PreTeXt and maybe the robots can help with that soon.

[00:12:56] Lily: Well, absolutely, I think that the robots are at a stage where they could help with that, a very interesting task. And you say that we host it and we haven’t said where we host it, I’m sure we’ll put it in the description, but it’s at cast.idems.international, just to throw out there where we do host it, in case anyone wanted to have a look.

[00:13:17] David: And bear in mind, this has not been developed now for a decade basically, it was really, in many ways, I think ahead of its time, but it’s not isolated in this. We think of technology as being this continuous progress, but there’s so many cases where really good pieces of work did not thrive because of business models, because of, in this case, technology changes, which they got caught up in, the number of things which got caught up with technology changes over the years is just huge.

Flash and you know, the number of really good applets for learning, which got lost when Flash stopped being supported, the same with Java. And a lot of them, they weren’t easily translatable. Now one might argue that the things that were really good got rebuilt. That wasn’t my experience.

And I’ll give one other example, the best ever statistics package for kids for learning is TinkerPlots, which was just so well designed, so well conceived, but it got actually lost in a buyout where the company wasn’t interested in the TinkerPlots bit and it just sort of got lost and neglected. But it was by far the best software for what it was as a commercial software. I was always very sad that I couldn’t really use it in my context because it was commercial, and you would’ve thought that that would then protect it to make sure that it evolved with the times, but it didn’t.

There was a team who were developing it who were amazing, who then got, once things got bought out, they got split up, and of course all that expertise was dispersed. It was that core team that was making it amazing. And still, there is nothing to come close to rival it, and yet it’s now basically neglected and lost and marginalised, it’s not widely used as it could have been or should have been.

So that’s another example. And this is within a very small domain area, statistics education. But I’ve given you CAST, which is the best sort of textbooks I’ve seen, which are designed in certain ways, and TinkerPlots the best sort of kids statistics analysis tool I’ve seen. And it’s crazy to me that we don’t have the systems to sort of enable technology, successful technology, to be able to always be competitive. These were outcompeting everything else, but they’re not what is used. Very interesting.

[00:16:14] Lily: Yeah, and I’m sure that that’s a complete other conversation we can go down another day, that kind of outcompeting and how to remain relevant. But I guess coming back to CAST, it sounds very ahead of its time, at its time. And it’s still used somewhat today and still accessible. And I know in courses that we’ve written, we do still kind of link to it at points. But I guess like what lessons can we take today from CAST?

[00:16:39] David: The huge ways it’s influenced my thinking are, one, the importance of assessment. The textbook had these incredible interactives, it had the videos, it had all these other things, none of those really impacted learning. It was the assessments and it was the exercises, they are what made the difference.

This is personal experience, but this was highlighted by my students over a decade ago. So the importance of assessment, be it formative or summative, and he ended up using bits of both, to be able to sort of really get that deeper learning. So if you’re looking at a textbook or if you are looking at developing educational resources, the heart and soul of it is the assessment piece. That’s one of the deep learnings that I took away.

The second really interesting learning related to statistics or data education specifically, was this element that you need to be thinking about it in terms of the practical skills as well as the conceptual understanding and the, if you want, procedural understanding. These are all different concepts, each of which add value to students or practitioners who engage with this, but they’re all needed and they’re all separate.

If you only teach one, you don’t get the others automatically. And so actually being able to sort of recognise the value and the need for these different components is really something I took away from the exercises and the assessment that Doug created.

And I guess the third thing, which is, it may sound silly but it’s deeply ingrained in me now, is that if we want to really put the effort in to building good textbook or educational resources, we want to do so in such a way that it’s independent of how they’re delivered because that delivery mechanism, watching the process with Doug as Java was gradually phased out and this life’s work suddenly, either he needed to go through a huge process to reimagine it as something else, but just the technology being swept out from under him, when he’d chosen Java exactly because it was, when he started, the technology of the future.

And so, recognising how the instabilities of our technological spaces at the moment mean that we should be designing, you know, embedding our educational, pedagogical resources into specific technologies. If we can, they should be living separately and then implemented or delivered through specific technologies so that they can continue to exist beyond the lifespan of a given technology.

[00:19:50] Lily: Yeah. And I guess it’s hard to read what will be the technology that’s still relevant. So you need to build in a way that you are more covered depending on…

[00:20:03] David: Absolutely, and this is one of the reasons I’ve got behind PreTeXt and these tools which are trying to separate out the authoring from the delivery, which will enable you to have that multiple delivery pathways for the same authored content, which means that over time you should be able to maintain relevance more seamlessly.

[00:20:28] Lily: Very interesting. Thank you very much. And so is there any final comments on CAST?

[00:20:33] David: It’s been a long time since I’ve thought about or engaged with CAST. It’s really nice to bring this back up and to recognise the work that Doug Sterling did. I think that it’s amazing, many of the things, many of the places we work, a lot of it is about teamwork, a lot of it is about, you know, what you can do together.

What’s really interesting is CAST was a labour of love really, from one academic. Yes, he engaged with everybody. The reason it was so good was not because he had all the ideas from the beginning, but that he listened when anyone gave feedback, and the next year he’d come back and he’d have implemented the things that he heard at the conference or at the interaction with someone else.

We ended up working with him and influencing him over a decade or so. The number of passing comments that we made where the next year suddenly something new would appear in his book based on that comment, because he took it on board, and he was able to do that. An incredible individual showing what a good academic can do, but also highlighting that, you know, he was fighting an impossible battle because he was stuck within a technology, which was the one that he had chosen, and he had invested so much time into. And as an individual you can’t then adapt to all of these things.

So it’s amazing how important it is that we do work as teams, we work collaboratively, we get people to be working across things. But it’s also, you know, my final reflection is what a good individual can achieve over time is really incredible. And we need to build systems that enable that to happen and support that happening more than I think we have at the moment.

[00:22:32] Lily: Thank you very much, David. It’s been a pleasure.

[00:22:35] David: No, thank you.