212 – Personalised AI-Driven Textbooks

[00:00:07] Michele: Hello everybody. Welcome to the IDEMS podcast. I am Michele Pancera, an Impact Activation Fellow, and I’m here today with David Stern, one of the founding directors of IDEMS. Hello David.

[00:00:21] David: Hi, Michele. I’m looking forward to another discussion.

[00:00:25] Michele: Yeah, I’m also looking forward for it because now we are talking about the big guys. We are talking about Google, and specifically about a recent paper that just came out. And the main point of this conversation is for me to get out some information from you because you’re the expert here. Would you like to introduce what the paper is about?

[00:00:51] David: Sounds good. And this is a paper called Towards an AI augmented textbook, and it’s come out of the Learn LM team at Google about a month ago. And we’re in the process of preparing a sort of response to this paper, which is highlighting the positives, but also highlighting some red flags if this actually got taken to implementation.

So maybe it’s worth just taking a step back and saying we’ve been working for a long time now, and we’ve been discussing on the podcast in different episodes, our interactions with STACK and the idea of actually being able to get feedback to students from automated assessment, and it’s really important to state that STACK as a deterministic tool is really very different from the type of personalisation that they’re talking about.

Both of these are personalisation, but they are different in nature. It’s not that one is better than other another, it’s just they serve different purposes. And so I do want to be clear that the, at the heart of what they’re advocating for is the idea that we can get personalised learning and we can have these multiple views and…

What they’re also proposing is what’s called textbook augmentation. And it’s worth starting with that part of what they’re saying because this is actually something very exciting in many different ways. There have been, there’s been research around this, some of which has been positive, some of which the results haven’t been as good as expected.

But the idea that students learn better when a textbook or the materials are related to their context, that’s broadly agreed. There’s a lot of evidence about the importance of this. Now, of course, that context might be linguistic as well. There were interesting things around language and that wasn’t discussed, but that is of course something which AI is known to be supportive in terms of multilingual content.

So that’s a whole nother discussion we could go into around how an AI augmented textbook could also be multilingual in interesting ways. But the personalisation they’re advocating for, that they’re proposing here is really context personalisation. If you as a student have a particular interest, one of the examples that they use is basketball.

If you are interested in basketball or if you are interested in arts, then the content will be much more interesting, you’ll be more engaged in it if it relates to your interest, and this is something where there has been some very important work on this. There’s some work which in mathematics goes back to a lady called Jo Boaler who found that in the maths classes there was a gender issue because a lot of the mathematics content was gendered towards what the boys in the class were interested more than the girls, and that was leading to gender differences in performance of maths class.

And so this is research, which goes a long way back. It’s been controversial in different ways, but there is a long history of research, which is showing that if you can get students engaged in content they’re interested in, they’re more likely to engage with the topics you’re teaching.

And so that’s really the heart of this first part of personalisation that they’re putting forward. What if our textbooks could be personalised on topics that the students engage with and are interested in? And this is a really great desire and what they are broadly looking into is can a standard piece of content be adapted and augmented related to a specific area of interest.

And the augmentation is interesting as well because they are going from the just a bit of text into some text, some personalised slides, a personalised quiz, and some personalised images. And these, all these different components have been generated by generative AI.

One of the things which I find very interesting about this is that they’re also then looking about if you want, content transformation. And so they have another context where they’re looking about taking some content and transforming it based on your interests. And so this idea of generating and augmenting is one thing, but also then transforming it so that the examples are more relevant to you, that sort of thing is another thing, which is something they were looking at and they have done. The, they included ideas of mind maps, they included ideas of audio and graphics, content, which can also be generated and a more immersive text in different ways, and all of these are things which you can now generate within generative AI, so it’s quite exciting.

The idea that they’re proposing is that we could have, when I say we, we as society could have textbooks, which are totally personalised. To individual interests and also maybe learning approaches. These are the things which could be enhanced because you can have the content which serves the way you like to receive content.

[00:06:36] Michele: The obvious question that I have to ask, given that we often think about STACK, the personalisation in STACK, even without AI, is mostly on the feedback. The first question I have to you is how are they thinking about personalised feedback?

[00:06:56] David: So it’s interesting you say that because their next big section is about what they call practice and assessment. And so they recognise just as we do that at the heart of learning, it is about having formative assessment, having forms of assessment, which have this. Now, admittedly, at the moment, as I understand it, they’re only looking at multiple choice questions, and there has been a lot of work around multiple choice questions showing what they can do, what they can’t do. They are limited, but they can be quite effective in certain contexts.

But they are looking at this and they’re looking about this, and I do quite like their division on this of glow and grow questions. So part of their personalisation. Is not just in the feedback, but it’s in also the nature of the questions they’re wanting to give feedback that actually highlights and targets what students are good at. This is what they call where they glow and where they can improve, where they can grow. They glow and grow questions.

Now it’s really interesting the way that they’ve done this, but I would be arguing that in terms of what you are asking, where we’re thinking about the sort of detailed feedback you get, which helps students on their reasoning with a multiple choice question, there’s only so much of that you can do.

So this is going to be very limited. So what I would argue is in terms of the actual assessment piece, they are not at the forefront of what’s possible. They are just doing simple, multiple choice questions, which I suppose is what one would expect for a group who are mainly, their main interest at this point is what can the AI generate sensibly and multiple choice questions are easy.

So this seems to be, I would argue an area where they could deepen much further. When we talk about assessment, Santiago within our team came up with this five quiz model. Of the idea of your prerequisite quiz, your content quiz, your mastery quiz, your assessment quiz, and your extension quizzes, and the idea that actually any given topic, you could have these whole series of quizzes which would enhance and really enable that student learning.

So this is something where, of course the assessment piece of what they’re doing is, I would argue still relatively simplistic, but they are thinking about it in interesting ways. They do recognise and they highlight the importance of assessment, which I think is good, and they do already have this division between, if you want the, what they call glow and grow, which is I think a good division where you are using assessment for multiple purposes it is both for confidence building as well as, stretching and enabling further growth.

So I think there’s some good thoughts they’ve got here, but it is an area where I would argue their work is still just scratching the surface of what is possible in terms of assessment.

[00:09:55] Michele: The next thing that comes to mind, at least to me, is when you and I talk about AI and implementation, and at this point we, we have had quite a few conversations. We also like to talk about responsible AI and one of the main ideas that, that we mention is having humans in the loop.

[00:10:20] David: Yeah,

[00:10:22] Michele: So I have to ask you how does that idea fit in this case?

[00:10:27] David: It is a really good question, and it’s something where, in this particular paper that they’ve presented, it isn’t absolutely clear how the implementation would take shape. However, the way this seems to be being implemented is what I would consider something which is not responsible. But the paper doesn’t present that, I would say strongly.

So the key question here of humans in the loop, I would argue, is framed better in terms of a thinking of this as a school setting of adults in the room. What do I mean by that? Your children, and your AI… Your AI is, as we often talk, still in its infancy, so having children talking to AI, you don’t have an adult in the room.

This is the thing, and this is something where there is really dangerous precedent where this can go wrong. And so in this context, in terms of implementation, I would feel very strongly we need an adult in the room whenever this interaction is happening. And what that might look like is actually what they’ve done in the paper, which is that they pre-, they use the AI to pre generate resources and they then got the adults actually experts to review that, to make sure it was sound and then they presented it to students and they did some studies with students where they did what they called in some sense a an efficacy study.

But before that efficacy study, there was what they called pedagogical evaluation. And so this is actually really important and I would argue that if we look at the way they’ve done this as a study, they’ve been responsible.

They got educators to do pedagogical evaluations of the resources that were AI generated for feedback. And by and large, the feedback was good. There were certain bits that were better than others. There were some things that were, that I, myself, looking at it picked up, and one of my favorites of a sort of slight critique is that in the original example, they had related to Newton’s third law they had this idea of you stubbing your toe. And the first thought that crosses your mind when you stub your toe, as they said, is that, is probably ouch. That hurts. And then they transformed this into the basket. The AI transformed this into the basketball case.

If you’ve ever dribbled a basketball. And you know it, you push the ball about it comes back to your hand and again it says your first thought is probably, ouch. That hurts. That’s not my first thought when I bounce a basketball. So I would argue this is an area where if this were to be used, humans in the loop could maybe improve that form of customisation where you could pick this out and say, no, that’s not maybe what I think of when I’m dribbling a basketball. And so I think there are improvements that could be made with human reviewers in the process. This is what they are talking about when they have a pedagogical review.

This idea of actually having, pedagogical evaluations and where you actually look at the content, the immersive text, you can improve it. That to me is about having humans in the loop. So having AI generated things which then get reviewed and get improved, and, that is really responsible.

And that could be done at any number of, we can dig into how that could happen in practice, but right now the, I just, the idea that you have humans in the loop as part of the pedagogical evaluation is exactly what they have done and what I would argue is responsible and is needed. And it comes back into then the fact that actually in their, efficacy study as I understand it, the 60 students that interacted with this interacted with material that had been reviewed by experts.

So they didn’t, they weren’t let loose. The AI wasn’t let loose on these. It wasn’t that it was creating, I believe it wasn’t creating resources on the fly for them. I believe it was creating resources that had been expert reviewed, although I’m not a hundred percent sure I’ve understood that correctly. And as I say that, that inclusion of the pedagogical component is part of what I would argue as humans in the loop. And this is, I believe, scalable and I believe something which could be built into these systems long term to make them responsible, but it is needed to make them responsible.

[00:15:14] Michele: I was wondering, since IDEMS has been thinking about textbooks for a while, what would you say are, you’ve been very good at highlighting the parts on which you agree and I think that’s very valuable. What are the differences between our approach and their approach on the textbooks?

[00:15:37] David: No, thanks, that’s a good question. And I, and maybe I will take a quote from the abstract, which highlights, I believe the real difference in our approach. So in their abstract. They say ‘any new material or alternative representation requires arduous human effort so that textbooks cannot be adapted in a scalable manner.’

Okay? This is part of the abstract, and I fundamentally disagree with that statement. You have teachers doing this all the time, all over the world. This is not an arduous human effort. This is a natural human effort, which is needed as part of any of these processes. Where I agree is that the current textbooks have not been leveraging that human effort to be able to integrate it into the textbooks in ways which are effective and so on.

But I fundamentally disagree that it is the arduous human effort we are trying to remove. We, in education, there is an immense human effort available and happening by teachers all over the world, which is so valuable and should not be undervalued or try to be got rid of. It isn’t a question of trying to use AI to replace a teacher or the teaching apparatus or the human effort that goes into education. We should be using AI to enhance it. So that is a fundamental framing issue. I don’t think it puts us totally at odds with what they’re doing and so on, but it is a framing issue, which is really important.

And at the heart of all the approaches that we’re working on and we’re trying to build, it is about how you enhance the teachers, the lecturers, the educators in general, so that they can have better interactions with their students. So you are removing the need for them to have interactions which are not adding value and enabling them to spend more time on the interactions that do add value. And this plays out in a number of different ways.

If you actually get down to implementation, and I don’t have a clear idea of what the Google team would have in mind for implementation. My dystopian fear is that they would hope that this textbook could now mean students don’t need to go to school and they can do all their learning personalised in an individual way without ever having to interact with other humans.

Now, that is the dystopian view. I don’t think that’s what Google are looking for. I don’t, I don’t think anyone wants that as our education system. The education system is really, it has to be around enhanced human interactions for education, for student learning. So that, that’s, if we take some of these ideas and we align them with that the other big question I have is, at the way, at the moment, in the way this is framed, it is about personalisation for individuals. And that’s another thing I’m concerned about. If I think about what I really would like in terms of personalisation, it’s maybe at the class level rather than the individual level, because I want peer to peer interactions.

And if everybody is doing something totally different, which doesn’t relate, it can be disorienting or difficult if there aren’t ways to then bring it back together. So the fact that there might be multiple variants, but that everybody could see the variant that somebody else in the class is looking at, there’s lots of interesting ways to imagine how individuals could have some form of personalisation, but in personalisation, which is really happening at a classroom level.

So that there can be that peer to peer interaction as well as of course the important interaction between teacher and student. So this is where that level of interaction and whether it’s happening at an individual level of personalisation or whether the personalisation is happening at the level of a class or a community, those are interesting questions.

Our work, by and large, is looking more community or the educator level. We are looking at being able to have, for example, textbooks, which take into account a rural context in rural environments. Whereas you might have an urban context in urban environments, in a particular country, in a particular region.

They need to be relevant to that context, but not necessarily individualised to the individual. And this is a personalised to the individual.

Really interesting and difficult questions. I’m not saying that there isn’t value in sometimes having some things personalised to your personal interests, but I am saying that I believe there is strong value in whole classes in interacting on common content, which is personalised to their context because then they can interact with each other.

[00:20:50] Michele: You often say that technology is never the solution. And I cannot agree more. In this case it seems particularly relevant and it would be a grave mistake to think about the technology as the solution. And something that I can add to this conversation be… having been a teacher for several years, I had the exact experience of the basketball example where I had a student completely uninterested in anything school related. And this student was passionate about some other things for example, basketball. So I tailored an exercise for him. It was suggested to the class, but it was actually for him about parabolic motion.

We were studying parabola and two dimensional motions. And I quickly realised that no matter how cool the example was and how tailored for him it was he was clever enough to recognise that it was a trick and it was very much unsufficient to get his attention. What he, I really needed, in my opinion, was some more human attention, some more self-confidence, something that you cannot, you can just not get for any kind of text or AI or anything else.

Then human interaction with someone who actually cares about you. And that’s the point that technologies should be enabling teachers to do their job and their job is interacting with students.

[00:22:38] David: Absolutely, and I love this example because I think it is this surface level adaptation. Smart students will see straight through that because it is only surface level. You’re not actually changing the substance. Actually engaging them in their interest, being interested in what they’re interested in.

That human interaction, that human capability of gaining an interest in that. I’ve seen really excellent teachers do that, genuinely take an interest in their students’ interests, so that they can create that human connection to be able to then bring in some of the other, some of the actual teaching.

But it isn’t just about saying, oh, you like basketball, here’s a basketball example. So that surface level personalisation, I agree, it’s not enough. And that is all that’s being offered here. That deeper element is also, I do come to question. Is it possible that actually an AI could also enable or enhance that?

Maybe, but I don’t know at the moment, and it’s certainly not this, so this, what we’re talking about with these personalised text, textbooks is that surface level adaptation and it can possibly have an impact. However, actually, if we go back and we look. Historically at the textbooks that have been really successful, a lot of them were developed in different ways.

One of the things which I really like is actually there was an amazing UK-based textbook project, which actually integrated textbook maths textbook development with teacher training and actually the teachers were involved in the authoring of the textbooks and it, and they, as well as experts.

And this was something which led to some of the most successful maths textbooks ever. And this comes back to this misconception about us not having access to the human effort. On the contrary. If we could actually make use of the local experts that are teachers and enable them to be involved in the customisation, the tailoring processes, and to be able to have systems which actually drew out from their experiences and their expertise as humans in the loop alongside the AI, I believe this is when, you know, this could become symbiotic and we could get really exciting, interesting results.

So I don’t want to dismiss at all the value that the AI agents… because I think that’s what they’ll have to become. Working with teachers and education experts together to be able to create instead of a textbook, this new thing, which is these very localised, personalised textbooks. I love that idea, and I think it is possible now because of where the AI technology has got to.

To be able to imagine this scenario where we are enabling teachers to understand their class and to tailor the textbook to their class. Maybe even on the fly, not quite on the fly, but something happened last week at school, and then this week it’s actually integrated into your textbook and your textbook experience.

Wow, that would be amazing. But I would see the teacher as working with the AI agent to do that integration in a way which is time effective. Just like you created and you try to do this for your students. This is exactly what I would love to see, as building these AI agents and systems to work with the educators, the teachers, the lecturers, not replace them.

And if we can do that, I think it is exciting. We could be having a totally novel approach to how textbooks could be interacting and working in our education systems. It’d be really exciting. And of course, I actually believe there’s some really interesting opportunities to doing this, starting in low resource environments.

Because that’s where things like the competency based curriculum are coming in. This is where the need for these textbooks is so huge. We have this African partner trying to build these system to build these customisable textbooks in a low resource environment context. So if the team, if we imagine that team in Kenya being supported to say how could you build this next generation of textbooks?

I think what they would come up with would be really exciting and very different from what we’re seeing here. And that’s yeah, that’s an exciting thought.

[00:27:51] Michele: I really like the idea of rethinking textbooks. There are so many opportunities and different ideas, so many good things to keep and and different aspects that can be discussed. It’s exciting. I do think I do agree with you that probably the best thing to focus on is empowering teachers, because then teachers can be the ones who really know what to do. They should be the ones to really know what to do with their classes. So again, I go back to this idea of individual versus community. Once you empower teachers, you are empowering the community or the individual, depending on the choice of the teacher.

I am, I’m really looking forward about what is going to happen in this field.

[00:28:48] David: Can I’m gonna push back on this because although I agree with you, the point is I don’t want to single out teachers. I want to empower teachers. I want to empower individual student learners. I want to empower, a community leader who is tailoring to their environment for all the schools under in their community.

I want to empower a nation to be able to tailor to the nation’s need. I want to empower the international community to be able to get learning, which is across multiple contexts. It’s not about prioritising one and putting them on a pedestal, be this the individual student, be this the teacher, be this, the international experts.

To me, the real opportunity, which was never there before, is we don’t need to choose. If we get the AI agents to support the processes in the right way, we could be supporting all of these levels at the same time. And there could be innovation happening where, yes, it could be an individual gets something which is really tailored to them because it’s valuable, but a teacher is able to tailor it for their class, and the community is able to do this, and all of these different levels are possible within an integrated system.

That’s the dream. The dream is we don’t need to choose who it is. We are prioritising, because in one school you might have a wonderful teacher who’s wanting to put in the effort, and I’m afraid in other schools, the teacher might not be wanting to put in the effort and therefore that effort is coming from somewhere else, and that’s okay.

Building resilient systems is building in multiple failure points. I don’t want a whole set of, and I want to be really clear on this. There are schools in Kenya where no one from the school has ever gone to university. Now, whose fault is that? It’s certainly not the student’s fault because when they entered the school, they knew no one from that school had ever gone to university and so they were condemned in some sense just by entering the school. Is it the teacher’s fault? Almost certainly not. There are some excellent teachers in those institutions, but the environment is difficult.

Is it the school’s fault? Almost certainly not, because it’s in a really difficult environment. It’s not getting the students who are naturally on a […] and it’s not getting the teachers and it’s not got the resources to enable that. It is, these are systemic problems. I want to have resilient infrastructure, digital and other, which can support that really difficult environment as well as your top performing school in California with fantastic teachers and amazing students who have been recruited as the best from all over the world.

I want both of those environments to be using the same technologies which work and which enhances education in all the technologies, in all the contexts. And where over time we could be seeing innovation coming out from anywhere within the system. That’s, that’s the dream. Okay, this is going a bit far, but I do believe that the vision which is portrayed here of a textbook system, which is totally different, which is not about an individual textbook with an author.

I think one of the big problems we’ve got is the fact that textbooks often have single author. The school math project, it was not considered as being something which was authored by an individual because it came from many different authors contributing. That’s the sort of textbooks I think we need in the future, which are quality controlled. They are going through a sensible hierarchical review process in different ways in terms of what’s getting released to different people. But they’re allowing contributions to come from anywhere, and they’re using AI agents to enable those really important contributions to come from anywhere and to be identified and brought in to what would be a very exciting textbook system rather than individual textbooks. 

Anyway, we’ve gone a little off track, but it is something where this, what is possible and the vision, although imperfect that is being proposed, is I believe the essence of a vision, which could be transformative and do something which is really rare: it could reduce inequalities in an education.

This is what’s also really important, actually a textbook system which helps those who are the best off, but helps those who are in challenging situations even more. And that is going to be, this is what we’re aiming for, a technology which serves everyone.

[00:33:35] Michele: This is a very exciting field. I’m happy you pushed back on my framing because yours is definitely better. 

[00:33:43] David: And it’s not that I don’t want to support teachers, but it’s just that it’s everything. We need something which doesn’t single out any individual part of the system. It takes this.

[00:33:55] Michele: Sure. David, do you think we have touched on the points we wanted to cover today? Are we missing something?

[00:34:03] David: I think we’ve touched on a lot of the really powerful things, maybe the one thing that I will just touch on, which we haven’t really mentioned is the way they’ve embedded this sort of, or the way they conducted an efficacy study design.

And I think this is important, and I think this is scalable as well. Let’s remember that it’s not just about having the technology to be able to get the content out to people, but actually you could embed research into an implementation in such a way that it is possible that the, you could get feedback in the system about how people are learning, how people are using it, the really exciting aspect that technology brings by being as part of these systems, is that ability to learn and to study what you’re doing and to embed rigorous studies within this sort of project. And they did, you can critique their study design. It’s a relatively small sample size and so on, but it was a sensible study and this is really important that we should be putting studies in to what we’re doing, and that’s, I think really valuable.

This is something which can be done at scale, which can be done across different contexts and where it should be informing how we develop not just the technologies, but the education systems. The key is it’s about recognising that the technologies are part of the mix, they are not the solution. And so yes, this research, I believe is a valuable step towards getting improved education systems which integrate technology, even if I am slightly nervous about how this technology might be integrated in certain contexts.

[00:35:58] Michele: This has been a very valuable conversation. I thank you very much. I think this is a good point to close it.