252 – AIMS Rwanda 2026: Why Humans are Still Essential for Data Interpretation

[00:00:07] Lily: Hello and welcome to the IDEMS Podcast. I’m Lily Clements, a data scientist, and I’m here with David Stern, a founding director of IDEMS.

Hi, David.

[00:00:14] David: Hi, Lily. What are we discussing today?

[00:00:17] Lily: I thought we’d discuss the AIMS Rwanda course.

[00:00:20] David: Oh, yes, yeah. This was in March 2026. It’s the fourth time this course has been given. I’ve given it twice, and then you gave it with James and then this was the fourth time, and I got to go to Rwanda myself on this occasion. 

[00:00:37] Lily: Yes, very lucky. It’s a lovely country.

[00:00:40] David: It was slightly selfish of me that with John trying to graduate, and so on, I thought, no, this time you don’t get to go. It’s my turn.

[00:00:48] Lily: That’s all right. You managed to get me involved somehow.

[00:00:51] David: In having remote participants, which was exciting.

[00:00:53] Lily: Yes. Yeah. But first let’s discuss the kind of course. So this is the course in AIMS Rwanda doctoral school, where there’s about, how many participants did you have in person? 40 or 50?

[00:01:05] David: Around that. Yes. It must have been at least 50, I think.

[00:01:07] Lily: At least 50 doctoral participants in their first year, I believe.

[00:01:12] David: No, not all of them were part of the doctoral training school. So the doctoral training school was maybe 12, I think, then it was opened up to a wider set of applicants from a whole range of different backgrounds. A few just finishing their MSc, some close to completing their PhDs, some postdocs, there were postdoc facilitators. There was a whole range of people in the room.

[00:01:37] Lily: Oh, nice. I guess that that’s where a course of this style works, with a range of, a range of, we’ll get onto that. But the course, as you’ve said, we’ve taught it before and it’s this kind of idea that we give the students a data set, a simulated data set, and they spend this week working through it, answering various questions.

[00:01:56] David: Well, the course is called Problem Solving in Data Science. When I first taught it, I actually had a whole different course prepared. And my initial task, which I thought people would complete in a day, was to take the simulated data set and I simply asked the question, “what model did I use to simulate this data set?”

Now, I thought, these are the top students from across Africa in data science, some had a statistics background. Getting the underlying model, this is exactly what we do when we analyse data. And it was only that first time I taught it that I realised from a statistics and data science perspective how ill prepared students were for this task.

And so we spent the whole week on it at that time and it was very rich. And one of the students at that time, of course, was John, who was my PhD student, and he was now the facilitator of the course this year, which was very nice, a nice circle for him to complete. He was now seeing the students go through things similar to what he went through four years ago, and seeing them struggle, and seeing that, “oh yeah, no, I’ve learned a lot since”.

But how much he’s moved on and at the beginning he thought, oh, they’re gonna do so much better than we did. And by the end he said, no, they didn’t. It was very interesting, it was just like us. And this was despite, of course, some of them used ChatGPT. We encouraged them, if you use AI to do your data analysis, go ahead. You know, it didn’t help. It didn’t help them get to where they needed to go.

[00:03:42] Lily: Yes. And it’ll be interesting to see, as we hopefully continue to give the course, how that continues to develop, how the robots continue to develop with answering these questions over the next few years, given the changes that have occurred there in the last four years since we first gave the course in 2022, I guess, 2023. 

But no, it’s a great course. We give them the simulated data and we say to them, “find a model”. And these are the top students across Africa, but this isn’t just an Africa problem at all, this is something which, when given to me, it took me a while, and guidance.

[00:04:15] David: And you did exactly the same thing as AI, or maybe I should say the AI did the exact same thing as you, it didn’t think about interactions.

So of all the models that the AI used and suggested in the visualisations, none of them actually had any interactions and so none of them were of any use to understanding this data set.

[00:04:36] Lily: Interesting.

[00:04:37] David: Really interesting.

[00:04:38] Lily: That is interesting, yes. 

[00:04:40] David: I was a little bit disappointed in the end because the course was squished into four days, normally it’s all five days. And so it was a little bit more intensive over four days, I missed that extra day. I felt at the end we were rushed in getting to conclusions.

And some people got it, but not as many. One extra day would’ve made a big difference, I think, for quite a lot of people in terms of really the big picture sinking in and them understanding. Some people, it was amazing, it was eye-opening that they got it. But I think more people would’ve got it with that one extra day. That was part of my feedback to the organisers, please give me five days next time.

[00:05:19] Lily: And then alongside that, as you say, we had the online component, so each day you would give a seminar, as it were, or a webinar to the room of people in Rwanda, but it could also work online, it was over Zoom and we had the online component of different participants joining in there and discussing in the chat. But also online, we had discussion boards and things to help guide their analysis and try and get them to a sensible point with the data.

[00:05:45] David: And it’s so interesting that one of the very bright students on day one used ChatGPT to do the analysis, and then for basically three days was sitting there smugly, not really believing that he hadn’t got the answers even though I was coming and showing and he wasn’t getting the right answers. And then on the fourth day, it suddenly clicked and he suddenly understood why the results he had weren’t right. Using AI, he was falling into the traps as it was designed in the questions, rather than AI having the right results. And it was eye opening to him in a way where other people had used rather complicated methods, random forests, and advanced AI methods in other ways, but they got exactly the same results, which weren’t answering the questions.

And so I loved the fact that the deep insights at the beginning were coming out of the people using spreadsheets. And so it was the people using spreadsheets who were making progress. And those who were using AI in different ways, they made no progress and they didn’t believe that the people using spreadsheets had anything useful to contribute to what they were doing because they were very satisfied in what they were doing.

And so the fact that they were getting the wrong results and the people using spreadsheets were making progress and they weren’t, didn’t bother them at the beginning. But come day four, suddenly the spreadsheet guys now couldn’t make further progress because you needed models, you needed to go deeper.

And then they could actually see, well these are the models that are actually needed, and that’s so far from what I got using AI. And to actually do that without understanding, I could never have got there. And so I needed that understanding that came alongside the modelling. And then the modelling was needed, but the results of the modelling suddenly came out in a way that actually made sense to them.

And I loved this with the diverse set of participants, as we had. Some were real experts and used to using complicated models of all sorts, and others were very much stuck in Excel, that was their level at the moment. They were comfortable in using a spreadsheet, but going beyond that, the coding scared them and they weren’t really comfortable with that.

They are the ones who made progress, whereas the supposed experts who were using code and using AI to write code and all these other things and very impressive, were not making progress. And that, come day four, it was clear why. Oh, it was beautiful. And it just highlights how it’s very easy if you are comfortable with code and you are comfortable using AI to write code, that you can do things which are really impressive and people are scared of, but they could be wrong and they could be missing out on the things which are really important. It doesn’t mean you are actually drawing out insights.

[00:08:47] Lily: This is what I really enjoy about this course, when it was first used four years ago, ChatGPT was just coming around there to some kind of very low level usage, it was being talked about, it was being used a bit, but not in the same way as it is now.

And I really like that this course ages alongside technologies. It still holds up, different tools and technologies coming out can still prove just different teachings and lessons. So the way that the course is done and the way that you do the course and the way that the course is designed, actually, I guess ’cause it’s software agnostic, but those same lessons and learnings can come out four years later and, okay, only four years, but a lot has happened in that four years in terms of generative AI and in terms of being able to analyse your data using gen AI.

[00:09:45] David: Yeah, and the fact that people who wanted to use gen AI to try and analyse it were encouraged to do so, and didn’t make progress, they actually spent days sitting there smugly saying, well, I used gen AI to do it, and nothing happened, and they didn’t make progress, ’cause they didn’t understand enough about what had come out.

And so it really, it was really humbling in that sort of sense to some of them who then actually got it at the end, and really saw, “wow, okay, so this is what I need to do”. This is why humans in the loop are still so important, that it’s not enough. Doing the analysis is not the hard part, it’s interpreting it, it’s understanding it, it’s actually drawing information out and doing the right analysis.

This is what was so beautiful because none of the AI agents did the right analysis. And that’s not surprising. I mean, this is designed to be complicated in different ways. But yeah, it was beautiful. 

[00:10:40] Lily: Even if they did the right analysis, whatever that is, I imagine that there would still be, that level of interpretation would still be needed. I guess what I’m trying to say is that I’m interested to see how as gen AI develops, how different lessons come out of this course.

[00:10:56] David: Exactly. And the point is, as you say, actually, let’s say, a few years down the line, gen AI is given this sort of data and is able to actually do an analysis, which brings out the results which are the correct results. Well, actually, one of the really interesting things is, what are those correct results?

Because the way they are simulated is not as a single simulation, it’s actually as multiple separate simulations, which are then all together in one data set. So would it display this as one model, which has sort of if statements and lots of interactions – huge layers of interactions – or would it be multiple separate models? So even that, it’s not clear what the best way to represent this is.

When I took people through it eventually, and we had to rush it on the fourth day so that people could do it, we broke it down and we got the individual models. And once you’ve got the individual models, they’re relatively simple. But because there were multiple models which you put together, if you don’t split it up as I know how to do, because I created the simulation, actually it’s really hard. And this is one of the things which is so powerful about this experience, because the reality is: if you have two different varieties of the same crop and you end up doing something, should those varieties be the same model or different models?

These are actually really difficult questions. And so, at what point should you actually be doing a different model versus building a more complex single model? There are advantages and disadvantages to either of these. And this was where it was beautiful, where those who were using spreadsheets, they got to the stage where they could ask any individual question, but without the model, they didn’t really know what the right level of complexity was, they had a lot of the information that they could have used to build the model. They could answer the questions in ways that were effective, but they couldn’t actually get the models. And because of that, and that’s where the modelling then would’ve come in, or did come in, the people who were then doing the models didn’t need anything more than just linear models with ANOVA tables, and so on.

What was interesting is that there were people who did random trees, and they actually got the simple model back out, but they didn’t get anywhere near the complexity, they didn’t have the interaction terms in their model. Even with the random trees, what you’re doing random trees on matters, and they weren’t doing it on the right thing. And so it was wonderful. It was great.

[00:13:34] Lily: And then to add to that, okay, you get the correct model out – and we can say the “correct” model in this case because it’s simulated data – you get this kind of correct model out and it says in it there’s a gender effect, there’s a region effect, and then you need that kind of interpretation of “why is one gender better than another in one region and another gender’s better than the other in another region?”

[00:13:56] David: And when we had people who found that, then they’d ask me and I’d say, oh, there’s a story behind this. So this is actually something which has been observed in this part of the world where in some villages, in some cultures, the men take the fields that are near the village and the women are pushed out to the outskirts, and in other places it’s the other way round where the really fertile land around the village is given to the women, whereas the men are using the land which is further afield.

These are actually genuine stories that we know from working in these regions. These effects, every effect mentioned or that exist in the simulated data corresponds to an experience I have either seen, heard, discussed with people in the West African region. So there’s always a story behind every effect, which has real interpretation and meaning to it.

[00:14:57] Lily: Yeah, and that would be something where, maybe gen AI will one day be able to do that if we started writing about this online and it trolled the internet and found this interpretation like that. But I guess, hopefully, it shows to the individuals the importance of humans in the loop, which I can’t see going anywhere.

[00:15:18] David: It’s not just the importance of humans in the loop for this, it’s the fact that real data – and this isn’t real data, this is simulated data, but it’s realistic data, ’cause it’s based on real experiences with data – that for realistic and real data, the interpretation is not just about the numbers, it’s about drawing meaning out of the numbers.

And that’s such a powerful process to put in a doctorate course like this, and for some participants was so eye-opening that they could not, with all their skills, they couldn’t draw out from the data that they had the right questions even to ask. And when they did, there were some things which were wonderful. My favourite one is still some of the problems that come with larger data sets, and when that finally hit home for a few people, you could see that they were so proud at using the larger data set and that this came with its own issues. It was a beautiful learning moment, which will stay with those participants forever. It is clear, you can see it marking some of them, some of them slightly traumatically, but mostly in a very positive way.

[00:16:37] Lily: Hopefully, yes, yes. Well, you need to learn somehow. And it’s much better to learn that way than it actually being in the real world and having real issues. 

[00:16:47] David: Yeah. I mean, we’ve had this with our team in different cases on rare projects, and it’s always the case. You have these real projects and there’s real world consequences to data decisions. There was a decision about data, which costed I think it was sort of 50,000 pounds or more to rectify. And one of our team members was just mortified that this sort of, this lack of attention to detail and a small detail related to the data led to this, the need for this additional study costing an extra 50,000 pounds.

[00:17:19] Lily: If this is the one I’m thinking of, it’s not even. Their attention to detail is fantastic, it was just one teeny tiny momentary lapse.

[00:17:29] David: And it wasn’t even really a lapse, there was a need in terms of the analysis, which hadn’t been made clear when the system was being designed. And so there was a detail that then meant that when the analysis happened, certain participants couldn’t be differentiated in the right way. And so therefore new data was needed to be collected, which had that differentiating characteristic.

I mean, this is happening all the time where there are actual research questions where after the work has been done, people then try and analyse the data and they find they don’t have the data they need for the research questions they have. It’s so common. And yeah, it can be expensive.

[00:18:06] Lily: I’m enjoying this at the moment because my partner’s just finishing off his undergraduate thesis. It is kind of the first ever research project and looking at it now, oh, maybe I should have asked different questions. It’s kind of in sports, PE, nothing to do with statistics, that is kind of like first ever try, that sort of thing. It’s a lot of fun.

[00:18:26] David: Yeah, absolutely. And it is, it’s one of the deep learnings that asking the right question is so important, the difference between interesting results and results which nobody really cares about.

Anyway, to come back to the course..

[00:18:41] Lily: Yes.

[00:18:42] David: I think it’s the only course I’ve taught in the last few years. I suppose I’ve given workshops and done a few other bits of teaching, but actually academic teaching, this doctoral training school is the only course I’ve taught in recent years, and it’s a joy to do, it is very nerve wracking. I still get nervous giving this course because you never know who’s in the room, what they’ll do, how you’ll have to react to it, how you’ll respond.

But it was a joy, a wonderful group of students, a wonderful group of participants who engaged, and a number of them, a few who might have been traumatised, but a number who were really grateful and visibly taking away the sort of lessons that this is designed to give, where they’ll never look at data in quite the same way again, and they will give it more respect. Actually, yeah, it is great.

[00:19:33] Lily: Yeah. Excellent. Well, I don’t know what else you could get out of a kind of four or five day course. 

[00:19:39] David: Yeah. It should have been five days. Then they even more out of it with the fifth day, even if it was the same number of hours, just spread out over that extra day. You could see them, you know, every day some people would make different cognitive leaps and, yeah, I felt we were rushed on that last day. 

[00:20:02] Lily: Well maybe if you had five, you’d say you wanted six. 

[00:20:04] David: That is probably true. But no, I think I’ve been satisfied with five before.

[00:20:09] Lily: Good to know.

[00:20:11] David: Anyway, this has been good.

[00:20:12] Lily: Yes. Yeah, thank you very much. It’s great to chat about it. It was a good course to give, from my end online, and to see the developments and listen in on.

[00:20:21] David: I should just recognise the work you did in putting together the resources so that the way the course was given this year was better than it’s ever been before. And so we’re really making progress, and that is your work. So thank you for that.

[00:20:34] Lily: Oh, thank you. Well, no, these are just, again, resources that have come from lots of different work that we’re doing and slowly, we’re building up this nice library of bits where we can pick that bit that’s relevant. Anyway, that’s, I’m sure, a whole other podcast.

Thank you very much.

[00:20:48] David: Thank you.