08. Human Capital and the Future of AI

[00:00:07] David: Hi, and welcome to the IDEMS Podcast. I’m David Stern, a founding director of IDEMS, and it’s my pleasure to be here today again with Kate Fleming, another director. Kate, are we still going on the Empires of AI?

[00:00:20] Kate: We are, we are. And this really builds on our last episode, our last conversation, where we were talking about when you have data owned in particular places, you don’t have to just be aggregating everything and collecting it and extracting it, there are different ways to think about data, about models, the way they work together, all these different pieces.

But a lot of what that conversation brought up to me is the issue of what should the role of humans be, not the future of work, not how they use AI, but in really building AI and building the useful AI that we are talking about. 

And we can keep it focused on social impact this time. We talked more about the enterprise context only because it’s shaping what a lot of AI looks like right now. But when you’re looking at systems that are more distributed, when there are different owners of different things, there’s a whole system that the more we understand that system, the better that is.

So I am trying to think about, and get your help on this, what are all the different things that humans need to be doing, ways that humans need to be involved, different kinds of expertise that are really building toward this vision of AI that we see would be very useful in solving hard problems.

[00:01:38] David: I think it’s really important to start with what’s actually happening because there is the whole narrative that AI is just magic and so on, which neglects the fact that there’s a large amount of human work that is going into this, but it’s often far removed from the users, in fact, almost disjoint. You have your user communities for whom it is almost just magic, and you have then the communities that are doing the work, which are very disjoint.

And the reason I think that’s so important to give us current framing is because I think that’s the thing that should change, that we should be thinking of this as much smaller cycles where the users and the actual people employed to code up, to tag are not disjoint.

[00:02:34] Kate: So the current system is you’ve got your AI company and you’ve got that team, which is computer scientists, modellers, you could tell me exactly what the profile of a company is, but then all of that tagging grunt work, what is commonly known as reinforcement learning through human feedback, that process of giving, assigning data, meaning, that is often done by very low paid offshore teams who are not connected in any way to the problem, the application. They’ve been given a very mundane task, tag all the lampposts in this image or whatever it is, and then that’s their job.

So, is there stuff in that system as well that’s really important in the current system that I should be calling out?

[00:03:23] David: Well, maybe just simply the fact.. I want to come back to this Amazon Fresh example. You know, this is all the artificial intelligence doing the work, and in the Amazon Fresh example, the idea was that would be true. But in the meantime, while we’re training the artificial intelligence, we are getting people to actually watch the videos pretty much live, tag what people are actually putting into their baskets, and then that video in some sense with a human watching it is the information which is then passed in. And that this didn’t actually lead to the learning that was needed.

So what I think is really important is that in a lot of the cases, there is not a clear definition of exactly what it is that the systems are doing versus what human effort is happening behind. That’s not to say that it is always smoke and mirrors as it was the case with the Amazon Fresh, but often it’s not clear what the human effort is actually doing in terms of the reinforcement learning.

And the reason that’s, in my mind, a big problem for the context we work in is that actually, if we don’t know what the human reinforcement learning is doing, we don’t really know what’s safe and what’s not – where I use “safe” to say where we know what the systems are set up to do well and what they are doing because of the internal algorithms, where surprising things could happen.

And so, really, if we want to take this back to social impact, we need to be much more careful. And so we need to put in place structures for reinforcement learning that are clear on what is and what isn’t safe, whatever “safe” means.

[00:05:26] Kate: So this actually relates to tagging, I guess, to the example that I gave of lamppost, you’ve given someone this very narrow task, you have one sense of what the value of a lamppost is, so you’ve just said tag them and then the value of that will just be decided by some external system.

Whereas if you really want that to be valuable, you might want data about where these are positioned in the community, does anyone actually walk here? There are other things that could be useful to be tagged alongside just that basic data that would start to give meaning, that would be really important for developing a system that was trying to do something in particular related to lampposts.

[00:06:09] David: Well, if we stick with the lamppost example. 

[00:06:12] Kate: It was a terrible one, sorry.

[00:06:14] David: Which is a difficult one, but there is a concrete use case I can think of which is automatic cars, self-driving cars, you know. Tagging lampposts and being able to identify lampposts is important because you don’t want the car to drive into one. Certainly, a desired feature of a self-driving car is not to hit lampposts. That would be an important feature of actually being able to recognise that, well, we need to identify these.

But now you also probably don’t want the car to drive into trees and maybe even bicycles with people, wheeling them across. And the first case of a road death caused by a self-driving car was exactly that scenario where the car just went straight through somebody walking a bicycle across the road because they didn’t recognise it as something that needed action.

Now, I’m not saying that you should therefore go through and tag people wheeling bicycles across the road as the tagging, what I’m saying is that’s an interesting case where a surprising feature happened. Nobody would have expected that to be a scenario which wasn’t recognised as a reason to stop. And there have been a really rather large number and a growing amount of data around self-driving cars of these anomalies where suddenly a self-driving car accelerates in a way that is not expected or in a way that is counterintuitive and the opposite of what you would’ve expected in that road situation.

And this is what we want to really be able to identify and avoid. Now, self-driving cars are a really hard example, and you set me up for that with a lamppost. But I think it’s a good example which highlights this issue of do you have control over the things that you want to be happening, when you want it to be happening, and what is unknown? Because you will always have the unknown, you can’t get away from that. The nature of the mathematics behind these AI systems is that they are designed to fill in the gaps. How they fill in the gaps, a lot of that depends on how we set up the human reinforcement learning.

The things that we then are able to control are the things where there was good human reinforcement learning. I’m gonna go way back to early AI systems in the 90s where they started replacing manual sorting of posts with automatic sorting of posts by reading postcodes. And when they started doing this, they were not as successful as manual sorters. But now they’re so much more successful than manual sorters could ever be.

And one of the reasons for that is, well, it’s actually quite easy to have good human reinforcement learning if when the system gets it wrong, the letter ends up in the wrong place. This is something which therefore you can then do a manual correction and the system learns because it has really good human reinforcement learning when things go wrong. And when things go wrong, it’s relatively low stakes because that just means a letter is delayed. So this is why that process, which started a long time ago, now has led to a situation where one would never want to go back to human sorting because one could never do it as well as the AI systems can do it.

It’s an easy example, but it does relate to other things as well. And it could, in the future, relate to self-driving cars. I could imagine a future where self-driving cars are so safe compared to human-driven cars that one could never allow human driven cars again because that is how you keep the road safe for everyone. We are not there, we are nowhere near there at this point in time, but I could imagine that in a future.

And that comes back to then, well, what are the guarantees that we could actually put in place to get to that place? And I don’t know, and I don’t think anyone would really know for something like self-driving cars.

So let’s come back from that example to the sort of examples we are facing within social impact spaces. Let’s say education, dealing with somebody’s maths homework for example. This is an area where reinforcement learning, what we want to do and what we don’t want to happen is actually better defined. And the systems that are being proposed right now where everything is given to the AI really scare me.

Whereas it is very easy to see how AI attached to assessment systems like STACK, like WeBWork, these other systems which have a deterministic assessment core, they could now achieve what I think we want to achieve from these assessment systems and from AI learning, with elements of guarantees where we have humans in the loop in the right way, creating the structures. So we could have AI authoring questions, which are then reviewed before they’re given to students. We can have that working being tailored to students, so different students get different questions based on what they’re doing and where they are and what they’re wanting and what their interests are and so on.

That personalization is all possible through AI, but where it could be centred on these actually quality assured material, and so on. So putting in those elements where the quality that the AI is given can be guaranteed, that’s something where I think there’s a lot of work to be done there because our current AI systems are not built like that, and we are not building them to that. But it’s something which is exciting and possible.

[00:12:47] Kate: It is. And what I hear too is there is a big difference between what you can build and what is actually useful and impactful. So I think the classroom example is such a good one. And we see so many examples of just like, we’re gonna launch all this AI stuff, they’re gonna be AI tutors, it’s just gonna be direct to students and they won’t even need teachers anymore.

It’s like, well, 1) is that what’s needed, 2) does it have impact? All these different issues where I think part of the way that we think about the building side of it is you need so much more involvement, particularly on the impact side, of the people who are actually on the ground, who understand how things work. That’s everyone from the actual teachers to researchers. Who is really understanding what’s needed, seeing where technology is adding value, where it’s a distraction, or even undermining education?

There is so much research coming out right now about how disastrous tech in the classroom has been, mainly because this narrative of every student needs a computer and this is the way to go has been so pushed when it’s really not the reality of how students learn and it’s not what they need.

[00:14:01] David: Just to come in on that specific point, there’s two layers to this. There’s the layer that actually technology in the classroom hasn’t led to the improvements that people expected, which is actually quite established now. But the very recent things are more exciting than this. They’re demonstrating that, actually, access to AI to help students in their learning is having a negative impact on what they are learning.

This is really recent, literally the latest article I read on this came out a month or so ago. And it is something where there’s an extra layer that the AI is giving on top of the questions around the technology enabled learning, where a lot of the technology enabled learning is very easy for people to dispute because having access to technology doesn’t mean it’s used well in the classroom.

And so you could easily say, well, that’s because they didn’t use it well. And that’s a hard thing to argue against. Whereas the simple fact that there is now growing evidence that having access to AI to help with the understanding of what you are learning is leading to lower understanding of what you are learning, that’s a really important result, which I think is interestingly actionable.

[00:15:18] Kate: And so how is that, what is that? Because we’re talking about, well, what does that mean for how humans are involved in building, what does that look like to change the course of AI in the classroom?

[00:15:29] David: Well, I think the key thing is, and it comes back to what AI people are using. The AI that they’re using are generalist AIs by and large. And so they take whatever they’ve got and they stick it into something which is a general AI system. And I think what this is indicating is if you’re gonna have AI in the classroom, you almost certainly want specialist AIs, that will do the job better, which can then be trained to have a pedagogically sound approach to the interaction with the students and with the learners, because almost certainly the way that the AI is giving feedback, if it’s leading to lower comprehension of the topics, it’s related to how that information is then presented and how the students then interact with it.

And so having an AI agent which is more specialised is the obvious response to that, where you can then try to actually study, well, what are the nature of the interactions which lead to improvements in student learning rather than distraction from student learning?

[00:16:42] Kate: So I think that points out that a big variable is we need – in the same way that a drug doesn’t get launched to the market without clinical trials, without tests, there’s a high bar to launch a new pharmaceutical product – it sounds like that should be the bar for technology in the classroom and technology, I think, we would argue, in other impact spaces. But there really needs to be research trials, evidence of impact and rigorous trials.

We see this, a lot of this actually came up in conversation with a mathematics educator who was enlisted by, I think it was an AI startup, to help on something, and she gave all of these recommendations, this is really problematic, this doesn’t work. And then they kind of cherry picked her words and she sees on their website that they say, oh, in collaboration with researchers, we developed this, and it’s this twisting through this public relations, communications, massaging of truth, to launch something out in the world that has the patina of impact and credibility, but is entirely lacking any real rigorous evidence. Is that the bar that we need to be working toward? 

[00:17:57] David: The slight problem with that bar is that it’s a high bar to jump and it’s expensive, it is really expensive to do that. And therefore, at the moment, who has the money to do that and to try and make that happen? Well, it is probably exactly the tools that are not actually trying to, their moneymaking tools rather than impact focus tools because the public dollar to do that is not really available.

It would be wonderful if that was the way technology was being built. But it’s difficult to imagine how we get there from where we are right now.

[00:18:32] Kate: And it’s the classic example of where regulation favours the incumbents because if you’re a small startup, you don’t have the money, you just can’t navigate that system. And so it just reinforces that the same people are introducing the same kinds of innovation, or whatever it is, but it just doesn’t enable any sort of meaningful impact innovation from the margins.

[00:18:55] David: Yeah. This is exactly where these are really complex problems. I certainly don’t disagree that I would love to have real evidence-focused approaches, but at the same time the barriers that it creates, they’re nontrivial to navigate. It stifles innovation.

How do we get the balance between those two? That’s a really hard problem and a hard question. One of the things that of course is possible and has been pushed towards this is, well, you can try to balance between those by sort of moving people towards open source, because then people can build from one another.

But that doesn’t work unless you actually have the incentives for people to build together rather than in isolation. A lot of this then comes back down to the incentives and the right incentives. Open source is a great idea, implementation is really challenging.

[00:19:49] Kate: You make me think of the Global Parenting Initiative example. So the example out of Oxford, which is when they were starting ParentApp – and I wasn’t even a part of IDEMS when this started, so this is just my recounting, correct me if I am not remembering it entirely right – one lane of their work was very rigorous, very research, we move slowly, we don’t watch anything without evidence of impact.

And another lane was like, well, how are we gonna figure out what people need? Unless we’re getting it out there, we’re starting to get feedback, a more traditional tech view, but with researchers very involved in analysing that feedback, assessing, is this doing more good than harm? And then what ultimately happened is that those two lanes converged because they both brought really important insights and validation. Either on its own would have not been as successful or not enough in the same way.

So yeah, I guess that’s it, the idea that there’s always a one size fits all approach is just not the reality.

[00:20:49] David: It’s really interesting looking at that now with, well, it’s been over six years, six and a half years of process on this. As you say, these multiple lanes that were at some point very distinct, have merged back together, but are also distinct again in different ways. It is really interesting how important it was that multi-lane approach.

The research rigour, which was expensive and slow, but really effective, and the adaptive learning, which came from the multiple small implementations of ParentText. This was actually the app versus the chatbot, that the app was slow and steady and the chatbot was fast and meeting people where they are.

But both streams have been needed to be able to navigate. This comes back to a broader approach, which I don’t think is recognised often enough, that actually we need this diversity. There were different teams working on them. At one point, the only connection they had was through us because we had a connection between the people working on the two projects, and I myself was involved in both.

But other than that, the two teams were almost totally disjoint because as research teams, they were focusing on their intervention, their approaches. And yet it was only together that they’ve actually been able to navigate and get to where they are now. It’s a really good example where this is complicated and we need these collaborative approaches, and different approaches to be impactful.

[00:22:28] Kate: I think it’s the different approaches, it’s empowering different people. I mean, what we see so much of right now is all of the money is going into these very small, these few companies. And it is so driving the direction of what innovation looks like in a way that is actually not. I think it has diminishing returns of innovation.

[00:22:49] David: It is stifling innovation.

[00:22:51] Kate: It’s stifling innovation. 

[00:22:52] David: And this is what Karen’s book makes so clear. Even if you take AI, AI has been reduced to this single narrative of more data, more compute, which is absolutely wrong. But so much money has been sucked into that, it’s left everything else out in the wilderness where there could be real societal value to be had.

[00:23:16] Kate: So to bring us back to the theme of this, who should be building? It’s that so many different people need to be building. And so to break up that throwing endless money, this is where we’ll talk in some future episode about policy, about different things like, how do you incentivize that?

But you have got to have such a more diverse population building what AI looks like, not regulating it, not having opinions on it, but really building it for us to start to have different models, different innovations, things that are really meaningful for impact, and have just other kinds of AI innovation.

[00:23:56] David: But this is what’s so exciting and this is where the real opportunity lies. Because, really, if we forget about the AI of the AI empires and we think about the future of what AI should be, what it needs more than anything else is human capital. It needs people to be working on it, to be building things, to be actually innovating, working in different ways.

And human capital, well, this is the one thing which if the current AI systems have their way, there’s gonna be a huge amount of unemployment. They’re firing a lot of their staff, which means that human capital is the one thing which is increasingly available. And so actually building these alternatives using human capital is exactly what can be possible and can then outcompete, because the most valuable thing to build good AI systems for the future is going to be this human capital.

[00:24:53] Kate: And we just need some realignment of actual capital for that to start to flourish.

[00:24:58] David: Yeah, that is true. That is true. That’s not my strong point in the same way.

[00:25:02] Kate: Thank you, David. This was a really interesting conversation.

[00:25:06] David: I look forward to the next few.