267 – The Forces Shaping AI

[00:00:07] David: Hi, and welcome to the IDEMS Podcast. I’m David Stern, a founding director of IDEMS, and it’s my great pleasure to be here again with Kate. Hi, Kate.

[00:00:15] Kate: Hey, David.

[00:00:17] David: I’m looking forward to another discussion in our series on the Empire of AI. 

[00:00:24] Kate: Yes, are we beating the dead horse here? I don’t think so, there’s actually so much to unpack and talk about that I think that, yeah, each subject feels meaty.

So, today I want to talk about really what are the forces that are shaping the current model of AI. We’ve come out of talking about the fact that none of this is inevitable, there are other ways of doing things, things might look quite different. So obviously there’s stuff that’s happening out there that is creating a sense of inevitability, what are those forces?

[00:00:55] David: My favourite thing that I read very recently – and I’m gonna paraphrase this – was from Sam Altman himself, where he acknowledged it might be tempting to try and do more with less, but no, we should resist that temptation and really shoot for the stars and go for this artificial general intelligence. And that’s the narrative which is driving everything we know, we need to do more with more and if we do more with more, then we’re going to eventually solve problems.

And I think that narrative has gained so much traction partly because there’s a whole economy behind it now. Doing more with more has created lots of jobs for people to build data centres, for infrastructure to be built and for us to say this is what we need to do and I can put a plan of action in place and I can say what the return on investment will be in some sense.

And so it fits in beautifully to the economic model of investing more and more in this narrow hope that suddenly all problems in the world will be solved by a magical artificial general intelligence. Sorry, I’ve got a little bit political there.

[00:02:17] Kate: No, no, no, and I will jump in with you. I think the idea that this is a utopian dream, if I’m really cynical, is a fig leaf on what is fundamentally Silicon Valley as usual, which I think is what you are identifying there if I look at the realities and practices of AI. So there are things about the way Silicon Valley works that I don’t think everyone necessarily is attuned to, or really thinks about the consequences of.

One is: if you are funding something as an investor, if you’re looking for exponential returns, often the way you’re looking to get to those is by creating monopoly power within a market. So if you have a lot of competitors, if the value capture is distributed, you’re not getting exponential returns. You’re gonna have a solid business maybe, but it’s not the same.

There’s this idea, you want to introduce something that’s 10 times more or better – whatever the word you wanna use – than what someone else has, so they can’t catch up with you. You wanna make it impossible for competitors to keep up with you, you are going after that monopoly power. Your metrics of success are very tied to getting profit, that is what success looks like.

[00:03:37] David: No, no, to getting users, profit is secondary. Because once you have the users, you can milk them. And so, it’s not actually about the profit. Many of the big startups aren’t profitable.

[00:03:51] Kate: Well, this is about the path to monopoly power. And again, I don’t feel like I’m the best person to talk through this process, but the idea, the way it works often is that you come in the market, you aggressively undersell your competitors because you have all this fake money to throw at something, so you can pretend that now a taxi ride only costs like $5 to get to the airport. And then people are, “this is great!”, and they start using it, they become dependent.

There are other things that have to work too, so you’re doing policy capture, you’re doing various things. Then you’re the only game in town. And then you can do whatever you want. And actually, I will say, Karen gets into this very deeply, she uses an Amazon example of how Amazon just took down its competitors to show that this is like a well established playbook in Silicon Valley.

And so there’s this false.. it’s not real competition. You’ve used a totally artificial money, economic system, to create some illusion that this is a functioning business. Then you, at some point, have to start creating the returns for investors. So there’s an end to that patience, you have to have the game plan where now that we’ve captured them, now we really put the screws to them. Now you start making it really expensive. You created dependencies, you have surge pricing, all these different things that then people are stuck in.

And I will say part of it too is that you are putting the screws to both consumers and your business customers. So you are playing all sides to make sure that everyone.. Yeah, you might start with delivering something that’s really great for one segment, and whatever that system is, you’re always looking for monopoly power, returns for investors, profits are the goal down the road, but you’re right that they’re not in the short term.

[00:05:45] David: And it is exactly the fact that they’re not in the short term, which means they can be uncompetitive in terms of actually returning profit, whereas their competitors can’t afford that because they don’t have the investment capital to throw at it. And that’s sort of part of the equation, which is so difficult.

But let’s put this back now to the AI, and what’s actually happening in this question of who actually has the data centres which are being used for the calculations for the compute. If you need more data and more compute, then this is not in the clouds, this is in the data centres. And so, therefore, who has the data centres?

[00:06:30] Kate: Well, I would also say what it started with was: who has the compute, who has the processing power? And that was the origin point of 10 times more, and that was the origin of OpenAI. If we can just get all this computing power, it puts us ahead of our competition. So if that’s your model of gaining your monopoly, that’s gonna drive a very particular vision.

And it’s why the Sam Altman quote you started with, he doesn’t wanna let go of that vision because it’s what justifies, it’s what lets him make the case that they’re superior because they’ve got all the processing power to do this insane vision of AI that’s just more and more and more. Where if you’re going less and less and less…

[00:07:11] David: Actually, this could be democratised, you could actually have a democratised AI where the compute is not the driving force. In fact, the existing compute is probably more than we need as a society. And yet so much is being invested in more compute.

So these are the interesting questions. What’s driving it now is these statements that more compute, more data is what’s going to drive the next innovation. Whereas the most likely next innovation is doing the same, or maybe even more, with less.

[00:07:50] Kate: And I would say this also takes us back to this seminal – I guess that’s the word – book by Shoshana Zuboff’s, The Age of Surveillance Capitalism. She creates this foundational narrative of data and data extraction and monetizing data, and that if you control that data collection apparatus, and increasingly put it into our lives, that you can monetize that.

And AI is on this trajectory. So surveillance capitalism to this model of AI is a logical progression because it’s like you are capturing the data, you are then capturing the compute, no one else is, and if this is your model of what AI is based on, no one else could possibly compete with this.

[00:08:37] David: Yeah, exactly.

The thing which is missing of course in that..  I loved the way it was phrased, and I don’t have the exact quote in the way Sam Altman recognised, you know, some people might “get tempted”.. But yes, it’s so obvious. If we actually think about what’s needed to solve the problems – and I’m not talking about social problems here, business problems, the businesses that are using or trying to use AI and finding it doesn’t work for them – well, if only they actually focused on the problems that they wanted to solve, these are probably solvable with very little compute.

And so, actually, this is the thing. I liked your fig leaf example. The emperor has no clothes here, this is actually what’s happening in the empire of AI. They have no clothes, there’s nothing really on them at the moment. There’s so much which is just based on this single narrative that our stock prices are going through the roof, if you keep investing in us, then your investment will go through the roof. And all this is based on nothing except a dream that artificial general intelligence will magically solve everything. 

[00:09:48] Kate: I would say this is an issue that’s driving what AI looks like, learned helplessness in society. Tech has been so aggressive and like, oh, you can’t understand tech, this is too complicated, that you can’t, it can’t even be regulated. This is just you’re against progress if you try to do anything here.

And so I think that sort of power dynamic that’s been set up, they’re riding that wave and I think consumers and citizens are looking at it and are increasingly sceptical, like, hmm.

[00:10:25] David: And what’s of course so ironic about this is that one of my favourite papers on small language models is from an Nvidia researcher. These are the people inside the beast who are the ones publishing to say, look, this is where the actual trends are gonna go. Beautiful article advocating and explaining how actually, from someone inside seeing what’s happening, of course we can do the same with less. This is exactly where the research should be going.

You’ve got evidence growing. There was another article which came out recently on the arXiv, and the arXiv does seem to be a good place for some of these to get it to the firsthand before it goes through the publication processes, which are quite slow.

But actually from an educational standpoint, students who are using AI to solve their problems are therefore not interacting with the material world. So what we’re actually doing, if we are using these general intelligence models, is that we are creating educational outputs that are worse. So our next generation is going to be worse trained than the last generation.

That’s a really big dangerous point for society. And there’s no reason why this needs to be the case. This is simply because the effort is going into these general intelligence systems rather than what it should be going into, which is actually how do we build these artificial intelligence tutors, which mean that people learn more.

There’s no way an additional technology should lead to us understanding less. It should be changing what we learn and how we understand, but not leading to less understanding, less involvement, less engagement. But that’s what’s happening and there’s evidence on this.

[00:12:11] Kate: And this makes me think about something you brought up in an earlier episode, which is research capture. So a lot of what has happened is that you should be having this very active, ongoing debate in a research community with all these different researchers pushing in different directions. What you have instead is that so much funding and so many hires have been made where that research might be coming from and creating momentum and getting out there into the public sphere. That’s all been co-opted, so you don’t really hear, other than these small little bubbling points, it doesn’t have the same momentum, it doesn’t have the same pushback.

[00:12:56] David: And I think what’s interesting is that – I think historically this was a US phenomena, but it’s become a global phenomena – that the research in Europe didn’t use to be, I’m going to use the word “faddy”. It didn’t use to have these sorts of fads, which came and went. You were respected as a researcher if you’d spent your whole career going deep into something and you were the world expert and you knew about that narrow niche topic, but you were the expert in it.

It didn’t shift or change. But as fundings have been more and more reliant on research grants – and this happened in the US before it happened elsewhere – it’s one of the negative aspects of how much research funding was available, that, well, the topics that got funded were the topics that were trending. And therefore, as the funding changed, the trends changed. And as the trends changed, the funding changed with them.

And so now it’s got to the stage where actually the funding trends are what really matters more than the actual underlying research. And therefore, when that funding trend is actually responding to the societal trend on what’s happening, you can get these cycles.

Now, of course, these could be virtuous cycles. As you put forward this approach of having something where the funding changes quickly, you can get these virtuous cycles. But actually what’s happening here with the capture of this is that you are getting what becomes a vicious cycle, where what should be the distinction between your commercial interests and your academic interests, your academic interests are now being influenced to be focused on something which is aligned with rather than providing breadth to their commercial interests.

And of course the amount of money involved is, interestingly, on the public side, of course, becoming almost negligible because so much money is being captured. The latest I saw, which I really loved, was that finally, if you take the five largest GDPs in economies in Europe, they are smaller than the five largest AI firms, and the largest of those economies, Germany, is smaller than Nvidia’s market cap.

So we now have a situation where the five largest AI firms are in some sense wielding more wealth, able to influence more just in wealth terms than the five largest European economies as a whole. This is, you know, it’s a very interesting situation we’re finding ourselves in.

[00:15:44] Kate: And I think it’s one where you see that these are systemic problems where the idea that you can just push one button and it’s going to change anything. No, you’ve gotta change lots of things that all relate to each other. You could think about tax incorporations and what that means for having more public money to go to a different kind of research.

I don’t know policy wise what the best decision is, but you can see that there are all these variables that have to be weighed. And I guess, as you’re talking, there are a couple things I hear. There is increasingly – you see this in the UK, I can’t speak to everywhere, but I think you do see it in EU grants too – there is so much awareness of how this gets to commercialisation.

And I get that, you want something to have a sustainable business model. But sometimes that rush to commercialisation is counterproductive, it’s rushing something. Also, the US is really good at commercialisation and I would argue the UK is way better at really interesting research. And so, as the UK wants to be more like the US – and again, I would say these are harder problems of how people invest and risk tolerances and all these different things that you would have to change the culture to really make the UK like the US, and do you even want it to be that? – there is this feeling that unless you look like the US you cannot introduce interesting, meaningful innovation. And I think that is also causing a lot of problems and very much affecting the direction of AI because I see calls in the UK where I think it’s trying to be the US.

[00:17:20] David: And, as you say, one of the problems with this is the nature we’ve talked about of these tech approaches, these Silicon Valley tech approaches, which are deliberately trying to create this monopoly in a way where if somebody else tries to do it afterwards, they just get locked out. And broadly the same is happening at the national level with this type of tech innovation. If anyone else tries to emulate the US’ model for commercialisation, they’re likely to just get bought out if they’re successful, because that’s exactly the model which has been developed.

[00:18:03] Kate: That was what I was going to say. You mentioned that Cambridge company that got 70, 80, whatever millions it was, and I thought, is that just a classic? the acquisition or aqua funding,whatever it’s called, where you are funding a company on the logic that it’s just going to get acquired and be absorbed by the beast.

And I think that is what happens. This is a big issue in the UK. There’s interesting startup momentum and a lot that’s coming up, and then it just gets funding from the US, often the team gets taken over to the US, all of that energy then just shifts, follows US money. And then you end up with the talent at the scaling phase going to this very commercial existing AI model of what AI should look like. 

[00:18:53] David: And so let’s come back to the key question you started with for this episode. It is this point: what is driving, what’s really driving the current systems? Well, it is the tried and tested. It isn’t new innovation, it’s exactly what has worked for Silicon Valley tech companies over recent years in which has created this dominance. And it is that, applied to what is a very seductive, simple message: artificial general intelligence, more compute, more data, will solve any problem you can imagine. How can you argue against it?

[00:19:39] Kate: Well, I was gonna say, the other thing that came to mind when you were mentioning that is that perhaps we are long overdue for a conversation on externalities and business. Because I read that quote from Karen in a previous episode where it’s like, “the cost of society in the short term, those are just to be borne because we’re gonna achieve this grand future, whatever problems we’ve created, they’ll be solved. So it won’t even matter, we’ll just fix them.”

And not only is it pretty clear that’s a fiction, but just the idea that you can run a business that’s introducing all kinds of harms – and we’ll talk about this a bit more, I wanna do a future episode where we talk about the product versus the system behind it – the fact that that system is actually introducing lots of things that society will bear the cost of, and the fact that a business never has to be accountable for that..

We, as taxpayers, pay for it. Communities pay for it. Whatever it is, the environment pays for it. I think we are entering an age where AI is the poster child of “we have got to start holding businesses accountable for the costs of what they do”. You are not running a profitable business if the byproduct of your pursuit of profits is so much harm with so many knock on effects and costs for society. That it’s not a real thing, I think, if you really start to make people accountable.

[00:21:14] David: Well, I think the simple thing is it’s no longer profitable if you start to make people accountable. But that’s the key thing, this is a question of, is it regulation which is needed to make people accountable to change the landscape? Is that even possible now because so much deregulation has happened? And really that’s what has enabled us to get to where we are. That’s a really interesting, different discussion, and good for another topic, let’s do it. 

[00:21:45] Kate: Okay, yes, we’ll talk about policy capture, I think, also as a topic. That would be a good one. Okay.

[00:21:50] David: This has been fun. I look forward to our next one.

[00:21:54] Kate: Great. Thanks, David.