06. What Lies Behind AI as a Product?

[00:00:07] David: Hi, and welcome to the IDEMS Podcast. I’m David Stern, a founding director of IDEMS and it’s my pleasure to be here today with Kate Fleming, another director. Kate, I’m looking forward to another exciting discussion.

I guess last time we did get a little bit political, which is beyond where we’ve often got to.

[00:00:24] Kate: This time I think we’re gonna get even more political. So, I have referenced this in previous episodes in this series, but the topic I wanna talk about today and really dig into is a technology product versus a sociotechnical system.

Silicon Valley, as we know it, has been very good at presenting AI as a product. You think about the output, you think about the user experience, this is how it’s sold to us, and we come back to that artificial general intelligence dream. That’s a product. There is a lot of just sweeping under the carpet, the actual system behind that product, the humans, the resources, all of these things that are going into it.

And I think we’re seeing these bubblings up of a lot of disgruntled people, when they’re actually feeling the consequences of what’s happening under the hood in their own communities for themselves.

And I will say in Karen’s book, Empire of AI, the theme that runs through this, really the theme of her book, I think, in many ways is calling out what is actually happening behind the technology that we should be far more aware of and thoughtful of.

And the other reason I bring it up is we are often in the position when we talk about the system in which we work and how we have thought about the different components and how they need to work and who needs to own different things – I think I say this every time – we are conceived as just being mad. Like, this is crazy, why would you think about all these things? This isn’t what you do.

But if you look at a business, they’re often holding all of those things as exploited employees, data centres, whatever all those different things are, they’re subcontractors that they’re paying, there is a model there. It just works on very different assumptions, which is that everything is just bought and sold and it’s a race to the cheapest provider or whatever.

So that’s what I wanna talk about, what is that system, what are the components of the system? I think the way I think about the structure of the system really comes down to data. So how is data held, how is it captured and collected, how is it cleaned, how is it processed, and then what are the outputs of that data system? And who does that work, what are the costs of that work? That is what we are thinking about.

[00:03:02] David: There’s two things I’d like to start with on this topic. The first is the cloud and the second is the Fairphone. And I think then I’d like to come back to what you are wanting to really dig into. But the cloud and cloud computing, I don’t think I understood the narrative, the product narrative work that had gone into that until I was at this event, the Earthkeepers event. And the slogan that they used is, there is no cloud, there’s only Earth.

And the point they’re making is that, actually, cloud computing happens in data centres, it happens on physical devices that live somewhere. I’m trying to remember whether it was Elon Musk or somebody else who was saying they’re gonna put the data centres out in space.

[00:03:55] Kate: Probably Elon Musk.

[00:03:59] David: It might actually have been Jeff Bezos. But this idea that you could actually take what is the physical infrastructure and just put it in space, so it is really in the clouds. But even so, it has a life expectancy, it’s gonna come down eventually, these are the things that we are seeing. Anything that goes up must come down at some point or other. But wherever it is, it is physical. And so this idea of selling the product that is cloud computing as something which isn’t physical, is just…

[00:04:31] Kate: Magic.

[00:04:32] David: Magic, exactly, it’s magic, it’s in the clouds, you know, you don’t need to think about it, you don’t need to worry about it, it’s just there. It is really good marketing. And it is exactly what you are saying, it is saying that you don’t need to worry about the whole system, you don’t need to worry yourself with all the complexity, it’s just there in the cloud somewhere, it’s magic.

[00:04:52] Kate: And it is incentivised. So when I held data on my computer, I regularly cleaned it because I was like, oh, I’m running out of space, I’m gonna have to get rid of things. Anything that is held on the cloud right now, I am so lazy. I have all these photos, I have all these things saved, I’m taking up so much space and oh, well, maybe I’ll pay a dollar more a month just to increase my storage a little.

But the blissful ignorance that we live in – and I don’t even live in blissful ignorance – but it creates that feeling, where I don’t really know how much storage it’s using, I don’t know how wasteful this is, I’m definitely not asked to question it. You know, what’s a dollar to me every month to not have to make decisions about my photos? All right, that’s easy.

[00:05:40] David: Exactly. So I think before we even think about this with respect to AI, which is I think the topic we want to dig into, cloud computing is such an important piece of that and of that narrative, which set up the narrative which is behind the AI boom. I think that’s a piece of the puzzle.

And of course when you then get to the actual physical infrastructure, this is why I love the Fairphone and the work they’ve done to try and actually worry about the sourcing of every mineral to do everything ethically and properly, and understand what does it actually cost to produce a device in a way which is properly sourced, where everybody along the value chain is considered. They’ve worried about all the details. Maybe they have missed some, but not that I’m aware of.

I’m so impressed by the level of detail they’ve gone to in this. Just in case anyone’s wondering, I have no association with Fairphone as a company. But I’m just in awe of the level of detail they’ve gone through. Fairphone have done that hard work. Nobody else does it, for good reason. And the cost of a Fairphone is substantially more because that’s what it costs to do this properly, to do it well.

I’d love, as societies, to get to the stage where actually the Fairphone became standard. What if instead of big companies buying everyone the latest Apple device or whatever, they actually said, no, we are gonna buy the Fairphone on mass and actually go through this and do this at a level. That would be incredible.

[00:07:26] Kate: And that would require, I think, a bit more, well, it wouldn’t require, but it feels related to our conversation we had before about externalities, where Fairphone is seen as making dumb business decisions by certain people because it’s like, why wouldn’t you be racing to the cheapest product, who really cares about those things? Let’s just get this product at scale to market. And in the landscape where everyone is just held, a business is measured just by profitability in this very narrow way, they aren’t running a good business by those metrics. This is the world we live in, where there’s something so broken.

[00:08:05] David: It’s not necessarily that it is broken, it works extremely well, that’s the problem. It’s just not working well for everything or for everyone. This is the point, if it was broken, then, in some sense, it would be easier. It’s only broken from things which are not aligned with the metrics that they care about.

[00:08:29] Kate: Yes, absolutely. But I think we’re saying the same thing, where that is broken and it’s what’s destroying the planet, it’s what’s making people unable to live. And I think Karen really digs into this, I think she’s successful at not feeling like she has an axe to grind. She is just genuinely looking at, well, who’s doing the labour, for example, of cleaning this data.

So you have all of these images just scraped from the internet. I don’t know what’s good and what’s not. Also, you have horrific things on the internet, you have murders, there’s so much sexual content, all of these things that are, who decides how that has value? That is very low wage workers.

And she follows the story of a company, I will call them out, it’s called Scale AI. Their whole business model is to go to desperate countries, they go to Venezuela, they go to Kenya, they go to countries where people are really poor, they’re really on a knife’s edge, they get them to do this work, they keep them really precarious. They’re contractors, they have no right to work, they increasingly see work they had depended on pulled out from under them. When they try to organise, the company is just like, well, forget this country. And that work goes away, which was at least better than no work at all. But you see this whole system of significant labour exploitation that is being offshored.

And these are not just unrelated issues. These are unstable countries where the politics of those countries affect the political decisions that our countries make. We do not live in little isolated bubbles. And I know this is obvious, but somehow in the context of tech, these things get treated like they’re not legitimate concerns or that’s just kind of do-gooderism or something, if you’re thinking about these issues. But these really are.. Yeah, I don’t know, it’s hard not to wade into politics here.

[00:10:31] David: Well, but it is also hard politics to wade into. As you say, the very simple thing is these sort of digital jobs in Kenya, they may not be great, but they are valued by people. This is what makes it so difficult and so contrasting. Actually, the “bring Trade, not Aid” is a big concern in these contexts.

[00:10:53] Kate: I don’t think that’s the argument though. I think the initial case is, great, if you’re bringing that work and it’s fair and it’s paid a fair wage for the context, great. It’s not like that. You are not after paying a fair wage, you do not want to have reliable employees. You want to have a really precarious workforce, you wanna have a race to the bottom, you need people who are good enough at tagging what this image is, or this is sexual content, or this is too violent.

Not to mention, you could imagine the harm. I think this is well documented in Facebook, various things that people who are in those moderator roles, the stuff they see is horrific. And I will say Empire of AI does get into that a bit where it’s not just about money, it’s about workplace protections. There are things that labour movements have fought for that are just being undercut consistently. And those things benefit society.

[00:11:50] David: And it would be illegal to have people doing some of this work in Europe, for example, in certain ways, and those protections are not afforded. No, I absolutely agree. But this is what makes it all so complicated that this is bringing trade, not aid. And so by other measures, this is part of what is desired. But not like this. And that’s the key point.

[00:12:14] Kate: I bring up Scale AI because she mentions that the founder is now worth like $24 billion or something like that. And it is the point of: do you need that? Do you really need that much money earned on the backs of very low wage workers? 

So, I think this gets us back again to: it’s not that we’re saying that things can’t exist. This is where we think about the system. I would love to bring trade, I would love for people to have reliable jobs. But that means that we can’t build a business where we’re exploiting them, where we are assuming that we are going to become billionaires on their backs. It’s actually a lot of letting them run their own businesses, often selling directly to whoever, whatever those systems are. You have to build very different assumptions about who gets rich, if anyone does, and when enough is enough.

[00:13:08] David: That is getting very political. Those questions, you don’t even need to answer those questions or get to that level to get to the stage where we recognise that calling out the fact that AI has human labour deeply embedded in it is what’s needed.

So that’s really where you started this discussion, that if we move away from product thinking to actually calling out these systemic thinkings, when you’re saying that you are using AI what you are really using is the product of this structure and this is what the structure actually entails. Actually calling that out and actually laying it out as what it would take to do this and that there’s actually choices that have been made all along this and you may not have access to all of those choices, but there are different companies who make different choices.

I was in discussion just earlier today with someone who was saying: which AI company is ethical enough that I should use them? These are questions that people are asking, and if you are just presenting it as the product, it is difficult. Whereas if you actually recognise the system behind the product and you can call out and you can actually have measures of indicators of what different systems are doing, then you can give them metrics to work towards that people care about.

[00:14:35] Kate: I think it also lets you see that you need different solutions that work in different ways. So the thing we see all the time, there is a model where the only way you get data is just through surveilling people. And that doesn’t necessarily get you good data, it gets you a certain kind of data and it reinforces a lot of harms, it reinforces a lot of biases, whatever.

So that would be an example of if you really want good data, you might need to give ownership to a community, you might need to think differently about what you are, how you are sharing to get to that thing that is going to actually be great. Where it’s like this is how we cure cancer because we’ve actually figured out a system where everyone can kind of own their research, but they share it and they reap the rewards of it. It’s not just that they’ve shared it and then somebody else monetizes it. And so they share more openly, they share more collaboratively.

You can imagine how when you build different systems of sharing, of trust, of ownership, that’s when you get much more likely to solve really hard problems.

[00:15:42] David: Your example of curing cancer is nice, but it’s too far removed. Let’s take simpler examples. Your example of labour exploitation. Actually, how do we build systems where you can be confident that the product you are buying doesn’t include labour exploitation?

Well, that could happen if the data was done differently. And if you’ve just got your surveillance data, well, that’s not going to be the right way to do this. And many clothing companies try to do this, they try to make sure that there’s no labour exploitation in their value chain.

If that’s something that you care about and you want to ensure, that’s where you might need a different type of data, and data which actually corresponds and plays out differently, as a really simple example.

[00:16:27] Kate: I think that’s too focused on a consumer, whereas I would wanna focus on what does it actually look like to deliver impact? And that is something that needs something very different from, I just wanna make sure my shirt has… And, I will say, by the way, supply chain is really hard. Managing exploitation in supply chains is nearly impossible because of obfuscated supply, like contractors and various things. So that’s one thing.

I think what we’re looking at is, if you are going to really make impact work in some individualised context, you often need a lot of information from that context. But if people don’t trust how their data gets used, they don’t trust that it’s not going to be weaponized against them, they are not going to share that data. And so you just keep producing solutions that aren’t really solutions, they’re not that good, they’re okay maybe.

And I think that’s a more interesting case of where you want to be able to see the system, because I want to be able to really see in this complex way that impact is meaningful, that it’s being driven by the right data. And how you do that is you have to externalise the system because you have to take into account, well, these people are worried about that, this data is bad because of this, but this data is good because of this. There’s so much stuff you have to understand to make that possible. 

[00:18:01] David: Absolutely. And, as you say, it’s all about recognising that there is a complex system to everything we do, to the food you eat from your supermarket or from whatever, right the way to the education we are receiving, or our kids are receiving. You know, all aspects of our society, to the AI we’re using.

It is not a product. It is a whole system behind it which is producing that. And we have, as a society, we have choices we can and should be making, which influence these and which might change the equation of what’s profitable. If we actually see, and we understand this as the systems rather than the simplistic products.

And I think what I like in what you are suggesting is that we are not arguing that people shouldn’t be searching for profit, we’re arguing that the profit, the really profitable things should be of benefit to society. That’s what capitalism promised. The promise of capitalism originally was that this is something which is in society’s interest for us to actually build things efficiently, effectively.

But by making the systems invisible and just presenting the products, we’re hiding the information that is needed to make good decisions and to be able to charge or tax effectively, to be able to have policies that work to the benefit of society, consumers.

[00:19:44] Kate: Yeah, and I think this is where we feel quite frustrated as a social enterprise because the fact that we are very focused on impact, we do think about those other costs, they are disqualifying so often because we are not single-minded in our pursuit of profits in this very narrow way. Forget the fact that there are all these other consequences that will be born by the state, by taxpayers, by whatever, that pathway is perceived as having so much more merit and being so much more legitimate, where we are seen as being naive, being do-gooders, definitely in this like charity, nonprofit lane where it’s not credible in the same way, you can’t be as ambitious if you think this way. And so it definitely doesn’t serve us when product thinking is just the be all, end all.

[00:20:39] David: We’re gonna have to have another episode where we dig into some of this from our perspective much more. But the thing I do want to just draw out in what you’ve said is that the truth is, I believe, really the opposite. We don’t even need a level playing field. We can outcompete, we can do better than our competitors, even in a playing field which is stacked against us, but maybe not stacked so heavily against us.

On a level playing field we could outcompete so easily, but we don’t even need a level playing field. This is actually what’s so interesting. We cannot access an overdraft facility from a bank, you know. What sort of level playing field is this? This is ridiculous. We don’t want special favours, we don’t need special favours, we don’t even need a level playing field. But not being able to get an overdraft on our bank to manage cash flow, no sensible business has to deal with that. And here we are having to play on those rules.

[00:21:41] Kate: Absolutely. I would say regulation so favours that product model and disfavours other models. And it’s not even us playing the victim. So we had this Knowledge Transfer Partnership funded by Innovate UK. We got outstanding, which is the highest thing you could get, we had all this great work come out of it. We wanted to apply again with the support of the university, we were told we were ineligible simply because we are a social enterprise. That is not a level playing field.

[00:22:11] David: It is absolutely crazy just how far the deck has been stacked in this way. And yet the narrative is that they’re more efficient, they’re more effective. That’s just wrong. We know how to be, we can get better talent more cheaply because we are giving people work which is meaningful. And so we can outcompete. If only the deck wasn’t quite as stacked against us.

Anyway, we’re getting sidetracked on this, but, let’s keep going.

[00:22:45] Kate: It’s all relevant, yeah. Thanks, David.

[00:22:47] David: Thanks.