241 – Twenty Years of RMS for CRFS: Proxy Variables

[00:00:07] Lily: Hello and welcome to the IDEMS podcast. I’m Lily Clements, a data scientist, and I’m here with David Stern, a founding director of IDEMS.

Hi, David.

[00:00:15] David: Hi, Lily.

We’re keeping going on our 20 year anniversary of the West African Community of Practice Research Method support piece, aren’t we?

[00:00:24] Lily: A bit of a mouthful, but yes, we are continuing with this mini series about statistics and describing statistics, exploring statistics, the use of statistics, that you have experienced and Roger have experienced in this work.

[00:00:41] David: Absolutely. It’s been going on for 20 years now, between Roger and myself we spanned most of that time as the key liaison person for the research method support. So we’ve got lots of stories to tell, and today we’ll tell another story linked to a statistical idea.

[00:00:58] Lily: Yes. Okay. This is a story I have heard before, but I know that I really enjoy this one. I think it was your grain yield story.

[00:01:05] David: Yes, this is about proxy variables. This is the statistical concept that we want to discuss. People often think proxies are bad things, but they’re not always. They can be very good things. And my favorite and most illustrative example of this was when I had colleagues who were starting to do large scale farmer experimentation, and they came to me and they said, “do we really need to measure grain yield? Couldn’t we get away with just measuring the weight of the head?”

This was for millet, and millet has lots of small grains on a head. And so, when you harvest, it’s really easy as you harvest to weigh the head, but you don’t have the grains separated from the head at that point in time. And so, what they then need to do is they go home and they process them and they separate the grains from the head and then you can go and measure the weight of the grains.

And they had previous data, which included both grain and head weight, and they had different types of data: they had data from station experiments, and they had data from farmer experiments, if you want, experiments in farmer spheres. And so we said, well, I don’t know, I know nothing about millet. I don’t know if you can measure the head weight and whether it’s reliable or not. What does this mean if you have different varieties? Surely it depends on your variety and all sorts of different things.

But I said, okay, let’s look at the data you have. And the data they had on the on station trials was extremely consistent. Even across different varieties, their head weight and the grain weight were highly correlated, they were very, very similar. And then we looked at the farmer’s one, the ones from the farmers’ fields, the farm experimentation. And the correlation was a lot less.

And I tried to understand why, and it was very interesting that, proportionally, sometimes the grain yield was a lot less than expected compared to the head yield. It was never more, but sometimes it was a lot less than expected in all sorts of different ways. And so I said, this is very odd, what’s happening here? And he said, “oh yeah, this is why we want to measure the head yield. Things get lost, things get eaten, all sorts of things happen between the moment at which you harvest and the moment at which you measure the grains.”

And so actually, although the grain yield was what everyone was interested in, you got much better data on these on-farm trials if you approximated it using the head weight, using the head yield, if you want, as a proxy for the grain year, because you removed these other sources of variation, loss in transport, somebody eating it, all sorts of other things that were happening. And so it is not always “what” you are measuring, it’s also “when” you are measuring it.

This was a really big deal because this was almost 10 years ago, it was about nine years ago, and they were just starting to do thousands of farm trials and they were dreading the fact that they were gonna have to go around and get this grain weight for everything. It’s not feasible for thousands of farmer trials. And, as soon as we got these results that, not only was their head weight a sensible proxy, but it was more reliable data in the context within which they were collecting data than the grain yield would have been, and so they were removing sources of variation and therefore getting higher quality data, it was a win-win. It was great. It was less work for them, and it was better quality data. A wonderful example from the region of using proxy variables really effectively. There’s a relatively simple conversion, as I say, if you want to get precise, it does change a bit from variety to variety, but actually that was irrelevant for them. They didn’t even need that because that variability was so much less than the variability that was coming because of grain loss.

[00:05:37] Lily: Nice. And I guess that variability between varieties and checking the correlation is part of the process, and you did check that correlation and you were confident. This is something, I guess, is good to check before using a proxy.

[00:05:53] David: Absolutely, you have to have the data, which means that you can check this. Not only that, you have to understand, well, what was actually happening? If I didn’t have that qualitative information of why these were different, I might have said, “well, wait a second, that has something to do with your on station [data], your on farm is very different”. But once we had that qualitative data which was there in explaining the lived experience of why these differences were occurring, it gave so much confidence to go into that sort of scaling process of looking at more data in a way which was really effective.

Now, I have plenty of examples where people have used proxy data badly. Don’t get me wrong, proxy variables can be a very bad thing, where they don’t correspond. Here’s a good example of where this would now fall down: let’s say you are a breeder, and you start getting dual purpose millet where the head minus the grains is now used as animal feed. And so you’re breeding for more head.

[00:07:07] Lily: Yep.

[00:07:08] David: Now you use the proxy variable of head to represent grain yield and you are messing up everything because your varieties are deliberately changing. I don’t know if this is even a sensible example, but I do know that there are cases where corn cobs are, for example, used as fuel for fires for cooking.

[00:07:26] Lily: Okay.

[00:07:26] David: So, you know, there are cases where there may be incentives in terms of the breeding, in terms of the study, where you need this separation. And those studies then couldn’t use this proxy in a sensible way. You’d still need to define both variables, because that’s part of your study.

But when you are wanting to do these large-scale trials of sort of input – they were looking at human urine at the time, they’re now looking at many other things – but these large scale human urine trials, it was a sensible proxy for that. So, always asking the question of what are you using this proxy for and is it appropriate for that is really important. But the fact that there are cases where you can reduce the cost, get higher quality data, by using a good proxy, oh, it’s wonderful.

[00:08:17] Lily: That is an excellent case and a really nice one because, as you say, it makes things a lot easier, it reduces cost, it just speeds things up a lot, I imagine. And it’s more reliable, because there’s less variability in there of people eating the grain, or of it going missing elsewhere.

My experience with proxies is once you know what the proxy is, then you start aiming to hit the proxy target rather than hit the actual target, you know?

[00:08:48] David: Absolutely.

[00:08:49] Lily: But in this case here, I can’t see how that would occur.

[00:08:56] David: Okay, what you’ve just articulated is actually very deep. In many contexts, proxies are used because they’re the only thing you can measure. And this is used at national levels, let’s say, to understand progress on education. I have a really good example of a bad proxy.

[00:09:15] Lily: Okay.

[00:09:16] David: This is not West Africa, I’m afraid, this is East Africa, this is Tanzania, and it’s not agricultural, agroecology, it’s education, but it’s a really nice one. So, there was for many years a proxy variable for education quality that was staff-student ratio. And so there was funding and all sorts of things linked to reducing staff-student ratio in schools.

[00:09:50] Lily: “Reducing” meaning you have more students per staff or fewer students per staff?

[00:09:56] David: Fewer students per staff, you have a higher quality education because you have more teachers available, and so your students are getting more attention.

[00:10:07] Lily: Great.

[00:10:08] David: This played out over multiple decades, I believe. And in Tanzania, reducing staff-student ratio was done very successfully, and they got them down to really good numbers. The only problem was that the strategy that was done on a national level to do this was, “well, what’s the cheapest way to train teachers?” It was cheaper to train arts teachers than science teachers. And so they’ve got to the situation of having these proxy variables for education quality, over decades, which they had been hitting, where they had been hitting them and creating a six to one imbalance of arts to science teachers.

[00:10:55] Lily: I see. So you had six times the arts teachers…

[00:10:59] David: Six times the number of arts teachers than you had the science teachers, and that means that your arts teachers had one sixth the workload of the science teachers within school. And you couldn’t, you know, it was really hard to recruit science teachers, there weren’t enough of them, and it was really hard to motivate students to study science because they had these wonderful arts teachers who had lots of time to engage with them and tell them how wonderful the arts were.

And so you had half your subjects that were being neglected because your teachers were totally overworked, and the other half, which all the good students were being attracted to because, wow, what a good life you get as an arts teacher compared to a science teacher! And so, this wasn’t just a problem at one point in time, this was now perpetuating, the whole university infrastructure was biased, the teacher training programs were struggling to do this. And so this had become something where we got involved to try and say, well, what could be done about this?

But I want to come back to the fact that this is exactly your instance of the proxy variable becoming the target. The proxy variable was “staff-student ratio shows whether you have a high quality education system or not”. That became the target, that target was successfully met, creating very serious quality issues, which are gonna take generations to resolve within the education system.

And so, it’s very dangerous when your proxy variable doesn’t represent the whole. I mean, it’s a pretty good proxy variable, staff-student ratio is a pretty good proxy variable for education quality, ’cause it represents the attention students get, it represents all sorts of things. But it’s not good enough on its own.

[00:12:49] Lily: Yeah, yes, you need to then look into that data to then see your staff-student ratio, but also look at other proxies, but then maybe that would create other issues. I remember reading about this in universities and different things being used as proxy variables to do university rankings. And one of the things that isn’t a proxy is the cost of the university, which could be a particular problem in, say, the US where if the cost of the university isn’t a proxy, but all of these other things that cost money, like your gyms and your teaching standards are, then very soon your universities will become more expensive so that they can hit the other targets.

But it makes me wonder, with that kind of example, should you know the proxy, should you know what you’re being measured, or should you know how people are measuring?

[00:13:45] David: Let’s get back to the context we want to focus on, which is the West African context related to agroecology. And, you know, what are the lessons learned? Well, actually, in agroecology, one of the proxies for agroecological transitions at the moment, which is being promoted by FAO, which is the big UN organization for Food and Agriculture, is a measurement system called TAPE (Tool for Agroecology Performance Evaluation).

And I won’t bother going into that acronym, but this is supposed to measure your situation in terms of agroecological transitions, but of course it’s a proxy for that. Now, it’s a multidimensional proxy, and so it is much, much better. But it is something where, if people were to start using TAPE related to funding cycles and having improvements in TAPE relating to being deemed agroecological transitions, that is a really concrete example of where we need to learn from these other proxy variables and be very careful. Are we actually achieving the things we want or are we just working to the proxy variables?

There are concrete examples within the agroecological community, where proxy variables are coming in at this high level decision making, through these measures. I really hope we are learning the lessons of the dangers of those proxy variables, while also understanding the value that a good proxy can bring.

So, to me, what is so valuable about – and this is why proxy variables are so valuable – the initial example we gave, is that this was a cost saver, while also being very high quality data. Now, the question is: cost saver for what and for who? Well, your big measurements you are wanting to do for your big decision making, you need to be much more careful because cost saving may not correspond to higher quality data, but it’s the only thing which is feasible, and so you have to work with it, and therefore you have to work with the data you’ve got.

But you then need also to be very careful about what are the incentives, the perverse incentives or the incorrect incentives that could be inserted by using these proxy variables. It’s a really interesting topic. But I want to leave on the positive side that, assuming you are wanting to start scaling your research, to be able to reach more people, do larger scale trials, proxy variables can give you better quality data and cost less.

That’s the message I still want to leave with. As you try to get larger research data, having research data which scales within context – that’s the thing which is very different from the idea of proxy variables as indicators for policy decision making. I’m not saying that’s bad, you need them there, but they are more dangerous there – as a researcher, you can do the due diligence to make sure that for your study, you can actually get better data potentially at a lower cost.

[00:17:32] Lily: Excellent. Well, thank you very much.

[00:17:34] David: Great. Thank you.