184 – Twenty Years of RMS for CRFS: Controls

[00:00:06] Lucie: Hi, and welcome to the IDEMS podcast. My name is Lucie Hazelgrove Planel, I’m a social impact scientist and anthropologist, and I’m here again today with Roger Stern as part of our special series on research methods for agriculture.

Hi Roger. How are you doing?

[00:00:21] Roger: Hello, Lucie. I’m doing very well. I’m looking forward to talking about your next topic, which I believe is controls.

[00:00:29] Lucie: Yeah, so controls are usually something that we discuss in terms of agricultural experiments, I think, and you’ve said before that it’s one of the key treatments that tends to be used or tends to be considered. I think we’ll see later, it’s also got its role in surveys, it’s equivalent in surveys.

But let’s start off, I guess, with discussing what is the point of a control or what is a control? How do we define a control?

[00:00:57] Roger: To me, controls are very simple and sensible. And then they get misused. A control is just simply one of the treatments, and it’s often thought of as the baseline with which one wants to compare other treatments. And it’s then easy to give some obviously good examples, and then that starts the trouble because we try and adapt those examples to a different area and they don’t adapt very well.

The obvious example is in a medical trial, where you have a new drug.

[00:01:37] Lucie: But before we get to an example, what’s the use of a control then? Why are they useful? So you mentioned baselines, but why would we want to baseline?

[00:01:46] Roger: In our analysis with these treatments, we usually find it’s much easier to compare treatments rather than look at individual treatments in isolation. So the analysis is going to be a comparison, and when we compare, we often say compared to what. And it’s nice to then say compared to the farmer’s usual practice, if I’m doing something interesting compared to doing nothing.

And so often the control is what would happen if you weren’t doing your experiment, the standard thing that people do now, that’s often the control. And many people assume that every experiment has to have a control. And they give themselves artificial rules about controls, and the positive reason for having a control is that that is often the comparison of what would happen if people carried on as they were doing now.

So this is the current practice, or “if nothing was happening”. And that forms a nice baseline where you are trying to measure change and you’re trying to measure differences, differences compared to what? The control is often the what.

[00:03:07] Lucie: And this is why it’s often usually in experiments and things too, because it is trying to identify that sort of cause and effect. So the, well, “what happens if we did nothing” compared to “what happens if we did something”?

[00:03:19] Roger: Exactly. And that’s the obvious reason for having a control. But, to my mind, the control is then just another level of your treatment. Now, it may sit outside, often you want your treatments – I’m obsessed by factorial treatment structure – it might sit outside those treatments, and just be an extra treatment, and that’s fine.

But, the general rule is “think of the control as another treatment”, and then the corollary is “in the experiment, it must be a sensible treatment. If it’s not sensible, don’t do it”.

[00:04:01] Lucie: I had perhaps an example of this, an example that I came across, Roger.

[00:04:06] Roger: Give me your example.

[00:04:07] Lucie: So it was a really interesting experiment that was investigating how to conserve vegetables in places where there’s no electricity and no fridges. I can’t remember which country it was in, but the people have a traditional way of storing vegetables, which is in these earthenware pots, large jars, and the experiment was looking at how the vegetables degrade over time.

And it was comparing vegetables in different jars, well, in jars, compared to vegetables stored outside of the jar. And so the control was vegetables stored outside of the jar. We saw the data, and the data was collected, I think it was over two weeks, but we noticed that for the vegetables stored outside of the jars, there was only data for the first two days.

Then, I can’t remember if it was zeroes after that or if it was sort of someone had written “not applicable” or literally blanks in the spreadsheet. So we wondered, well, what happened to the controls? You started off with an interesting control and then your measurements of weight disappeared. Did an animal eat the vegetable? What happened to it?

So, I’d be really interested to hear your thoughts about what should somebody do with a control, what makes a useful control. They just mentioned something sensible. Here the researcher was taking data about the visual appearance of the vegetables and their weight, but the data suddenly disappeared after two days.

[00:05:47] Roger: So, let’s compare that to an example with a control that’s very sensible, but is similarly extreme. We have a new drug that we want to test, which is going to help cure people. A standard test would be, a standard test often might have three treatments, the best existing cure, and no cure at all, namely a placebo. This is a very common medical trial.

And so your control is like the placebo, that’s given, and then the measurements are taken over the two weeks, but a feature of the medical trials is that the placebo is often an equally sensible treatment and therefore you can take the measurements over the two weeks, often it’s what we call a double blind. The people having those treatments don’t know which they have.

Again, in a medical experiment that’s only sensible if the placebo is a sensible treatment, so you can take the same measurements as you can for the other treatments. Otherwise it becomes a nonsense, and you learn very early on that the placebo was very different.

[00:07:05] Lucie: So, in the vegetable example there, you can’t have a placebo effect. But as you’re saying in terms of a sensible measurement, if people are not storing their vegetables just outside on the counter, or outside on a table anyway, then does it make sense to have that as a control?

If people are storing their vegetables anyway in these jars, then is there a way of altering the control so that you can see, well, how is it best to conserve vegetables in the jars? Should the jars be inside? Should they be outside? Should they be open?

[00:07:39] Roger: Yes. Well, your example would fall into my instance that it should not be in the same experiment. That doesn’t mean you don’t include it, but you include in the experiment the things you want to compare in the analysis. And, if storing outside those jars only lasts for two days and you want to take measurements over two weeks, you are welcome to consider it somewhere. But, don’t think of it as part of the experiments and have to repeat it as you do the others. You just observe that it’s only two days it lasted outside the jar.

That doesn’t have to be in the same type of jar or the same number of repeats. That’s just an observation because it’s blindingly obvious that it’s a big difference. When differences are so blindingly obvious from one treatment to all the rest, don’t think of it as part of the same experiment. It’s a different study, and possibly you just have to have one replicate sitting outside, and you say if we didn’t have jars, then it would last only so long.

However, it’s interesting to take that example further and to think, if you didn’t have earthenware jars, but you still wanted to conserve vegetables, what would people do? Is there a sensible treatment for people who couldn’t afford the earthenware jars? Or, do you have to have earthenware jars if you want to keep them for two weeks?

So you can turn often your unrealistic control and say, “could you have a sensible control”? And that often can add something to your experiment, another objective. So, I’m very happy with the idea of controls, but they must be sensible treatments for that experiment.

Otherwise, it’s a totally different experiment where you’re taking different measurements. You are just measuring that after two days or three days, it didn’t work. And it might be sensible to do that, but everything is different and so it doesn’t need to be part of the same experiment, it can just be off to one side.

[00:09:51] Lucie: That’s really interesting yet to separate out, you know, for some experiments to think of the control as something very separate, or to have multiple controls.

[00:10:00] Roger: Yes, that’s right. Because the control is just another treatment, often once you stop thinking of “the” control, but start thinking of the sensible objectives of the experiment, you will have more than one control. And they’ll be different.

Let me explain Tricot as a nice study, where they are thinking of adding a control and they haven’t had controls so far. Tricot is a series of on-farm experiments, where farmers are given three varieties of a crop. And often the experiment, and usually the experiment includes maybe 10 or 15 different varieties, but they’re given three plots of land and they don’t know which variety is which.

They call their varieties A, B, and C, and at the end of the experiment, after it’s been analyzed, they’re told which varieties they had. And their results are fairly meaningless for just one person, but there’s a thousand people. So, they actually have all the varieties. There’s a lot of discussion as “could they add a fourth?”, and the fourth is a control variety and they do know which it is.

What do you do with the fourth? The fourth is different because they do know what it is. But should they, instead of having three plots, should they have four plots? I think that’s a very nice idea, and that means that you still have your experiment as before, but the farmer can add their own local variety and can also ask further questions of the varieties they don’t know against the fourth one, which they do.

And, that could enrich the experiment and also include a treatment that the farmers know as well. And so it could be more realistic for the farmers.

[00:11:58] Lucie: But this is a really good point in terms of controls with farmers, we sort of mentioned it right at the beginning that often, if you’re doing an on-farm trial, then your control may not be “not to have any treatment”, it might be “to look at what the farmers are doing already”.

[00:12:14] Roger: Yep.

[00:12:15] Lucie: Or “what they want to be doing”.

[00:12:17] Roger: That’s right. And that’s often an additional aspect of what the farmers are doing. And one can take that a step further, because you could now ask “that local control, which the farmers do know, is it the same for each farmer?” Or is it a “village” control, so it’s the same for each village, because often there’s a group of farmers in the village?

It could be different for each individual farmer. It could be at a village level, or there could be the standard for the whole area of the experiment, the standard variety that people seem to use, which could be one of the varieties in the experiment. But they know what that one is, they don’t know the others.

[00:13:01] Lucie: Exactly, but depending on the practice, sometimes it’s different farmers who do completely different practices, sometimes it’s cultural practices.

[00:13:10] Roger: That’s right. Now, that control could be the same “local variety” for all the farmers. But it’d be interesting to add a fifth plot, maybe, to say “would you also like the one you would prefer if you could choose any variety, or the one you habitually use”. So you have five plots, you have the three you don’t know, you have a common local one that you do, and you also have one of your choice as your own control. Would you like to do that? And then you’d have two controls.

Now, bear in mind, these are all sensible treatments. But the fifth one now is different, potentially, for each farmer. And one of the aspects of that is you may now have different questions, namely, in addition to the measurements that you’re taking on all the other plots, you might ask the farmer, why did you choose that variety? What is it that is particularly attractive about that variety? Is it the taste? Is it the color? Is it the yield? Why did you choose that?

[00:14:23] Lucie: And that is so important.

[00:14:25] Roger: Let me give you an example of an experiment like yours where I thought the controls were a disaster. This was an experiment that was quite expensive and was repeated in many sites, and it was an experiment where farmers had four plots. And it corresponded to the perception that many climatologists have that if only farmers listened to climatology and understood the climate better, and followed climatic advice, then they could grow better crops.

By the way, I don’t believe that, it’s a complicated question. And that would apply equally whether the crop was fertilized or not fertilized. So they had two plots, one of them was fertilized, the other was not fertilized, and they gave local climatic information and advice, when to plant, when to weed, how to weed, et cetera. Those were the two treatments.

And then they had two controls, which were what the farmers would do if they didn’t get the climatic advice. And so those were three and four. The conception was sort of okay, except many of the farmers, once they got a bit of the advice, they applied some bits of it to some of their three and four, their controls, and others they didn’t.

[00:15:54] Lucie: And that’s very sensible of them. I mean, if they hear that they are being given some advice, then obviously they would think, well, this might be interesting.

[00:16:02] Roger: And the experiment was a disaster because the farmers took different bits of the advice differently, but nobody measured what the farmers did for the controls. They measured what they did for the treatments because that was the advice.

So had they adapted that and asked the farmers some questions on “do you like these bits of advice”, you could have turned that into a sensible experiment. But as it is, and they just measured the yield at the end under these four treatments, the controls lost all their meaning, and the whole experiment was a disaster.

[00:16:41] Lucie: But that’s a really interesting word of warning there, ’cause a lot of the researchers we work with are developing new solutions, well, new solutions or new options for farmers to work with. And so they want to know how do they compare, and how other new farmers who weren’t involved in the development phases, how they respond to it.

But yes, if you are introducing it as advice or if you’re introducing it as something which could be new and more positive, have more positive effects, then I think there’s some interesting challenges there in really analyzing it, considering what the treatment can be and how to take that into account.

[00:17:18] Roger: That’s right. So adding controls can be difficult. Having a baseline can be difficult. And linking that into an experiment, just make sure that all your treatments are sensible, so you can relate to those objectives. And then often comparing the treatment to the control is a very sensible thing to do, but sometimes it leads you astray with the fact “as a rule, you must have a control”, and if there isn’t a sensible control, you’ve got to think more imaginatively.

[00:17:51] Lucie: And to me, I always think of baseline more like in terms of surveys. So if you are developing an experiment and you want to use farmer practice as a control, to me it would make sense to do a survey first to understand what is this to farmer practice and to try and decide what the farmer practice, control or controls should be in that experiment.

[00:18:11] Roger: That’s right.

And that brings us back to the experimental one and to your experiment, which was a disaster, namely, what do you do to conserve vegetables if you don’t have earthenware jars? So, if you’d done an experiment before, sorry, a survey before, then you might find out that those people that don’t use earthenware jars use something else to conserve, or else they do use a fridge, because they don’t need the earthenware jar when they have a fridge.

And so, you might find that you can, out of this, develop a realistic control, so that if the farmers didn’t have access to earthenware jars, what would they do? Well, they would have to search for a fridge, or they would not try and conserve the crops, they would go to the market more often. What is it that they do?

So, the controls are just treatments – make them sensible. And if the differences between the results from the control, if the differences are major, then they don’t need to be part of the same experiment. If they’re blindingly obvious, they’re not part of the same study.

[00:19:20] Lucie: But they can still be useful things to have data on, if you think of a different way to incorporate them, then.

[00:19:28] Roger: Absolutely. Your particular example, no problem having those vegetables off to a side, they don’t need to be part of the same study. But finding out whether it’s two days or three days that they’ve died might be sensible as a separate little exercise.

[00:19:44] Lucie: And I’ve got to also mention one other big aspect which is fairly obvious perhaps in medical types of experiments, but people perhaps think of a bit less in farmer type of agroecological trials, is the ethics of these controls. So, if you are doing a research trial where your control is just to have no treatment and that has no relationship to what farmers usually do, and you know it’s going to give a poor yield or something, it’s going to give a poor result, then is it even ethical to use that as a control?

I think you’ve mentioned previously that if in a medical trial you’re not going to treat some patients, if you have some drug that you want to test the effects of on the population, but you want to compare that to not treating people at all, then it can be really unethical not to give some treatment to people who would otherwise you think benefit from it.

So same with farmers. If you are asking farmers to do an experiment on their farm, not applying any fertilizer or something, it’s not going to be of use to them, it’s not going to be, to me, ethical. Perhaps the risk is less high, But, asking them to do something which you know is not going to be beneficial is not advised.

[00:21:05] Roger: I think that’s right. You make it ethical, and there is the question of do you need to worry ethically with plants the same as with animals, and the same as with people. Clearly with livestock and with people, absolutely. With plants, there’s lots of experiments where you might know that applying no fertilizer or not weeding is a disaster, and should you?

Some people don’t worry so much about the ethics, which maybe they should. But even without the ethics, you find that the experiments that have ludicrous controls, like not weeding and the weeds rampage all around, you have to worry about in so many ways, don’t include it as a treatment, it’s not a sensible treatment. And you finally get dragged back to that often because it’s weeded so much that your guard rows don’t work, and so it has impacted the neighboring plots, where you do have sensible treatments.

[00:22:05] Lucie: Yes. Interesting.

[00:22:06] Roger: And in the medical experiment, you are absolutely right that there are studies where you are not allowed to have a placebo, and others where even if you are, the minute you know that your drug is better, then you need to stop the trial and then give everybody the treatment.

[00:22:28] Lucie: And this ties in also with new technologies. If you are promoting a new technology, like a bio pesticide, if you know that the bio pesticide works or is proving to have some positive impact, then is it ethical to still ask the farmers, the other farmers not to use this bio pesticide when you know it is going to be helping them?

[00:22:50] Roger: That’s right. Of course, the pesticides are interesting because they may bring in other side effects that may be undesirable as they bring in some positive benefits. No, absolutely.

[00:23:01] Lucie: But then also, sorry, I’m just thinking with the bio pesticides, you also come with the thing that, well, farmers themselves may be choosing a bit like your example of climate advice, farmers may be hearing that there’s this bio pesticide that other people are using. They may be hearing that it’s having positive effects and they may decide to give up on their control, which is no treatment or something, and to start using this other miracle solution, whether or not it’s a miracle solution, of course.

[00:23:26] Roger: And remember, if you can’t control it, then measure it, so ask the questions. So often a poor control that people are abandoning, you can turn into a positive benefit in your study by asking questions of the farmers, why are you abandoning it? This is where you should be open to collecting the data that is needed for a study, even if it’s not quite the way you designed it.

[00:23:56] Lucie: Great. Thank you very much, Roger. Do you have any final pearls of wisdom to close this conversation?

[00:24:03] Roger: If I wanted to add another, I don’t know whether it’s a pearl of wisdom, remember we get very quickly from having an unreasonable treatment into deciding to do some measurements that weren’t previously planned. A big example of that is if you’re doing an experiment for two years, often you can improve on doing it exactly the same for the two years by learning what you could do better at the second year, and not just saying we have to repeat something for two years. Let’s learn as we go along, to collect the best possible data.

[00:24:36] Lucie: Yeah, and we’ll have another discussion, I think, about that sort of multi-year trial, and how you can learn and how you can evolve your study as the study progresses through different seasons.

[00:24:48] Roger: Sounds good. Looking forward.

[00:24:50] Lucie: Thank you so much, Roger.