180 – Twenty Years of RMS for CRFS: On-Farm Agricultural Trials

[00:00:07] Lucie: Hi and welcome to the IDEMS podcast. My name is Lucie Hazelgrove Planel, I’m a Social Impact Scientist and anthropologist, and I’m here today with Roger Stern continuing our series of conversations about research methods for agriculture.

Hi, Roger.

[00:00:19] Roger: Hello, Lucie.

I think the general discussion of trying to work on farm has had a very nice history. Maybe not as long as it should be, but the general argument is that restricting research to on station studies is very artificial because the farmers don’t live on research stations. So quite a lot of the research could usefully be done on farm.

What you do on farm is either do a trial on farm, and remember, a trial means that you have treatments which are allocated to plots in the farms, or you do a survey where you ask the farmers questions or you take measurements, but you don’t actually allocate treatments. So the distinction to me between a survey and a trial or an experiment is in a trial, treatments are allocated, and in a survey, you just ask questions.

Just to give an example, we might do a variety trial, and as a trial, we allocate the four varieties to a group of a hundred farmers and they each have these four varieties. And then you take various measurements, either you or the farmer takes the measurements, and then you have your data.

Each farmer then had four plots, and somebody takes the measure. And as we’ve discussed, there are different types of trial depending on who chooses what and who measures what and who manages what. So if it’s researcher designed and managed then four plots per experiment would be rather small, but it’s like an on-station trial done on a farmer’s field.

[00:02:05] Lucie: Yep.

[00:02:06] Roger: Much more common is where the farmer manages the trial with the four treatments, let’s say, but the researcher designs it, the researcher chooses the treatments and discusses the layout and the plot size, and the farmer then manages this small trial. And then the next level is where the farmer is involved in the choice of the treatments, either in choosing one or two of the treatments, the researcher chooses the other, and so the farmer can have the treatments for the farmer’s agenda rather than always having to fit in with the researcher’s agenda.

[00:02:48] Lucie: This sort of trio of three levels, it’s really interesting to think through different ways of working with farmers. And you don’t need to think of it in terms of whole numbers, that can also be, one point fives and two point fives and all the rest.

[00:03:01] Roger: Yes.

Now the problem as you go along this sequence is, partly again, the teaching of statistics. But as you move along this sequence, of course, the researcher gradually loses control and hands it over to the farmer.

[00:03:15] Lucie: Is that the problem?

[00:03:16] Roger: I don’t think it has to be a problem, but it’s often made a problem because of the objectives of each of the players. And the researchers’ objectives are often related to writing a thesis or some articles that are going to be published and they have to be rigorously defended. And the less control the researcher has of the study, the less confident they feel they’re going to achieve something they can defend.

This often leads people to try and exact as much control as they can. It can be very useful for the farmers to have control, so things are uniform between different farmers, and it can also be very limited to have control. I would like to get to the point where the researchers are less worried by losing control, so anything they don’t control, they must still understand what’s going on.

[00:04:13] Lucie: And because also, in losing control, as you say, they are gaining in terms of working with the farmers, they’re gaining in actual applicability of their research, that if there are interesting things coming out of it, then they will directly have people who are interested and have tested it and are, there to try it and actually put it into practice.

[00:04:32] Roger: Yes. And for you as a social scientist, this is difficult for you to understand because if you almost lose total control, then you don’t have any treatments and you just ask the farmers what they’re doing. That’s what we’d call a survey. So I want to give people confidence that the on-farm studies are often a combination between a survey and a trial.

So what you are losing control over becomes more the survey part. And surveys can be just as rigorously undertaken and defended as trials. Therefore, they shouldn’t worry about this loss of control, but they should recognise that many on-farm studies have a survey component and an experimental component, and they should try and do both of them to the best of their ability.

[00:05:23] Lucie: I find this really interesting too, the idea of this sort of survey, the added observations, the added questions that you want to be monitoring, I find it really interesting. There’s so much added data and information that you can collect and try to understand in doing a trial on farm. And in thinking of it, not just in terms of the actual trial on the plot in the fields, but in thinking of it in terms of all of the additional information about how the farmer manages his field, how the household interacts with it. There’s so much sort of side information that you can think about.

[00:05:58] Roger: Yes. And if you like, we can take it one step further. There’s a parallel concern that many people have. People who are used to doing surveys are used to having total control of the questions. So they dictate the questions just as an experimenter usually dictates the treatments. And there are many social scientists that say, that is rather limited, and shouldn’t you allow the farmer to discuss their own agenda rather than just respond to the often limited questions? And so there’s a sort of parallel between who dictates the treatment and who dictates the questions.

[00:06:40] Lucie: Absolutely. And I’ve heard researchers say that they don’t want to ask open questions because it makes it difficult to analyse, but it isn’t difficult to analyse. You know, with computers nowadays, it makes the trial plots easier. But if you ask a good social scientist, they will have no problems analysing the open questions.

[00:06:57] Roger: And remember as well that this doesn’t have to be a competition between open questions and closed questions, you can ask both. And this is just the same as in your on-farm trial, you don’t have to necessarily allow the farmer to choose all the treatments. Maybe you could have an interesting design where you choose some and the farmer chooses others.

Similarly, in a survey, you can have the questions you want and also have some open-ended questions as well. Life is fun as long as you don’t limit yourself by unreasonable attempts at rigour when you don’t need to.

[00:07:37] Lucie: Exactly, ’cause rigour often isn’t actually rigorous, it’s just trying to squeeze things into a box which can’t be squeezed into that box. So you actually, in a way, lose the rigour that you are looking for because of the quality of the data just goes out of the picture.

[00:07:52] Roger: Yes. And if I take your thing further, I find many people try and squeeze their studies into what I’ll call an easy to analyse box. That’s because they haven’t realised that with the computer doing the analysis, things that are not easy to analyse by hand could be very easy to analyse when you have a computer.

And I think that leads us nicely to Tricot, which is an interesting design. And this isn’t an advert for Tricot though I think Tricot is extremely interesting. Some of the ideas behind Tricot are useful both within Tricot and are more generally useful.

[00:08:31] Lucie: Okay, tell us more. So what is Tricot?

[00:08:34] Roger: Well, Tricot is the first example I’ve had recently of an agricultural experiment, which is very similar to a common medical trial. And I find that interesting because if you go back in statistics, it’s agricultural applications that really started off a lot of these ideas of design. And not so relatively recently, I find that medical research, in particular drug trials, has taken over as being exemplary in terms of the way in which designs are trialed and then the data are analysed. Therefore finding that Tricot is very similar to a medical trial has been very interesting for me. So let me give you a simple medical trial first, and then we’ll look at Tricot.

A simple medical trial, you have a particular disease or illness and you have a new drug that you think is going to help. So that is one treatment, your new drug. You are going to compare it now against the best available treatment, which is the standard. So you’ve got new and standard and you have a third one, which is not having a drug, which is a placebo.

And so there is potential medical study with three treatments, the new one, the standard one, and the placebo. And now you recruit your volunteers, and you give them one of the three treatments. Usually it’s what’s called double blind, where the patient doesn’t know which treatment they’re getting and neither does the person administering. And that for our case is a detail here.

But obviously, you only get a little bit of information from each patient, so you’re going to need quite a lot of patients. You also collect information on the patients because you don’t assume that your drug will work equally well for all patients, it might work better for one gender, it might work better for older people and so on. So you’d actually do a little bit of a survey of the people who receive the treatment, there’s our survey component, and then each one would have one of these three treatments.

And then you get a lot of data, and you do the analysis often quite quickly and you report the results. There’s a typical medical trial.

[00:11:06] Lucie: Yep.

[00:11:06] Roger: Now Tricot, it’s usually three varieties of a crop. Let’s consider maize as a popular crop, or sweet corn as sometimes it’s called. And there’s lots of possible varieties of maize or sweet corn. You have a lot of farmers in your study, and each farmer is going to have three of those varieties. That’s why it’s called Tricot, try for the three. So each farmer has three varieties.

But there’s some novelty in this. And one of the bits of novelty in Tricot is that usually there are more than three varieties in the trial. For example, a recent one I looked at had 16 varieties of maize, each farmer at random had just three of those varieties. Similarly to the medical study, for the first time I’ve heard of sensibly in agriculture, and it’s not essential to have this, but in many instances of Tricot, it’s also a double blind.

So the varieties are known to the farmer and to the person giving the treatments to the farmer as A, B, and C.

[00:12:18] Lucie: Okay, wow.

[00:12:18] Roger: But they are three chosen at random from the 16.

[00:12:23] Lucie: Because otherwise, you know, what people’s heard previously or what they’ve already experienced of a particular variety might influence their results, you know. There may be two fields that have to look the same, but because they have a preference, an unconscious preference for one, then they may say, oh yeah, this one’s way better.

[00:12:39] Roger: That’s right. Now there’s a big difference between this variety trial and the medical trial. And that is, attractively, we are learning more from each farmer because the farmer has all three varieties. There’s only three out of 16. The novelty is the medical trial only has the three treatments. The variety trial has 16 treatments, but at least the farmer with three has three of the 16 within the farm.

[00:13:10] Lucie: So you can understand what influence the farmer is having on those three trials.

[00:13:15] Roger: That’s what we call a within farmer comparison. And the variety trials each respondent has just one of those treatments, so this is quite nice, but it’s three from 16. So each farmer will have three plots, you will also collect information about the farmer, that’s the survey element, often how old they are, when did they plant. And they then have their three varieties and they report and take measurements on those three varieties out of the 16.

Maybe I should also add, like the medical trial, this doesn’t work if you only have a few farmers. You need hundreds and maybe thousands of farmers to get the information because you are only learning about three varieties and there are 16 in your study.

[00:14:04] Lucie: Yeah.

[00:14:04] Roger: That’s something different from the medical, and it seems to be working very well for this Tricot. This is called an example of an incomplete block design. I’m sorry to say it, but the farmers are called the blocks.

[00:14:19] Lucie: Yes, exactly. So it’s incomplete because not each farmer has the 16, none of the farmers have 16 varieties.

[00:14:26] Roger: That’s correct. So none of the farmers have all 16, they just have three, and those are different for each of the farmers. But when you analyse the experiment as a whole, from say, 500 or a thousand farmers, you are learning a lot about all 16 varieties. And the analysis is quite easy to do, and in fact, the software that’s distributed with it makes it even easier, and so you don’t need to feel concerned.

You’ve got a lot of support from the Tricot people to do a good design and you then have support again from them, so the data can be easily analysed. They do emphasise as well, which I really like, the importance of the analysis being for two separate groups, you’ve got to report back to the farmers. Otherwise they’re not going to remain involved. So the reporting back to the farmers at the end of the experiment, when you’ve analysed the data, the smallest report you give is to tell them which varieties each farmer actually has and how their results compared with other farmers.

[00:15:33] Lucie: Well, I would say that it’s not only reporting back at the end, it’s also having that conversation as it goes to sort of help analyse the fields with the farmers to see what they’re thinking and how they’re experiencing the study as well.

[00:15:46] Roger: You are pinpointing an area which I don’t know in some Tricot studies how well it is done. Remember the researchers are often focused on getting the results for papers and articles and recommendations. And they are very committed to the farmers being involved, but how much they’ve taken that step further that you’ve identified that this farmers being involved doesn’t have to be just at the end.

They’re obviously involved at the beginning, but if instead of being reported to at the end, they’re reported to in the middle, then you can at the same time collect more data from them of their reactions and their thinking. And that makes it an important part, a component of the study, which I think could be even more exciting.

And I have to say, I’m not sure how often that is done, and I must ask the Tricot team whether that is seen as an important component, namely reporting back in the middle, and at the same time collecting further information as you report back. Very interesting. So that’s Tricot.

Now, what I want to do is generalise from Tricot, because on-farm experiments were not always as imaginative as Tricot. They were often on too small a scale, and then they got into trouble because the farmers were very different. And then if they’re on too small a scale, you don’t learn as much.

[00:17:17] Lucie: Because the farmers are so different that you can’t tell where the variability is coming from, then.

[00:17:23] Roger: Exactly. So you know there’s a lot of variability, you know there’s a lot of reasons for the variability, and first some studies didn’t measure the reasons for the variability, which could be, let’s say, different dates of planting or different climate at different places.

[00:17:39] Lucie: Or different farmers’ experience in doing research.

[00:17:42] Roger: Yes. To some extent you gain by measuring these things, but then you find that the sample size of being 20 or 30 farmers is just too small to learn very much. You just learn that many farmers are very different one to another. And so you really have to consider the size, and this is a concept which I think McKnight is concerned with, and I love it, it’s Options by Context.

[00:18:13] Lucie: Yeah.

[00:18:14] Roger: In terms of the results, like the drug trial, you are not assuming they will necessarily apply to everyone. So you’re not going to aim to have the same recommendation for every farmer; thinking that there could be farmers who have different options and different solutions could be those options, but the context of the farmer might dictate which of those options they would like to adopt.

Farmers who plant early might like to adopt a certain option compared to farmers who are more relaxed and want to plant later when there’s less risk of having problems. So there’s all sorts of reasons why you might not have a one size fits all solution. And therefore you need, like in many surveys, you need quite a large sample size. You therefore have to find a way with a limited budget of how to cope with that.

Learning how to cope with that is in itself a good exercise. And that means we’re back to something we mentioned earlier of the importance often of pilot studies. You must make sure that your pilot study isn’t your substitute for the full study because it is going to lead into something in the next year. It is just a pilot, but sometimes the pilot study, you can learn a lot.

One of the things you often learn in a pilot study is, for example, that you are asking too many questions and the farmers are losing interest, and therefore you can’t rely on the data, not because of the farmers, but because you haven’t designed a very good questionnaire or data collection tool.

[00:19:52] Lucie: Absolutely. Yeah.

[00:19:54] Roger: And the same thing goes along, the measurements, you’re taking too many and therefore it’s becoming too expensive. A particular example of this, and this goes along with the type one, type two, type three, in on-station experiments, you almost always measure yield, of course. In, on-farm experiments it becomes very expensive to measure yield, and do you gain a lot out of measuring yield?

For example, asking the farmers, could they say what the yield was on a five point scale from a very good yield to a very poor yield. Do you actually need anything more than that in terms of the analysis? This goes along by the way with the Tricot, that the Tricot does not ask essentially for people to measure things, it does just ask which are the plots out of the three that the farmer thought was the best and which was the worst, and claims there’s a lot that can be gained from this more categorical data that you get rather than more detailed measurement data.

[00:20:55] Lucie: Yeah, because, as you say, farm environments are different. So what is the recommended average or something for that sort of farmer doesn’t mean that the farmer who chooses that variety will actually get that yield. Whereas if they know that’s the preferred variety for people like him or like her, then that can be just as convincing.

[00:21:17] Roger: Yes. One of the things that I think is to be queried is how complicated should the study be on farms? So Tricot is just three plots. And maybe that’s a good number because you want to concentrate on having a large number of farmers. They are now, by the way, considering that you could improve on a Tricot experiment by having a local control as well.

And so it goes from three plots and you compare those three, each one of them with the local. And so you would actually have four plots in your study. So they’re going upwards a little. And in general the more information you can collect within a study, the more helpful it will be, just like my on-station where I was trying to add factors. Each treatment comparison answers another question.

And so you are adding to those. So useful to do that, with the price being paid, the more complicated it gets, there comes a point where you lose interest of some farmers or it’s too complicated for them. So the number of plots and treatments per farm is something to be thought of very carefully. Three to me is really rather low, but it adds tremendous simplicity because three is a nice small number and as I say, the three has now become four. In general, I wonder about collecting more information.

And I’d like to finish with a design question, if I may. An experimental design where on station I would fight tooth and nail against it, and I mistakenly tried to do the same and then found on farm it’s a really rather attractive design. Now I mentioned in a previous thing a split plot design. There’s something even more complicated, which is a strip plot design.

[00:23:08] Lucie: Okay. Things in lines. Yeah.

[00:23:11] Roger: Things in lines. Exactly. So a strip plot design, supposing you have three varieties of maize and three levels of fertility. And you have not fertile, middle fertile, all the fertiliser needed, is three levels, and you also have the three varieties. So this is what I call a factorial treatment structure with nine treatments. So what could be simpler?

On station, I would argue that’s too simple, and often you could have more than nine treatments. On farm, I would be querying whether we could go down to six. But I’m still liking the idea that I’ve got two factors on the farm. So now I was thinking about the design question and I was then put in my place. So a strip plot design would mean that the varieties go one way and the fertiliser level goes the other. So varieties go across and fertilisers go down. Can you picture that?

[00:24:15] Lucie: Yeah. Okay. And what’s the aim of that?

[00:24:19] Roger: The aim of that, basically, in technical terms, your varieties go across, your fertiliser goes down, and so the interaction is the only thing on the little plots, which is how the varieties do differently at different levels of fertiliser. Please don’t do a strip plot design if it’s on station because you’ve got three plot sizes. You’ve got long thin ones, you’ve got long ones the other way, and then you’ve got little ones.

And then what put me in my place is how you’re going to take the measurements. And the researcher who was keen on doing them said they like it because the farmers stand one way on and they give their views on the varieties. And then they move 90 degrees and then they give the views on the fertiliser.

So the whole question was dictated by the fact that instead of having nine small plots, which is rather complicated, three plots going one way helped the farmer to be able to answer very simple variety questions. And they move along from one variety to the other, which one do they like? And they answer questions about that. And then they move 90 degrees, and they ask questions about the fertiliser levels.

That seemed very nice in terms of recording these measurements and the farmer’s views on both the variety and the fertiliser. So there’s a design, which as I’ve said, I wouldn’t like at all on station. I can’t picture why one would ever need that. And on farm, it seems to me quite neat.

[00:25:59] Lucie: So that’s a really nice example of where the researcher is designing research to involve the farmers and to really get them thinking as researchers, because it’s laying out an experiment so that the farmers can really, in a very simple way, as you just said, just by looking and by comparing the three varieties that they have, three plots, they can make comparisons themselves.

And even, during the time of the experiment itself, the farmers are thinking about that and reflecting on it, not just at the end once there’s a yield, but as the plants are growing and everything. So I find that a really fascinating example.

[00:26:37] Roger: Again, one’s got to think about the data collection, but you are absolutely right that I think that the way on-farm trials have moved, has been extremely attractive. But the next level of innovation is to emphasise, as you say, just how much you can learn from the farmers, which is one of the reasons you’re there, and this is an example that maybe takes that exploitation a little bit further.

And it’s the contrast to spending all your time worrying about the variability. Instead, let’s worry about the advantages of being on farm in terms of the amount that can be learned that’s really rather important.

[00:27:22] Lucie: Absolutely.

Thank you very much, Roger, it’s been a wonderful conversation.