176 – Twenty Years of RMS for CRFS: Multimethod Agricultural Research

Lucie: [00:00:00] Hi, and welcome to the IDEMS podcast. My name is Lucie Hazelgrove Planel, I’m a Social Impact Scientist and anthropologist, and I’m very pleased to be here again with Roger Stern, expert in research methods for agriculture. Hi Roger. Thanks for joining me today.

Roger: I don’t like the word expert, but yes, hello and nice to be back with you.

Lucie: For someone who’s been doing it for your whole career. 

Roger: Yes. Okay.

Lucie: I’m not sure how else to describe somebody who is very knowledgeable about it.

Roger: Okay. 

Lucie: So we are going to be discussing how to sort of get started in agricultural trials or how to get started in planning agricultural research, and give a bit of an overview and introduction about really what to think about when designing agricultural research, and I’m gonna keep it broad like that by saying just agricultural research, as opposed to something specific. 

Roger: And I think in relation to the McKnight project that most of the [00:01:00] projects early on, you’d have an inception meeting. And I went to quite a lot of those inception meetings where the project team was discussing with the research support team and the, I don’t know what to call Bettina in this. 

Lucie: It’s the regional lead and the regional team. So there’s Bettina who is the lead scientist, there’s Batamaka and Oumar who are regional leads, I think. 

Roger: So the first thing that I found was that the different members of the team each had a different data collection method, which they’d been taught. 

Lucie: Within a project. 

Roger: Within a project.

Lucie: Within a project, often there’s different people coming from different domains. Absolutely.

And as part of our role, we really, we are very fortunate to be involved at those early stages of research when people are still at the planning stages, still deciding what to do and how to do it. And they’re very fortunate to have access [00:02:00] to experts who, and by experts I really mean Bettina and Batamaka and the rest of the others, to voice, to have a sounding board, to reflect on what’s the best way of doing something to get good research. 

Roger: Yes. And one of the dilemmas was as, as I say, coming back to my feeling that if you were an agronomist, your statistics training had been on the design and analysis, or really the analysis of agricultural field trials, usually on station trials. That’s what had been in your statistics training. Whereas if you were a socio-economist you knew all about doing a survey. And if you were a broader social scientist you knew how to criticise standard surveys and do a data collection exercise that was more participatory.

And I found the interesting component was that we wanted to encourage the project to work as a team, and therefore each member of the [00:03:00] project would often get involved in more than one data collection exercise within the project. And that would include collecting data in a way that they hadn’t been taught, and therefore were less comfortable with. And therefore I found we often had to outline what are the different possibilities for collecting data at the beginning of a project, that looked at different aspects. That’s what I thought we’re talking about today.

What’s the range of activities you could envision? And then getting to the point that often we found that scientists would want to do one of them, whereas doing smaller studies, but more than one was often going to be more productive or seemed often to be more productive. 

Lucie: And there’s often researchers, you know, they tend to think that the method that they know is going to be the best method, and they really want to encourage that one rather than being open to, as you say, combining methods and [00:04:00] seeing which method, what value each method can bring. 

Roger: That’s right.

Lucie: And the value is often it’s better together than just one thing separate, there’s a combined value, which is better.

Roger: So if I go through the list that that I have and that we’ve mentioned before, when I moved out to work in Niger in agricultural research full time, it was still in the era where the standard method for agronomists and breeders to collect data was an on-farm trial. And this could be a randomised block experiment or a split plot. These terms will be familiar to people who’ve been involved in that.

And round about that period was the beginning of the move to thinking that the problem with on-farm trials is that farmers don’t live on research stations. Sorry. The problem with on station trials is the farmers don’t live there. 

Lucie: [00:05:00] Exactly. They will have different types of farming environments and ways of farming even.

Roger: And there’s a whole legion of ways. Usually the climate might be quite similar where they live, but the soil could be different and also the farming practices would be different. And so the move got serious while I was working for ICRISAT to move to on-farm trials.

And that was a big movement that has continued and I’m, I’ve been very happy with the continuation of that. Of course, whenever you do a sort of revolution like that, you can go too far. And that I think is part of my point that I sometimes found that people thought that these were alternatives. Should I do for my research, for this project, and on station trial, which I’m comfortable with because I’ve been taught that, or should I do an non-farm trial where I need to understand more about what it is?

But the assumption was you only did [00:06:00] one. And we almost always found that they each had advantages and problems. Therefore instead of thinking of one humongous exercise to which you add lots of components, you could do two or three different data collection exercises that complimented each other. That could mean that instead of a very complicated on station trial on a large scale, which is very expensive, you did a smaller trial and you had something on farm and something else on station. 

Lucie: This is really interesting, Roger. Can you give an example of how someone might want to split their research between on station and working with farmers?

Roger: Let me first tell you a bit more about the on-farm trials to see what the options are. Because even within on-farm trials, I think the dilemma was how to do a good on-farm trial. I thought ICRAF typified on farm trials. So I’m going to spend a little [00:07:00] time explaining those, but I’m going to talk about type one, type two, type three. And please remember as I talk about this, that one and two and three are whole numbers, but often you could think of type one and a half or two and a bit.

A typical type one trial was where you went on farm and you hired the farmer’s field and you did an on station trial on the farmer’s fields. And that solved one problem, the soils were genuine farm soils because you were on farmer’s fields. And you might do this in four villages, so you choose four separate farmers, one in each village, and you basically hire their field and you do an on station trial, whatever design you want, with full replication or a split plot design, whatever you like on their farms. And that is with a type one.

So the key components are the treatments. In a [00:08:00] trial you have treatments that you allocate to plots. And the treatments are designed by you, the researcher, and then you manage when you plant, how you fertilise, how you space things out, when you weed, how you harvest. The management was also done by the researchers’ team. So the one difference is instead of doing your on station trial on the station, you basically did on station trials on farmer’s fields. 

Lucie: And there, the researcher’s designing it, the researcher is managing it. 

Roger: Exactly. The way you phrase that takes us neatly to type two. Type two is where it is designed by the researcher and managed by the farmers. 

Lucie: So they do all the work.

Roger: They do all the work. But you decide what treatments they’re going to do. So if this is a variety trial, then you [00:09:00] specify which varieties are going to be on each field. And often you would find that it’s the same set of varieties, although it doesn’t have to be, for each farmer. So treatments are your concern and the management of the study is the farmer’s concern.

Lucie: I can see that that’s already bringing in some variability because farmers will be managing their farm in different ways.

Roger: That’s correct. So immediately, one of the early problems that researchers had is that the farmers managed in such different ways that it seemed that even if you had, let’s say, 20 farmers, it was like having 20 separate trials and you couldn’t combine them. And you really wanted to combine them because usually you did a single replicate.

If you had six treatments, like six varieties, then each farmer would just have six plots. [00:10:00] So the replication would come from having the 20 farmers each doing the same thing, and you have 20 replicates. But if the farmers are doing whatever they’re doing in very different ways, then it’s much more like having 20 very small experiments. And combining them was a nightmare.

Lucie: Yes, I can imagine.

Roger: And therefore people didn’t like this at all. But that was because they were missing a trick. Namely, what you don’t have as a treatment, you must have as a measurement. Experiments, trials, whether they’re on farm or on station, have three components.

There’s the treatments you apply, there’s the layout, and I could think very broadly of layout, which could include the data planting. And the layout in an on farm trial is the fact I’ve got 20 farmers in four different villages, let’s say. That becomes part of the layout. And everything that is not a direct objective, your [00:11:00] objectives might relate to getting the highest yield and maybe the nicest taste, so those become things you must measure. However, the things you can’t control, which I can consider as part of the layout of the study or the measurement, you must measure. The date of planting, you need to know what that is. 

Lucie: So all of the things which make differences between how farmers manage their fields, you need to measure those, is what you’re saying.

Roger: If you can’t control it, you must measure it. It is very simple. And in the early days people didn’t, they just complained that things were very variable, but they didn’t know the cause of the variability. That’s the whole idea of the measurements, that if you didn’t control it, you see it on station, you’d plant the whole trial on one day. On farm, different people plant in different ways and on different days.

Now you can’t control that, but you can measure it. So things change. And that became the fact that an on-farm [00:12:00] trial had certain things you would measure. And a lot of those were measured at the farm level rather than the plot level, because the farmer typically would plant the whole experiment on the same day.

So the date of planting would be measured at the farm level and maybe some other characteristics of the soil, was it sandy soil, could be recorded at the farm level. So you now have data at the farm level, a lot of it coming from measurements or from questions you ask the farmers. And then you have information at the plot level.

That’s already a nice change. And you can think of that in general terms, that what you are collecting at the farm level is like a survey where you interview the farmer and say, when did you plant? Did you weed? If so, how many times did you weed? Et cetera. So those are usually questions. They don’t have to be. The farmer might say, I weeded three of these plots twice, then it’s something [00:13:00] at the plot level. But usually they would say I weeded just once for the whole experiment, and I weeded so many days after planting. So then these are data at the farm level.

Lucie: And I find it really interesting , you call a lot of these things measurements, for me, measurements means more of centimeters, it means kilograms, but it’s also just monitoring and actually taking note of these important measures.

Roger: Yes. Anything that could contribute to the variability of the yields you want to understand.

Lucie: You mentioned then if you are doing one of ICRAF’s type two levels of on-farm trials where the researcher designs it, but then the farmer manages it, then you might want to not only have your agricultural trial data coming out of the crops that they’re doing an experiment on. But you might also want to find that data out, find complements to that data out through a survey with the farmers. 

Roger: That’s correct, [00:14:00] yes. 

Lucie: Which you would not be doing then in the type one where you’ve got researchers designing and managing all the fields. ’cause there the farmer doesn’t have much to do with, doesn’t have much to do with it, I imagine.

Roger: Sometimes there would still be, you had some exercises where you would collect the farmers together and go around and discuss the different experiments. But let me come to a type three because I think type three becomes even more interesting. And type three sounds very extreme. But that’s because a lot of studies usefully could become type two and a bit.

So type three, the farmers are involved in choosing both the treatments and the management. So they still continue with the management and they choose the treatments. And this frightened people a lot because they then said the farmers choose different treatments. And the answer is yes. And therefore, can I analyse the data?

And people are very worried about analysing [00:15:00] data, which is what you’d call unbalanced, that different trials have different treatments. But the analysis is very easy, so don’t worry about that. And that was part of these inception meetings to give the researchers confidence that if the design led to slightly more complication, that they could easily be dealt with.

But type two and a bit is where the researcher chooses some treatments and then says to the farmer, you could also choose a few treatments as well, if you would like. And this tied in often with having demonstration plots. Even while I was working many years ago there would often in villages be demonstration plots of new varieties, and sometimes there’d be new varieties and one row of them would be, let’s say 12 new varieties not fertilised. And then next to it was 12 new varieties [00:16:00] fertilised. So you had an idea of those varieties. 

Lucie: And so that would help the farmers to identify which crops they would want to experiment with.

Roger: Exactly. And so you might have particular interest in new varieties from the 12, three of them, or even other new varieties as your treatments. And then you ask the farmers, would you like to choose from those varieties as well? What I’d like to give confidence to the researchers is they then often got nervous and said, we’ll do this, but in a village, all farmers must choose the same. So we’ll have a meeting so they choose the same. And they had to choose the same number.

None of these are important. And I think there’s tremendous benefits.

Lucie: Yeah, that sounds a bit complicated if everyone has to choose the same.

Roger: Many people would say that’s simple because you just have to sort out, everybody’s got to agree at the end of the meeting. I would like to suggest it doesn’t matter. [00:17:00] Let’s take a trial with four varieties that the research chooses and the farmer is then asked if they would like to choose more varieties, either from the 12 or other varieties, and add those along as well.

But, remember this idea that everything you don’t control, you’ve got to measure. So now some farmers might choose one, some might choose three, and that means their little replicate would be different to the others. Not a problem. So some of them have four varieties you’ve chosen and three of their own, they have seven plots. Others might have five plots, and the varieties five, six and seven or just five are different, and that’s interesting and nice.

Now, one reason that you shouldn’t worry is that it’s easy to analyse, but the other reason is that you don’t just want to let the farmers choose. You want [00:18:00] to know why they chose that. What is it about those varieties that made them choose that? That’s continuing your survey component. 

Lucie: Exactly, it comes into the survey component of really understanding the reasons and the motivations behind, and it’s the whole point of including farmers and involving farmers in the study in the first place, to try and understand what interests them and what motivates them in terms of their choice of crops or in terms of choice of varieties. 

Roger: And just think carefully now about the fact that you would like the farmers to be part of your team effectively. So they should be interested. And understanding why they chose, but letting them choose is going to give them a lot more interest in managing their particular replicate. And there’s another interesting component of on farm that you don’t get on station.

Lucie: Can I come back to those three types of involvement, no, three types of on-farm trials you mentioned by ICRAF. [00:19:00] So we started off with researcher design, researcher managed and then went up to three where it’s farmer designed, farmer managed. And I’m wondering if, you mentioned at one point there being difficulties when people were involving only 20 farmers in an experiment. I think my question is, as you’re going up in these three types of level, I think there’s more variability, because there’s more sort of different things, more different factors are they called, which can influence the results. Should you be thinking about increasing the number of farmers you involve or the number of farmers you work with as you increase those or go up those three steps?

Roger: Not so much as you go up the three steps. I think even step two would benefit from thinking of the fact you’ve introduced variability for lots of different reasons. Therefore it’s going to be very helpful if you go up a scale. And that also is what you want to do with the results and the sort of results you expect. So 20 [00:20:00] farmers for a PhD sounds quite a lot, but if you think of 20 as a survey, a sample survey of 20 observations is rather little.

Lucie: It is. 

Roger: So remember there’s a big survey component. And, honestly, thinking of the survey in the hundreds is not unreasonable. Now, in my day, that wasn’t done because it was far too expensive. And I’d understand that some recent trials with McKnight, which have involved the farmers organisations have integrated this research study with the work that the farmers organisations want to do and the discoveries they want to make.

I found that very nice. But you are absolutely right. I think that you are aiming for much larger sample sizes. Remember that when you are aiming for very large sample sizes, your methodology could do with a testing. And so you might find that it’s very useful [00:21:00] to do a small study as a pilot. Surveys, it’s often extremely useful to have a pilot study as the beginning of things. And the pilot study could be 20 or 30 or 40. And then you are making the case for doing something on a larger scale.

Lucie: Yeah, that’s really interesting, of course. And that’s where also involving students in those pilot phases, ’cause their study, you know, that smaller study can be the pilot basically, and the larger project comes in to do the rest.

Roger: That’s right.

Just generally, let me just emphasise that I find that a pure type three where the farmer chooses all the treatments is often relatively rare. Whereas what’s very attractive is for the researcher to choose some and for the farmer to choose more. And I call that two and a bit. And then you can decide what proportion of the treatments are usually chosen by the [00:22:00] farmer and why, and what are chosen by the researcher, and also why, and that’s part of the initial objectives of the experiment, or this study.

I shouldn’t say experiment more because you are getting the idea that the discussions with the farmer are now becoming partly a survey and partly an experiment. And the distinction between a survey and an experiment is simply that the experiment component has treatments that are allocated, usually at random, to the plots. Whereas a survey, it results from questions that you ask. Now separately from that is that you sometimes wanted to do more in a more participatory way, and that often meant that in addition to your survey data collection, you might go back and do a more detailed discussion, which is much more open-ended.

Often that could become [00:23:00] a sort of village level meeting. I found many of these studies involved focus groups. So if you think about that with your on-farm study, a focus group could be a whole group within a village. Remember your on farm trial has got plots and it’s got fields. Now you’ve got a set of fields or a village as another component. And so usually some of these focus group discussions were at a higher level than the individual farmer, which is often the household level. 

Lucie: Great. Thank you so much, Roger. We’ve had quite an overview, an introduction of planning agricultural research there, and we’re going to go into more depth, I think, in some of the different aspects in our future conversations.

Roger: Sounds perfect. I was going to add about crop simulation studies and things like that, but let’s leave that for another occasion. 

Lucie: Yeah, [00:24:00] absolutely.