Description
As part of their continuing conversations around research methods for agriculture, Lucie and Roger discuss the utility of conducting research trials over multiple years. They consider the importance of understanding climate variability, and the value of pilot experiments as well as the implications of adapting research methods based on initial findings and the benefits of engaging farmers extensively in the research process.
[00:00:06] Lucie: Hi, and welcome to the IDEMS Podcast. My name’s Lucie Hazelgrove Planel. I’m a Social Impact Scientist and anthropologist, and I’m here today with Roger Stern, one of the experts in research methods support, who was doing research methods support well before David and I joined that program, and you’re taking the time to discuss some of your learnings with me. Thank you so much for being here and welcome.
[00:00:28] Roger: Thank you.
[00:00:29] Lucie: You asked a question a little while ago, which wasn’t a question for me. It was a question in general for how researchers think about their trials and their experiments. And that was, should you do one year or more? And I think quite a few of the researchers we work with, they have grants which last several years. So I think this is a really interesting question to work on. Perhaps could you start off by saying a few words of, you know, why ask this question? Why would you, why does it even come up?
[00:00:59] Roger: Well, one reason that this often comes up is with traditional experiments. Traditional experiments to me are on station trials, for instance, and in on station trials there is either truth or a myth that journals like you to analyse on station trials for more than one year. So you send in information about your trial, which was a single year, and I don’t know whether it’s true or not, that journals will reject it because experiments should be repeated over two or three years.
And I always found this was rather odd, because when you dug into why they should be repeated, it was usually because the rainfall could well be very different in a second year, and it was good to have a sample of years, so to speak, and therefore worth repeating.
[00:01:56] Lucie: Okay. That’s interesting. And we know both that, well, in West Africa where the community of practice that we work with is based, the rainfall patterns certainly change from year to year. There’s huge amount of variability within that.
[00:02:09] Roger: Yes. The thing I found curious about this was the idea that if one year was not enough, then two years was enough, because usually the implication is, for example, that a dry year, your treatments might behave very differently to a rather wet year. However, if you just have two years, then they may both be dry years.
Now, there’s lots of corollaries for this. One of them is that it would be good generally to understand the background to an experiment. So in the methods section, you would normally report on the general conditions of an experiment, and I would like to stress that it’s rather useful to record some impression of the climate in that year. Was it a very stressful year? It may even be the farmer’s impressions of the climate, did they find on their fields that this was a rather difficult year or this was an easy year?
[00:03:10] Lucie: That’s really interesting. So describing the context of that year in terms of its season, in terms of its climate, and not only in terms of what the Met Office perhaps said, but also what the farmers experienced themselves, or their perspectives what they experienced.
[00:03:25] Roger: The general discussion, which is to be reported in the method section, namely the things that you don’t treat, that aren’t involved in the treatments, you still report on. So in an experiment, you would report on how you planted the crop, was it on the flat or was it on ridges? How large were your plots? Did you apply fertiliser? Was it very fertile or not? So the general background is also included, and I’m just arguing that that should include some impression of the climate as well, just quite naturally.
[00:04:00] Lucie: Yeah.
[00:04:01] Roger: Now, I’m leading to the fact that I don’t really feel that two years is the magic formula when one year isn’t enough. I’m quite happy that experiments are repeated over two years. If they are, then there’s always the interesting question to me, after the first year, do you repeat exactly the same experiment? And I find that sometimes the researcher wanted to keep it exactly the same, even if they’d now thought of some new ideas, because they thought the journal would like two things exactly the same.
I hope that is untrue because it must be better to learn from the first year and maybe apply something slightly different in the second year. And this would apply equally whether the study is on farm or on station.
[00:04:50] Lucie: That’s really interesting. Okay, so, there is potential to change within the same trial, within the same research, if you’re going to do a second year of the research, of the trials, on farm or in a field, then you can actually adapt from the learnings of the first year.
[00:05:06] Roger: Yes. And this brings in some other issues. If you are repeating over two years, do you apply to the same plots? For example, in an on-farm experiment, do you go back to the same farmers, or do you use different plots in the second year? And very often you have no choice but to use different plots because the crops sensibly would be in a rotation.
[00:05:29] Lucie: This then gives loads more complication. ’cause I have seen this in data where the previous crop has created a huge difference in terms of how the crop that they’re actually studying has grown. So something, you know, nitrogen rich, it’s going to really help that crop that you’re researching.
[00:05:48] Roger: And often you would not have the same crop in the second year. And so you might have two years of an experiment because you’re studying two years of a rotation. That to me would be different. That would be a single experiment, which naturally is going over two years.
[00:06:03] Lucie: Yes. That’s interesting. Yeah, of course.
[00:06:04] Roger: So this would be rather different.
I also would like to give the idea that if this was a survey, then it’s really quite common that the first time you run a survey, you might run a little pilot survey ahead of time.
[00:06:19] Lucie: Yeah, that’s interesting.
[00:06:21] Roger: And you don’t tend to run pilot experiments. Now, there’s no reason why not, except the idea is that the pilot survey is quickly and easily done. Whereas often an experiment might take a whole year, and so you can’t afford it, it seems inefficient to try and do something on a small scale. But there are instances, I’m sure, in your research where it would be quite interesting to do a smaller experiment in the first year and treat it sort of as a pilot.
[00:06:54] Lucie: Yeah. A lot of the researchers we work with are really pushing the boundaries on the sort of current research methods or the sort of traditional research methods in their respective disciplines. And so it does make a lot of sense to pilot them first and see with few farmers, how do they respond to this, how did the different variables that they’re testing, can we get enough data from those and, you know, are we testing them in the right sorts of regions and in the right sorts of conditions? That’s an interesting idea.
[00:07:22] Roger: There’s an interesting difference though in that sense, a normal pilot survey is often to test the questionnaire. Are you asking the questions in the right way? And there isn’t a thought that you would then change things radically. You would just refine the questions.
Whereas, piloting an experiment with a few farmers over one year, you might then decide if you were sequentially doing this, that one of the objectives of this first year, and in general this would be the case, is do I go on and do something similar for the second year or maybe identical? So I’m very keen that, if you do do things for two or more years, that part of the decision is taking place after the first year of what should be retained, and it could be everything, and what could usefully be changed?
Another issue that comes up as soon as you repeat something in on farm experiments is do you go back to the same farmers in the second year? And if you do, do you try and discuss with the farmers things about change. This is classically what’s called a paired T-test, that you are looking for differences within a farmer in the first year and the second year.
[00:08:40] Lucie: Okay if you’re doing the two years with the same farmer, then…
[00:08:43] Roger: If you are doing the two years with the same farmer, are you taking sufficient advantage of the fact you are going back to the same farmers? And that again would imply new and different questions in the second year.
[00:08:56] Lucie: You just mentioned, you know, something about change and discussing with that farmer, change. With some of the people I work with, they’re promoting agroecology at the same time, you know, the researchers are promoting agroecology. And so, listening to you, I’m thinking that the researchers are interested to know if the farmer’s not only doing research, but also actually getting really interested in some of these new techniques, perhaps that they’re trying, and adapting them in different ways or sort of adopting other practices that they’ve heard of. Is that the sort of change that you’re talking about?
[00:09:30] Roger: I think so. I think this is where you might plan a two year study in that way. And that would be very valuable. And that’s very different to just repeating the same thing blindly over two years just because you think the journal needs it. Planning a study over two years to measure change could be fascinating in many studies and would be relatively common I would think, in a project that is a multi-year project.
[00:09:54] Lucie: Well, I think a lot of people think of it as their monitoring and evaluation. They don’t think of it so much as integral to the actual project of identifying that change and understanding, well, which changes have been happening across the board and which changes have not been happening perhaps.
[00:10:11] Roger: That’s right.
I’d just like to come back briefly to the climate, because as I say, it’s often used as the overriding reason. And then is incorporated in the research by people doing something for two years and thinking, okay, I don’t have to worry about the climate anymore because I did it for two years. And I feel that I would like to understand, and in some cases a very useful objective, the effects of climate in much more detail.
And I would like to mention that in a different podcast, but it leads to me quite naturally into the idea of crop simulation experiments because the minute you start thinking of climate variation, then I have to say that two years is very little. Studying the effects of climate, you really want to have many more than two years, and the traditional experiment may not work very well. Therefore, is there a tool that can enable you to consider the climate variation of a standard trial over many years?
This would not be an on farm trial so much, but it could be a crop simulation trial where instead of growing the crop in the field, you grow the crop in the computer. And I think that’s something we should discuss briefly because it’s a very useful research technique, which is used a lot in some countries, but not used in many African countries.
So there’s a pointer that the standard reason of repeating trials, which often mentions climate, I would like to see climate mentioned, it’s one of the subjects that interests me greatly. But I come back to the point that we started, that if you’re thinking that climate is a driver here of the variability, then two years is a very small sample of years.
[00:12:01] Lucie: So someone else mentioned to me that if you are looking at only two years, then it’s more, perhaps you can only understand the reaction in terms of weather, you’re only seeing weather as opposed to the whole climate. Climate is more about this sort of longer span, historic span of sort of decades.
[00:12:20] Roger: Yes, except that that is implicitly bringing in the idea of climate change, which is important because it’s one of the key drivers of a lot of the research now. But I think whether or not there’s climate change, the fact that the climate is extremely different from the crop’s point of view from one year to another means that a lot of the variability of your results has come about because of the variability of the climate, and it’s often ignored. And therefore maybe it’s worth considering, and one can consider, well, what are the tools to bring in climate into these studies? And going from one year to two years is a really rather poor way of doing this.
[00:13:04] Lucie: I’ve just been thinking while listening, what anthropology and ethnography is when you spend a long time in a community, how they deal with this question of one year or more. And there I think, in the sort of methodology, in the context section at the beginning of any writeup, it’s always situated. The study is always situated in a time and in a space. And it’s very important to stipulate and sort of be clear that someone is only writing about that particular time. And actually by the time you have started writing about it, the community or the event or whatever you’re writing about may have completely changed.
[00:13:41] Roger: I think that’s right. But I think in anthropology you have some other opportunities, which of course you’ve got to take advantage of. So one of them is you are looking in much more detail in an anthropological study, and that gives you an opportunity to discuss with farmers this year in relation to many other years that the farmer has experience of. So how does this year sit?
And so the discussions of the farmers can bring in the same sort of history that the crop simulation models will bring in. So I think there’s opportunities with imaginative studies with farmers, and you can do a little bit of that, of course, in, you know, the fact there’s an on-farm trial is also a survey, and you can bring this into the surveys.
But I think the anthropologist has a much richer opportunity to discuss past climate from the farmer’s point of view for the last 10, 15, 20 years, depending on the experience of the farmer, if that becomes an objective of the study.
[00:14:42] Lucie: So similarly, on another point that we were just discussing in terms of, you know, adapting and learning between years or between trials. In more social sciences, for example, if you’re doing an interview, or if you’re doing a series of interviews, you would notice that some of your questions, you know, you mentioned this in terms of pilot surveys, but no interviewer would ask the same questions repeatedly if they saw it’s just not working when they ask it to different people.
There’s that process of adaptation, which is inbuilt in the methods, in the sense, because you’re doing the same but with different people and it’s over a different period of time, or it’s over a more stretched out period of time rather than simultaneous often.
[00:15:22] Roger: Yes. But let me give you one concrete example. If you are doing the little bit of survey with your on-farm study, one of the questions you might ask is when was the planting date, that’s often recorded. You might then observe that one farmer planted rather late. And it’s then interesting to follow up with the farmers that planted late, do they always consider that they plant relatively late, and are there reasons for this?
For some farmers, the reasons are that they think there’s more security if they plant late. But for others it would be maybe that they have a job in the dry season and therefore they don’t arrive to plant early, and so they don’t have that opportunity. This can be really rather important in studying the effectiveness of, and the reasons for variability between one farmer and another.
[00:16:14] Lucie: And often these sorts of questions, you can go back to the farmers and ask them even at a later date, even if you didn’t think to do it immediately, especially if you have students helping with the research or something, then it’s not a concern to go back and actually check and try and find out more about these reasons.
[00:16:30] Roger: You are absolutely right. And I wonder in the studies that we are supporting in this way, the extent to which it’s quite expensive to collect survey data always, and you have to visit the farmers. But in an on-farm trial, you have the crop in the field for the whole year. And so taking the opportunity not to have one big survey, but to go to the farmer on a number of occasions through the season and ask different questions as the time goes on may be a more effective way of conducting that trial.
[00:17:04] Lucie: Effective in the sense of you understanding what’s going on, but also effective in the collaboration process of actually working with farmers and understanding and having that knowledge sharing process of involving them really in the research.
[00:17:19] Roger: And you’ve hit the last point very, very well. I believe it’s not just doing on-farm trials, but the more the farmer is genuinely involved in the research process so that they are as keen as the researcher to understand the results, the more effective your research will be. And this is reinforced if you are willing to take the effort to go to the farmer on more than one occasion.
[00:17:43] Lucie: Absolutely. Well, thank very much, Roger, it’s been another interesting conversation and lots to learn.
[00:17:50] Roger: Thank you very much, Lucie, always a pleasure.