178 – Twenty Years of RMS for CRFS: On-Station Agricultural Trials

Lucie: [00:00:00] Hi, and welcome to the IDEMS Podcast. My name’s Lucie Hazelgrove Planel. I’m a Social Impact Scientist and anthropologist, and I’m very pleased to be here with Roger Stern, one of the first people to give research methods support to the West Africa Community of Practice, of what is now the Global Collaboration for Resilient Food Systems.

Hi, Roger.

Roger: Hello, Lucie. 

Lucie: So there’s a few things which are central to agricultural experiments, if I understand correctly. And we’ve discussed this trio of treatment, layouts, measurements, which I would love to dig into more with you. And then also try and understand, you are a statistician, you’re the person to ask about this, what does stats do, what’s stats role within all of this?

Roger: I’m going to give one or two examples. Let me just say that when I was working in West Africa, it was in the pre-revolutionary days of everything moving on farm. Therefore [00:01:00] most of the guidance that I was supposed to provide was to help researchers design and then analyse their on station trials.

If we take that to the present day with your McKnight research, I still think that in many studies, as we’ve mentioned previously, there’s often scope for an on-station trial. But an on-farm trial is probably the main activity in many of your studies or maybe a survey.

Lucie: Yeah, I don’t know. I don’t know on the balance overall.

Roger: I think the principles of designing a good on station, understanding those principles is really helpful, whether it’s an on-station or an on-farm.

Lucie: True. 

Roger: We thought it was worth just discussing these principles because I found that they were poorly understood.

Lucie: Exactly. Could we start off by, you know, what do the three, treatment, layout, and measurement, what do they [00:02:00] really mean? 

Roger: Let’s take the example that I’m going to use later on, which is a very simple example, which was a proposed experiment, nothing to do with McKnight. And I was in a discussion with this irrigation engineer who was designing the trial with the director of research and myself to discuss the trial before the season.

The trial had four treatment, no, three treatments and four replicates. And the three treatments were, it was in the dry season, quite a lot of water, a middling amount of water, and the minimal water. And there were to be 12 plots, so four replicates, and these were laid out in lines, and so these were called blocks. So this was a randomised block experiment with four blocks of size three. Then a particular variety of maize was used, and that was the [00:03:00] study.

So that’s an example of a trial. And so the three components were the treatment, which was how much water was applied, and it was applied to each plot, the layout was the fact it was laid out in blocks of size three, and then the measurements were various measurements of the crop, a little bit through the season, but also the yield. And so these were measured.

And that’s characteristic of every study. Just to repeat, if it’s a trial, it has treatments, which are allocated usually at random by the researcher, it has layout variables, in this case, it’s the blocks that you lay it out in blocks, you don’t just put the treatments haphazardly, and then you can’t learn anything unless you measure something, so the third component is the measurements. So that is a trial. And the question was, is that going to [00:04:00] be a useful trial and is there anything that can be done maybe to make it more useful? Am I okay so far?

Lucie: Yeah, sounds good. Yeah.

Roger: Okay. Now, have a think that this is a research trial. I found a problem with that design, and that is if it’s research, you need to discover something new. And I queried with the irrigation person whether there was anything new he would discover, because it seemed to me that even not knowing much about irrigation, I was fairly convinced that water is useful.

So I didn’t need another trial, I don’t think, to find that maize needs water. Therefore, probably you’re going to get better yields if you give a lot of water than if you give a middling amount, and you may not get much yield at all if you give minimal amount of water.

He then announced that he also knew that, which I was pleased because he’s an [00:05:00] irrigation engineer. So my question then was why was he doing this experiment? It was interesting that he had not thought of it quite that way. And this is where you need to have clear objectives, and the only objective I could think of for that experiment when you write down the research protocol was, is water useful? And I wasn’t clear how much more research we needed to say the answer is yes, it is useful.

So then the question is, do we abandon this experiment or could we turn it into something more useful?

Lucie: Exactly, I’m interested to know what happened, I guess in that example.

Roger: This is where we have something that I found, usually at a much more complicated level, I usually had people that came with something like 10 or 12 treatments and maybe 48 plots. And I was still making the same point to them. And this was: would you like to introduce another factor into the experiment? He has one factor, which [00:06:00] is amount of water and it has three levels. Is there anything else about maize that he might like to learn about at the same time? And he immediately said I could use different varieties. So we now queried whether he could have more plots and introduce varieties into this.

And we then came across an interesting feature of this particular experiment, and that was that in irrigation experiments, the plots are often quite big. So these 12 plots were each quite large, and if he did introduce, let’s say, four levels of varieties, so four different varieties, he could put those into small plots within each of the big irrigation plots.

Lucie: And they can go into the, you can have the different varieties within the bigger plot because it’s the irrigation itself, which needs… 

Roger: Which sort of needs often a large [00:07:00] plot if you’re going to dictate the irrigation. And so there’s plenty of scope for having a smaller plot inside. The question is what do you gain from that? And now he’s started thinking which varieties, and there’s an interesting question about varieties now. Do you just choose four varieties or do you choose varieties that have particular properties?

And that’s really rather different, just four varieties, because there’s hundreds of varieties of most crops, if you choose four just for the heck of it, then when somebody comes along with a fifth variety, you know nothing about it. But if you chose varieties for particular properties of the variety, such as the duration or the depth of the roots, then if a fifth variety comes along, you know something about it, you could say it was more similar to variety three than any of the others. Variety three did very well. So this fifth one might also.

Choosing the varieties is quite useful because there’s lots of them. Don’t choose varieties that are very similar because again, [00:08:00] you expect similar varieties to behave similarly. Choose varieties that are interestingly different, so you learn something. 

Lucie: That’s very interesting, yeah.

Roger: Now we have 12 treatments, four varieties for each of the three irrigation levels. And suddenly he brightened up because now something could happen and he could learn about the different varieties. But actually there’s a lot of variety trials, so you probably knew about the different varieties. So the new thing that would possibly be not known before is how the different varieties react to these different water regimes.

Lucie: Yeah, exactly.

Roger: That’s maybe the thing which not so many people know about, and that’s research. Now that’s called the interaction, namely, he’s not so interested in the variety effect because he knows about that. He’s not so interested in the water effect because he knows about that, but nobody yet knows [00:09:00] how these varieties react. So this is which varieties do better under different water regimes. Then you are planning for drier years or wetter years, and that could be quite useful.

One of the key components I found that I was doing, which was introducing students or researchers to the wonders of factorial treatment structure. So adding another factor was one of the things I questioned. The more factors you add up to a reasonable limit, because things are getting a bit more complicated, the more questions you can answer, therefore more objectives are satisfied.

Now here you are probably still only answering one question, which is which varieties do better under different water regimes. 

Lucie: Yeah, that sounds like the key question there. 

Roger: But in general you have questions about varieties, you have questions about spacing, you have questions about planting, you have lots of possible [00:10:00] factors. You’ve got to choose a level of all the factors that aren’t in your experiment. And so why not put two or three, or even four factors into your experiment?

Now the interesting aspect of factorial treatment structure, please remember there is a thing called hidden replication. And so the questions on the main effects of the factor have quite a lot of replicates, and that’s one of the wonders of factorial treatment structure. And if people listening, I’m not at all clear on that, then we should have a whole podcast on this hidden replication because it’s really rather important.

So that was one of the items that I always spoke about. And you can see that because we have multiple factors in most studies, they mean we can have more objectives and we can have objectives about each factor, which variety does [00:11:00] best? We can have objectives about the other one in this case would be how much water is needed? And then we can have objectives to say, does the water factor affect all varieties in the same way or are there some varieties which survive drought better than others? And that’s the interaction.

So that, I found that if there was one thing that I tempted people with, it was the idea of introducing factorial treatment structure into experiments. I have to say, when you come to on-farm experiments it may be that you are not so keen on this because the structure is reasonably complicated. But I think that’s a question for another day.

I’ve mentioned an item, which is factorial design. So in the treatments, I would like people to consider having often a decent number of factors in their design. The second component is [00:12:00] affecting the layout. So in the layout, this little irrigation experiment lends itself naturally to what’s called a split plot experiment. And most people will have heard the names, but I think it’s fairly obvious in this irrigation example, you need big plots for the irrigation level, and you don’t need big plots for the varieties, and so you split a big plot into, let’s say, four little plots. The varieties go on the little plots, and all the four varieties are on a single plot from the irrigation point of view. That’s called a split plot design.

Now, if you read the literature, there are lots of reasons given in the literature why split plot designs are a good idea. I would like to have the contrary point of view that the only time you do a split plot design is the example I’ve mentioned, where one of your [00:13:00] factors needs large plots and the other one doesn’t. 

Lucie: Yeah. 

Roger: Many people seem to go into automatic to say, if I’ve got two factors in my study, which one should I put on the big plots and which one should go on the little plots? And if it’s variety and fertiliser there may be no good reason why one is bigger than the other. It could be, sometimes fertiliser leaches, and therefore you need bigger plots for fertiliser than you do for varieties. And the contrary is if the varieties are very different heights then they may shade each other, the tall ones. Therefore, you want quite big plots for variety. You may not need big plots for your fertiliser or your weeding regime.

But in general, most experiments should be a simple randomised block. So in most experiments, if you’ve got one factor like irrigation at three levels, and the other factor like variety, at four levels, you’ve got 12 treatments. [00:14:00] So the simple design is blocks of size 12 with the treatments at random, those 12 treatments, three times four at random, within the block. So a simple randomised block with everything at what I call one level, namely there’s only one plot size, is the norm. And you’ve got to argue for anything more complicated.

And it is more complicated when you have a split plot because you have what’s called multiple levels. You have big plots and you have little plots, those are two levels.

Lucie: And that’s why you don’t recommend it, because it complicates things unnecessarily.

Roger: It complicates things unnecessarily. Well, it complicates things and unless it’s necessary, don’t do it.

Lucie: Yeah.

Roger: It is my simple rule, and people will have often lots of other possible reasons of which factor is more important than the other factor. They don’t work in practice. You lose much more, it’s the little plots that give you more information than the big plots. But [00:15:00] you generally lose so much more on the big plots than you gain on the little plots that that very rarely works.

So just keep a simple rule. Do you need to have small plots and big plots, namely plots of different size? If yes, do it, that’s a split plot. And you can even have a split plot where you have three factors in three levels. Ah, do you realise, we’re not talking about standard errors, but if we were, split plots have 11 different standard errors and very little software gives them.

So actually analysing the data, it becomes tricky, it looks standard, but because it’s a split plot, it’s difficult to analyse. Don’t do it unless you’re forced into it by, as I say, some factors need bigger plots than other factors.

Lucie: So, from your perspective then, what does statistics teach us about how to plan and design agricultural trials? 

Roger: Now, it helps us be comfortable with the researchers [00:16:00] that if they’ve done a sensible trial, then the analysis will also be quite easy to do. And they don’t have to worry as much about having things, which are very neat. The statisticians are the professionals, we’re now often called data scientists, and the scientists and the researchers are collecting data. Often the statistician can help on turning data into information.

Now, you can only turn data into information if you’ve collected it in a sensible way. Otherwise things don’t work very well. There’s lots of simple mistakes you can make with your experiments that would mean that your data aren’t as useful as they otherwise would be.

So, first making sure that it’s all sensible data, and when the data comes, it’s going to be logical and easy to analyse. And then often helping people with [00:17:00] the analysis is where we fit in because we are professionals dealing with data. And that concerns the collection of data as well.

We’re also quite good at merging data, so one of the things I find is that often there’s a lot of data that’s already around on the same subject. And students will often be encouraged to look at past studies to say, can I do now a study which compliments the information? And the statistician might be able to help if they’ve got access and can do further analysis on studies that were done previously.

That takes the idea that your report ought to have a sort of literature study, doesn’t just have to have a literature study, the literature study should be what was available before I started. So therefore, the newer features can compliment that. I think people often could take experiments that were done before they started and maybe from their same [00:18:00] researcher and analyse them a bit further and find there’s more information there. That’s very rarely done. And the statistician can help and make it all easy. 

Lucie: You mentioned that there’s some sort of traps that people can fall into, which make the analysis very difficult. Can you say any more about what those traps are?

Roger: The main aspects of that would be when you collect data, where there’s a different reason for an effect. I’m trying to think of a good experimental one, let’s say that you had a block which is very fertile and a second block, which isn’t fertile. And so you have two areas of land and you’re going to use those as your two blocks. And you have four varieties, and you put two of the varieties on the fertile block and the other two on the not so fertile block.

Now that would be a rather poor design because the other two varieties don’t do well. Was it because of [00:19:00] the variety or was it because of the fertiliser? It would be much better to do what is a standard randomised block in the fertilised block or the fertile block, you have all four varieties, and you have also all four varieties in the not fertile block. 

Lucie: And this is like, it makes me think, I think often researchers have difficulties in terms of what the previous crop was too, and how that can influence the current crops or the current year’s trial. It seems like a minefield.

Roger: Designing a good experiment is a minefield.

Another area that I found was very useful when I was working in ICRISAT was I found that the statisticians tended to stay inside and wait for the data. I found it was rather useful to go on field visits and look at the data as they were growing. And then I found some interesting aspects of the whole field, which we discussed.

So one of these was I found some [00:20:00] experiments where some areas of the field, not a total replica, were very much affected by previous termite mounts, and therefore their fertility was very different to the others. And the experimenter was rather confused by this. And we discussed how should they account for that.

And if you remember the rule that in an experiment, you try and plan for as much as you can. If you haven’t planned for it, you measure it. So now I would like at least to record which plot was affected by the termite mount.

Another example I found was that we would go around the plots and the end of the experiment was measuring yields. But we then found in some experiments that some of the plants had died. So there were plans to consider measuring the proportion of the plants that died, and if you don’t measure that, you are not understanding, there’s two different components now. [00:21:00] In plants that died, you don’t get much yield because it’s dead.

But recording that it has died and therefore, how many plants made up the yield, was an extra variable you wanted to collect. And this brought in the fact that we then started discussing something that wasn’t planned for in the objectives, but now could come in, which is why had the plants died and why had they died so differently on different plots? And did we have another objective that was introduced?

And that brought me to something which I found people hadn’t thought of enough, which is really rather important. Before the experiment, you have your objectives. You now design your experiment, you collect your data, and you go and see the statistician. The first question I usually ask was, can you show me the objectives?

And they were very surprised. They said I don’t have to show you the objectives, I just want you to do the analysis. And I was saying, well, we’d like to make sure the analysis satisfies the [00:22:00] objectives. And then the next question is, once you’ve now got your data, are there any of those objectives that aren’t feasible anymore?

And also are there any new objectives that came in where you took advantage of this experiment that you hadn’t planned for, but because this year was rather interesting and odd you measured some new things or else the measurements you made were helpful and now you actually have a new objective? The effect on termite mounds is an example of that.

Lucie: Exactly. 

Roger: And then what happens with the termite mound data? What should you do about it? And this is where the statistician often could help by emphasising that the fact that you designed your experiment with equal number of replicates for every treatment wasn’t important.

So one easy possibility, if you had 32 plots and 6 of them were affected by termite mounds, now [00:23:00] termite mounds is now introducing another factor. It’s a two levels, was there a termite mound or wasn’t there? And one thing you can do is to say there’s only 6 plots with termite mounds, I have 32, I’ve got 26 without them, it’s a perfectly valid experiment, but I think I’m introducing unnecessary variability. If I think of it just as an experiment, I’ll get rid of those six, which are affected by termite mounds, or I’ll report them separately and then I’ll analyse the other 26 together.

Lucie: Okay.

Roger: And that brings in the fact that if you’d known about the termite mounds before, you might have been able to allocate your treatments better to the 26 plots, which you hadn’t known about. And so, again, understanding your plot, and you were saying that’s where the previous crops become important, understanding the area that you’re doing your experiment is really very useful.

Lucie: So just to finish off then, I also wanted to know what the limits [00:24:00] are of statistics in helping people design on station trails. So you’ve mentioned already that, for example, crop varieties, we are not experts in whatever crop or whatever irrigation method and things like that.

Roger: The limits to me are very clear and they’re not always well understood. At some stage I might mention my first advisory when I was asked about how many tadpoles should be in how many jars. And the first rule is that the agriculture trumps the statistics. So that if there are agricultural reasons why plots should be big or small, those are much more important than any statistical reasons. So agriculture wins.

And this comes to, do we need guard rows, are things feasible because you have leaching, or you have shading, or various things like this? All these practical issues need to be taken into account, [00:25:00] and don’t invent artificial statistical rules that we did it this way so we had equal number of replicates for each treatment, those aren’t important.

But practical rules of biology and agriculture of doing sensible experiments, those are very important. And make sure that as a practical study it is sensible. The main thing that goes wrong with that is that you are trying to do something on small plots, which would be equally applicable if it was whole fields. So make sure that the small plot could be representative of a larger area. And there’s all sorts of reasons why small plots may not be. But most of those are practical agricultural reasons rather than statistical reasons. Don’t let the statistics have undue importance.

Lucie: It’s a conversation that we could spend hours talking about, these and all of the different examples you’ve come across Roger, in terms of people trying to set up experiments and needing support in that. [00:26:00] It’s been very interesting though, and I have still a lot to learn, I think. 

Roger: Okay.

Lucie: Thank you very much, Roger.

Roger: Always a pleasure.