186 – Twenty Years of RMS for CRFS: Measurements

[00:00:07] Lucie: Hi and welcome to the IDEMS podcast. My name is Lucie Hazelgrove Planel, I’m a Social Impact Scientist and anthropologist, and I’m here today with Roger Stern continuing our series of conversations about research methods for agriculture.

Hi Roger. How are you?

[00:00:21] Roger: I am very well Lucie.

[00:00:23] Lucie: So I’d like to dig into the topic of measurements today. It’s one of the three key aspects of agricultural trials that we’ve been discussing. So where you’ve got the treatments, layouts and measurements. It also exists in surveys, which we might discuss as well. But firstly though, I’m not sure if other people will really understand what this concept of measurements is, I can’t think of a better word for it.

I don’t know how you would introduce it to somebody, how would you explain the concept of measurements to somebody? I mean, there’s variables you might want to monitor, things which explain the context. Is there a simple way of explaining what measurements are as a general concept?

[00:01:05] Roger: I think so. I like the categorisation, which we have of context, measurements that directly relate to our research objectives, and measurements that help to explain variability. And whenever we categorise things, we have to accept that it won’t always work.

[00:01:26] Lucie: This categorisation, I think you helped develop when you’re at the Statistical Services Center at Reading?

[00:01:32] Roger: That’s right. So, level number one, now I’m back to thinking that I really would like some examples. My first example is an on-station trial, which we’ll have as a simple randomised block with some treatments, which could be varieties, and we are doing this in four locations.

The second one is an on-farm trial with a few treatments, which we’re doing in four villages, and we have 10 farmers in each village. And you mentioned an anthropologist might spend quite a lot of time studying the situation in detail in one village and looking at households within a village.

[00:02:18] Lucie: Shall I try and explain then the example of these three different types of uses of measurement for anthropology? What the equivalent would be for anthropology, how they would use them.

[00:02:28] Roger: I think that would help.

[00:02:30] Lucie: So, in anthropology, you often, you can’t split yourself up between many things. You want to do an in-depth study, especially when you are at the PhD level or sort of a Masters, you want to spend a lot of time in just one place. So the first type of measurement you might be looking to get is really this context. What is this place that you are spending time in?

So if it’s a village, then your context measurements would be about where is this village, what sort of population lives in it, and any other sort of key information to describe that place that you are in. Or perhaps it may not only be the physical place, it may be also the political situation that is happening at the same time, or any other sorts of social things that are happening, which really give the context to that time and place that you are living.

The second type of measurement that you might be looking at is really about your objectives, your research objectives. So in my case, my PhD was about craft work, so all of the measurements there was about who does craft work, when do they do it, how do they do it, what do they, you know, it was all of those sort of questions which are directly related to the research objectives.

And then the third level then that you were discussing is the different sources of variability. So this is looking much more, within the place you are studying, looking at, well, what can be changing or what can be affecting your results. So, when you’re doing research with people. Different types of people can have different types of views or different perspective. So if you’re talking to a village leader, they will have a perspective that is very different to if you’re talking to a child in the village.

Or in my example of craft work, if you’re talking to someone who is very knowledgeable about craft work, who has been doing it their whole life, they will have a very different perspective to somebody who only occasionally touches the materials that they use, who doesn’t really have an interest. So that knowledge of the person, the source of your data, that is an important source of variability.

So it’s a different level to the general context, it’s looking at where your data comes from and how it may, how you can use it. Can we compare this then to perhaps an on-station trial?

[00:05:02] Roger: I think we can. I’m very happy with that example. I noticed with your example, the context variables are what you’ve defined to be at the village level, and then you have gone straight to people at the person level, omitting what I would call the household level.

I think in terms of rectangles of data, and I think in that sense, some of your, what do you call it, your variability level, so you are interested in the person level, anthropologists are interested in people. And in that sense, often the household level information is quite a lot of explaining of variability, but not maybe of direct interest.

[00:05:50] Lucie: No, it is, it is, and you’ve picked me up on a mistake because, for example, if the partner or if there’s somebody in the household, you know, in terms of economics, if there’s someone else in the household who has another source of revenue, then it completely changes the situation of who needs to do craft work or the reasons why somebody is doing craft work. So you’re quite right, that is another level within there that I should have mentioned.

[00:06:16] Roger: Okay. But I think you’ve shown very clearly how we have these three reasons for taking measurements, and we’re now going to show the parallels. But also many people would consider a sort of a study in anthropology as very different to what they’re doing, and showing how similar it is, and this links very well with my feeling that we should be gathering evidence in a wide variety of ways. And I think that fits quite nicely in the sort of planning of a study, that people think rather narrowly, often, of a method of data collection that they’re familiar with because they’ve been taught, and it’s good to think whether a different method of data collection can help with the objectives.

So, let’s take your example and think of our on-station trial as the opposite, if you like. So this is an on-station trial, which we’re going to do in four locations, and it’s going to be a randomised block experiment, with eight treatments, that we are going to observe. And notice your anthropology, we didn’t have treatments, so your anthropology study is closer to a survey than a trial.

[00:07:40] Lucie: It is an example of a survey. Yeah.

[00:07:42] Roger: It is an example, which we can think of as a survey. And surveys don’t have treatments, the measurements is where you get your information. But some of the measurements discuss location and are for variability, and some of the measurements are because of the objectives of your study.

So in our on-station we have our four locations and we still want to make measurements of the context, when was the planting date for the experiment, and maybe the total rainfall and the soil type. And a feature of the context variables is often that they’re recorded at the experiment level, there would be four recordings of those in my instance, because I’ve got four villages and I’m doing the experiment in four places.

If I’m doing the experiment only in one place, which is like your one village, I would still want to record the context. And often when we report this context, it’s often recorded in the materials and methods part of the paper.

[00:08:54] Lucie: This is really interesting, yeah, because that’s where the context would be for describing your village or your location in anthropology too. It would go right at the beginning to set context, as we say.

[00:09:05] Roger: And so those are the context. And then we have measurements which we make at the plot level. I like to think of rectangles and some of the columns show us the treatments at each plot and now other columns tell us the yield, and so on. Maybe the date of flowering and the yield might be measured in a number of ways.

So those are the measurements. And again, often we think of the measurements for our objectives. If our objective was to get the best yielding varieties, we can’t satisfy our objectives without measuring yield. We might also find, and sometimes you find this during an experiment, that some of your plots behave oddly because you observe they are within a termite mound. You are not interested in termites as one of your objectives, but being in a termite mound means that the plot is very fertile, so you record that it’s in a termite mound or not, because that could help to explain variability.

[00:10:13] Lucie: Absolutely. That sounds important.

[00:10:15] Roger: And you often find the things that are to explain variability, one of these, of course, is the layout variables. You’ve often done the layout, which is which block they’re in, often, in an experiment, you put the experiment into blocks to help reduce the unexplained variability. So you do record each block, and we call that part of the layout. But you see how these different columns or variables that we’ve got to analyse, they’re all collected.

[00:10:47] Lucie: So we’ve discussed three different uses for measurements. But it should also be said, I think that not all measurements are useful. And you have an interesting example of this, Roger, if I’m correct.

[00:11:01] Roger: Yes, and I want to give you the context of that set of measurements as well. So this is somebody who had done an experiment with different varieties but had also measured the height of the plants for five plants in each plot. And had done that also at a number of times during the season, after 20 days, 40 days, 60 days, they’d measured the height of five plants.

And so they had many measurements at the plot level, such as the yield and the date of flowering. And they had these additional measurements at the plant level. And on each location were five columns, five more columns in each of the data because they were measured at the plot level.

And the researcher was keen to analyse the data as a whole, and in particular to analyse the plant heights. And plant heights was an interesting variable because it was very time consuming. So when it came to the analysis, I queried what was the objective of measuring the plant heights?

And his answer was initially that different varieties had different heights. And I was surprised that he didn’t know that ahead of the experiment because I think that’s often learned about different varieties at the early stages before this particular level of experimentation.

And then it turned out that the measurements he took were just because he looked at the measurements that were taken on a similar experiment last year and copied that section, deciding which measurement to make. And that’s not a good idea. It is very good to reflect on what you need to measure for your particular objectives.

[00:13:09] Lucie: And as you said, you know, measuring plant heights, it takes a lot of effort. It takes a lot of time.

[00:13:15] Roger: It takes a lot of time, and it’s sometimes indicative of the person designing the experiment, not being the person that has to do all the work, so it’s often done in a way that hasn’t involved enough thought. Anyway, we analysed the data.

[00:13:32] Lucie: Okay.

[00:13:33] Roger: And a result of the analysis was that he was initially quite happy that the results were significantly different at whatever level you want, they were statistically significantly different. So he proved beyond reasonable doubt that the different varieties had different heights. And when we discussed how to write that up in his report, he realised that he knew that already, and that in fact he hadn’t needed to measure the heights.

[00:14:04] Lucie: And he was perhaps not the only person who already knew it.

[00:14:07] Roger: Everybody had known which varieties tended to become taller and which varieties tended to be shorter. So there’s lots of good reasons for measuring plant heights as they grow, but not in this particular experiment with these objectives.

And I found that a very useful example, because this is the same idea that when you do a survey, you’ve got lots of questions you could ask, and each one of those I’m considering as a measurement. But make sure that those questions are either because they correspond to an objective or because they can help to explain variability in your data.

[00:14:48] Lucie: And something which I think is important too is, when you are planning your research, and especially something like a survey, is to think about how you’re going to analyse each question. And that can help perhaps reduce the number of questions or sort of limit the number of questions to only what is needed and necessary.

[00:15:04] Roger: And also, people don’t spend enough time on data quality. Another corollary of this is that long surveys, it’s not obvious that the respondent or even the interviewer can keep attention for so long, and therefore the quality of the data may be reduced. So there is a cost to measuring lots of things often in terms of the quality of the data.

[00:15:29] Lucie: And when we talk about quality of data here, we mean that people may not be saying what they really think or they may not be saying what really happened, they’ll just be giving a quick, a quick answer. And then, what’s the point in actually collecting that data then if it isn’t real, if it isn’t useful, if you can’t rely on it.

[00:15:48] Roger: That’s right.

So now let’s think of the sorts of measurement that we want.

[00:15:53] Lucie: Okay, to answer the question of what measurements. Yep.

[00:15:56] Roger: I’m very taken recently by the fact that people, when they learn statistics, they learn about the normal distribution. And the normal distribution is useful for variables like the yield, which is a continuous variable where you’ve got a number that you record.

What’s much rarer in experiments is to record things in what we call a categorical way. So is the yield very good or good or very poor, which could be five levels of a category. And often that is much quicker to do than weighing to measure yields, and so the question is whether that would be sufficient.

And I would like to encourage people in terms of the measurements, to consider whether they do need to get the precision of the numbers, which is often extremely useful, or whether measuring categories is good enough.

[00:16:57] Lucie: So for age, you know, age is a silly one perhaps, but you often don’t need their precise age, you can know just, well, what sort of age group or what age category someone is in. And similarly with income, I mean income is a very difficult, very sensitive question to ask people. But if you give them categories, then that might be easier and it might lead to more reliable data.

[00:17:20] Roger: That’s right. In many studies, measuring income is almost impossible, but measuring possessions in a household, such as whether they have a television or a mode of transport is an indication of wealth, which is much easier to record and much more reliable.

[00:17:41] Lucie: Yep.

So that’s what sorts of things you might want to measure. And obviously you can measure them at different levels, as we’ve mentioned at the beginning, you might be wanting to measure things at the whole village level, you might be wanting to measure things at the individual or the household level or at the plot level, of course.

What about when you measure? Because I think, you know, if we come back to your example of plant heights, I know people measure that sometimes at the date of flowering. So, I mean, there’s some things that you might want to measure just once, and there’s other things that you might want to measure at different times.

[00:18:14] Roger: I think that’s right. And sometimes yield is measured just once because it’s the yield and that’s obvious. And sometimes with grass yield, for instance, you can harvest grass yield and then it regrows, and so you could measure it on a number of occasions.

In typical agricultural trials, you have a decision as to whether you are really wanting to have a repeated measure, as it’s called, and that brings in another feature that you may want to have that at a different level. For example, in animal trials, you may want to measure weight each week on animals to see what the weight gain is. And so now you are measuring multiple times on the same animal.

Now your animal may be your unit, like the plot. But sometimes a set of animals might be in your plot and then you must make sure you go back to the same, or it’s very valuable to keep going back to the same animal because now your measurements are made at the animal level or the plant level within a plot, and so you are following through a particular animal or plant.

[00:19:32] Lucie: And you can see if one particular animal was an extreme case or not.

[00:19:36] Roger: Yes. And when you come to the analysis, you often want to summarise that to get an idea of growth, which would be measured as a single number rather than the repeated measures. And so, you often get quite a lot of columns of data for each of your plots or each of your animals, and now you do an initial analysis of those, which takes your data, in a sense, up a level. You have occasion within an animal, so you are now trying to summarise that at the animal level.

[00:20:11] Lucie: But with this question of when do you measure, it comes back to, well, what’s going to be useful to analyse? How are you going to use it? ’cause there’s no point in taking many repeated measurements if in the end it’s not going to give you any more useful information than just taking it at a weekly basis or less frequent occurrence.

[00:20:31] Roger: That’s become even more important now because we often have equipment to measure things automatically. So whereas by hand we might have measured weights every week. Now we can almost measure them continuously and it’s not clear that’s giving more useful data, it’s giving a lot more detail. The question of how often do you want to measure becomes a similar parallel to how many questions do you want to ask? And so that’s another discussion that you need to have as you are planning the experiment.

[00:21:06] Lucie: Something I would really like to discuss too, Roger, is measurements with farmers. So you said at the beginning that we can think of taking measurements categorically. They don’t necessarily need to be always numerical measurements. And this is, I think particularly the case in working with farmers, you want to make taking measurements as easy as possible for them. So what would you recommend, what sorts of suggestions do you have for that? Do you have any practical experience of this?

[00:21:36] Roger: Well, I think you are absolutely right that when you’re doing an on-farm study, whether it’s an experiment or a survey, you are asking the farmers questions and you want to make the reporting of those as easy as possible. And I think recently now this Tricot popular series of experiments provides a very nice example where instead of asking the farmers to measure yield, you ask them which of those from a yield point of view do they think was the best out of the varieties, and which was the worst? And that’s much simpler measurement to take.

So, I think there is a good lesson for us that Tricot gives, whether you are doing that or not, that asking questions and getting, well, maybe it’s not the asking questions, but it’s getting categorical information, often it is all that’s needed for your objectives. And it is often an order of magnitude cheaper than becoming embroiled in physical measurements such as measuring the weight of the crop. Again, you often measure the weight of the crop before it’s shelled and after it’s shelled, and getting an estimate of that can be all that you need for your objectives.

[00:23:01] Lucie: And what would you say to people who say that, well, that’s not very objective.

[00:23:07] Roger: Well, just a minute. There’s two bits of objective. One of them is because it’s categorical, you are not getting the quantities. What are you omitting? And so you are losing out in that way. And I would say you very rarely lose too much there.

And the second aspect of objective is whenever you record something that anybody says, maybe they are telling you the truth or maybe not. That’s a general issue on questionnaires, even when you are weighing things, you may make mistakes in recording the weight. And so you must always be careful on taking these measurements, and try and ascertain that you’ve got high quality data.

[00:23:56] Lucie: So part of that is trying to make it easier for taking the measurements, whether that’s by, you know, using measuring equipment which is easier to manipulate or whether it’s with farmers asking them questions and asking them to make measurements, which they will understand and which means something to them.

[00:24:12] Roger: And sometimes if it’s a particularly important area, trying to do things in more than one way, so you can have an idea of consistency.

[00:24:22] Lucie: So I was just thinking, yeah, for example, if you ask someone their household size, I mean, it seems like such a simple question, but do you include the people who are just, you know, there temporarily? Do you include related family who are staying there? So different people have different ways of answering that.

Whereas if you, if you think of different ways of measuring it or if you, as you said, think of different questions, like multiple questions you can ask to get at the same answer, that can have some interesting results too.

[00:24:52] Roger: And I, I think that brings us to another subject of you learn a lot about which questions are difficult by doing a little pilot study. So be prepared to practice and do pilot studies, and you learn a tremendous amount. And household size, as you say, for some households is a very simple question. So just knowing which proportion of households is household size a complicated question. So you may ask an additional question about how fixed your household size is.

[00:25:26] Lucie: Ah, yes. Yeah, interesting.

[00:25:28] Roger: And that can be a very useful thing to have as well.

[00:25:31] Lucie: So what I’m getting out of this conversation, Roger, is that knowing which measurements to take, when to do them, how to do them, it’s not as simple as it perhaps seems initially, that you do need to think through how to do it, and it’s good to do a pilot study, as you say, to actually see what the results are going to be. And it’s good to think through how you’re going to use those, that data beforehand to make sure that everything is relevant, all to do with getting good quality data.

[00:26:00] Roger: That’s true. And when we come back to the training, I’m very, if you like, taken by the fact that if you were training people to do surveys, this would be a well recognised part of the training. Namely how to ask questions, and the fact that it can be rather difficult.

When you do experiments, if you look at the textbooks on experiments, they don’t discuss this area, and most of the textbooks, I find, only measure one thing in a trial, and it’s usually the yield. And they’re obsessed by the methods of analysis of that one variable. And then they don’t discuss any of these issues of the difficulties of taking measurements, which are just as difficult for a trial as they are for a survey.

[00:26:49] Lucie: And sometimes it’s worth to be creative. Yeah.

[00:26:51] Roger: Yes.

[00:26:53] Lucie: Well, fantastic. I think we’ve given some suggestions and ideas to people in this conversation. So thank you so much, Roger. It’s been really interesting.

[00:27:01] Roger: Okay. Till the next time.