181 – Tricot Participatory Breeding Trials

[00:00:07] Lily: Hello and welcome to the IDEMS Podcast. I’m Lily Clements, a Data Scientist, and I’m here with David Stern, a founding director of IDEMS. Hi David.

[00:00:14] David: Hi Lily. What are we discussing today?

[00:00:17] Lily: I thought today we could discuss Tricots.

[00:00:20] David: That sounds great. Yes. You’ve come back from a workshop where we’ve been working on this now for about six months. Maybe I’ll give a little description of my understanding and you can then come in.

This is a group who have developed a method primarily around breeding trials, participatory breeding trials, where they work with lots of farmers and give them a very simple set of experimental protocol, which they have three things to choose from, or three sets of seeds if you want, it’s the most common scenario I’m aware of. And then the farmers plant the three sets of seeds, and then they give very simple information about the plant as a result, the favourite, the best, the worst, and hence, you have a ranking of those three varieties. And this is then used at scale to try and actually make comparisons of large numbers of these varieties.

Have I described that roughly correctly?

[00:01:33] Lily: Yeah, that sounds about right. Yeah. Giving different farmers three varieties each, and they asked questions to rank, which one was the best taste, which one was the worst taste and so forth. So Tricot stands for triadic comparison of technologies.

[00:01:48] David: Exactly, and the key to it is three.

[00:01:52] Lily: Yeah. So farmers are testing three varieties of something like crops.

[00:01:58] David: Exactly, and the statistics behind this is really quite interesting, of course, because normally when you do these sorts of comparisons, you would want to have, let’s say, a quantitative measure of yield to determine which is the most or the best. Whereas here you have a qualitative measure of comparison, which is the farmer’s perception of, let’s say yield, to determine which is the best.

And what’s so interesting is that it seems like you’ve lost a lot of information to go from having a nice, precise number of yield to now just having the farmer’s perception, is it the best, is it the worst? What a lot of information you’ve lost.

But in practice, actually, the stats have shown that you do pretty well with this. And there’s a few really interesting things which come out of that. But anyway, maybe we should dig into your experience of this workshop where you’ve been helping people, because the hard part, of course, is this is hard data to analyse, people aren’t used to analysing this sort of data.

[00:03:16] Lily: Yeah. No, absolutely. To add it’s a really, it’s a different approach, but it’s a really good approach or a really interesting approach because it’s on top of things very simple for different farmers to understand, it’s simple to talk about. Unlike, we could sit here and talk about an AB testing trial, and that would probably take up the whole podcast. This is hopefully simple enough to follow, okay, you’ve got three things and I’ll rank them this way.

So before we get to the workshop, I’ll just introduce what we did, which was we were looking at these kind of Tricot analyses, and the people who work on this have already written these incredible R packages which analyse data because it follows this like different structure that you’ve alluded to.

[00:04:05] David: Let me just come in on that very quickly because this is the power of statistics, this is relatively traditional statistics. A lot of what we do now relates more to data science. But this is the power of traditional statistics that if you have elements of design, which constrain the data you get, you can use those in the analysis in powerful ways.

This is actually quite a constrained design, this sort of having three treatments and the nature of the data, which is extremely powerful when you want to then do the analysis at the other end. But this is a type of analysis where there’s been statistics research on this to understand how to do this.

And so they’ve had to build the packages to do the analysis themselves so that becomes accessible to others because it’s non-standard. One of the things which I found so interesting about this approach is the approach is constrained in very specific ways to enable the powerful analysis at the other side. But as you said, these R packages which have been developed are needed to enable that analysis to become accessible. And that’s where we came in, is that correct?

[00:05:32] Lily: Yes, exactly. And so these packages were what they were using in their training, or in their workshops. But they were finding that it was a little bit much to learn R, and to learn these analysis. ‘Cause while the idea is relatively simple, the analysis behind them are quite complex, and learning that alongside learning a new coding language alongside learning how to code these analyses in this new coding language can be a lot.

[00:06:08] David: In the education field we talk about the cognitive load of trying to learn to code, and the R language, and the statistical approach you’re supposed to take all at the same time, is not effective. And we’ve been involved in this and one of the reasons we got involved is they were aware of some of our other work where we’ve made certain types of analyses more accessible, easier for audiences that are less comfortable with the coding.

And we’ve actually done quite a bit of work trying to make it more accessible, approachable, to be able to use complex analytic techniques, which are tailored analysis for a specific audience. This refers to our climate work in particular, but in other cases as well where we actually try to disentangle this and enable people, if you’re wanting to learn to analyse this data, focus on trying to analyse this data. Let’s make it as easy as possible for you to be getting stuck into what matters without worrying about learning to code or learning R itself.

[00:07:24] Lily: Yes. And so this is where R-Instat came in, which is this front end to R that we’ve been working on for 10 years now, I’ve been working on it for nine years on and off, but 10 years now. And so what we did was we created this menu in R-Instat where you just had to run through the menu, because Tricot data follows this very specific structure and it has this kind of certain analyses that make sense for this sort of data, then we could take these very powerful packages that the Tricot team had worked on, and just simply put that into a front end menu.

And of course there was a lot of back and forth with them of different options and the menu design and how to get it to work in the way that they would usually run their workshops. But we put it into the menu and so then that’s what the workshop was that we did that you’ve mentioned at the start of this was us piloting the menu with a group of people to see how it works, to see does it actually work in practice to run through Tricot analysis.

[00:08:30] David: Let me again come in very quickly because what I love on this, and you’ve discussed the menu, you’ve discussed R-Instat. You can make it really easy through dashboards and they already have dashboards.

So what’s really interesting to me is that this is this in-between zone. It’s not just that somebody’s got a dashboard, which is a predefined analysis. The menus within R-Instat, what this is allowing the participants to do and the researchers involved, is to dig into not just the analyses that are laid out for them, but also just more general data analysis related to the data in a way where they are making decisions, they’re understanding their data, they may be bringing in new variables. So this isn’t, the data this is used on is all of a similar type.

But even in the workshop, which was a really short three day workshop, you then were introduced to new data sets. How did that go?

[00:09:42] Lily: So we were then introduced the new dataset, which was fun. I have a nervous laughter there because obviously we built this menu for our data sets that we’ve been practicing on. But then with every new data set you come across, you encounter new challenges, which we were fully aware of. We knew that if we were to come across new data sets, there’ll be something new in there that could just help this menu further and help strengthen it to help these different kind of cases that you can have.

For example, handling missing values in your Tricot responses. What if people haven’t said that something’s best and worse? Those sort of challenges meant that since this workshop we’ve come back and we can amend the menu as needed to then continue its development to keep improving it.

[00:10:31] David: Absolutely. And this is where I’m quite excited about this work more generally. There’s only a finite set of sort of new challenges you’re going to get with this type of data because the data’s well structured. And before the workshop you kept asking for different examples and you were given a few interesting examples to work from. But at the workshop, you were then exposed to another five or so new examples or new sets of data, a couple of new sets of data, and you therefore had new challenges.

But there’s at least 400 or so trials ongoing or recently completed using this methodology around the world. And actually getting it so that any of the people using this method could be benefiting from this sort of approach to analysing that data, there’s a few more to come across. You know, when I last talked to you and Roger about this, we sort of identified maybe 10 or so different ways in which we could see people using this in the trials, which would maybe need to be adapted to.

But that’s still a relatively small number. And so we could imagine that pretty quickly anyone doing this sort of trial could be finding that the menu would really help them to facilitate the analysis in a way which will, we hope, be really transformative for the partners using this. Just leading to more analysis of the data, leading to more insights, and hopefully making the whole process smoother.

[00:12:19] Lily: Yes. And this isn’t just a problem that would come with, you know, we are developing this menu and we are getting these different cases. But what would currently happen would be to handle these issues, like these nuances within your data that different data sets give, at the moment, you’d have to do it in R.

[00:12:40] David: Yeah.

[00:12:40] Lily: And so hopefully once we collate, once we have more and more data and we get more and more of these nuances come out, it actually becomes really simple for people when they analyse it, oh, this issue has also occurred to someone else and so we’ve gotta fix for this, or we’ve got a way to handle these kind of missing values.

[00:12:59] David: And the missing values to me are a fantastic example because this is a case where you don’t tend to think of this when you first design the analysis plan for this sort of data, but they do occur in certain cases, it might even be that there’s not just a single set of missing values, there’s multiple sets of missing values. A missing value for the worst when two of them have both failed is different to other types of missing value.

And so there’s interesting ways in which this data could be collected which could determine how you interpret missing. But once you’ve built this into these systems, this is something which can then be reused relatively simply by others doing their analysis. And that knowledge of what to do in those cases, from your PhD, this can be hard, you spent years and years dealing with some of these issues in a specific case.

So this isn’t something which everyone would know how to deal with automatically, but it only needs one person to build it into the system and everyone else can then benefit. And this is exactly where we’re quite optimistic that a little bit further down the line, we could be really seeing people having the tools needed to analyse this specific case.

And then of course, one of the interesting things is that, actually, if you think of other explicit agricultural designs or designs for agricultural experiments, there’s many which are just ad hoc, and we can help with those partly on what we’ve learned from Tricots. But there’s some others which are specifically structured in other ways to be powerful.

And so you could be getting other tailored menus, which would then support other specific methods. And the more precise you are with your design, the easier it is to then build these tools which really help guide the analysis in ways which are much stronger than you could do without this approach to tailored menus.

[00:15:22] Lily: Yeah, it’s very exciting the kind of different possibilities and just avenues that this could go on to take.

[00:15:30] David: Yeah.

[00:15:31] Lily: So at the workshop we did, we had, as we said, this three day workshop. On the first day introduced Tricot and what Tricot is. We had Kaue, the developer of these R packages was there with us. And he’s the one that usually runs these workshops. So he is very well versed on this and knows Tricot, he’s an expert on it. And so he introduced Tricots to everyone, and then following that we gave some presentations on us and then on introducing R-Instat, the software.

And then through there on day two and three, looked at actually analysing these specific Tricot data and looking at new different datas to analyse using this Tricot menu. And on the whole, I think it went well. I think the feedback we got was positive and it was all very good.

People were very impressed by the menu. People were very impressed by the ease of kind of using R-Instat as a tool. And the people in the room were not what our kind of typical audience would be, I believe, for these sort of workshops. Some of them were people that are incredibly good at R and have worked in their own areas, but then that meant that they were asking some really deep, they were giving some really good insights on it, and requests I suppose.

There was one where they were asking about using Markdown in R-Instat, that ultimately to them is what they would love to see in there. So it really helps give this new perspective. There was another who has a Linux machine and was asking about working on R-Instat on Linux, which is in the pipeline.

[00:17:04] David: Yep.

[00:17:05] Lily: So some really interesting kind of conversations came out, and I think that the main problem was that we only had three days. I would’ve loved to have had longer to keep going through this and to keep exploring these different data sets, using the menu and to go through that with this group of very bright people and to look at, okay, how can we get this to work? How can we really improve this menu and make it very usable?

[00:17:33] David: And from their perspective, how can we analyse our data? This is what they were wanting to do, or some of them had data they were wanting to analyse, which is where you got the new data sets. And that feedback, in the future such workshops could benefit from being five days was a very interesting one, I think it was pretty consistent.

[00:17:51] Lily: Yeah, that was a very consistent bit of feedback.

[00:17:54] David: Which is extremely positive. We were originally pushing for a five day workshop because in our experience, once people get started into this sort of analysis, they really want to keep going and get things out of it. And the experience we were being told from the group who were doing this before was that a three day workshop would be enough.

And I think that the big difference is that if people are expected to be learning how to code, if the cognitive load is too high, then extending the time doesn’t really help as much, in this way within an intense period because people are maxed out already.

But by actually enabling people to actually get on and be more comfortable in the analysis, because they’re actually able to do the analyses they care about, this actually enables them to take more in, in that period and to keep going at it, I believe. So it’s a really interesting question and I’m really excited that this went so well, congratulations.

[00:18:56] Lily: Thank you.

[00:18:57] David: And I’m really pleased with the fact that the software’s held up so well. We are conscious, as you say, that really for it to be ready, as a software’s been developed for the last 10 years or so, but for it to be ready, it does need these few extra big changes so that it works in a way that people expect international software to work right now, this sort of cross platform, web-based interface is a really important component, which is a big change.

And the fact you mentioned integrating Markdown, this is something which I think is not such a big change, but is definitely on our roadmap, that, actually, if you are writing your paper now, if you write your paper in Markdown and you embed your analysis in the paper, this now is a really powerful way to work. If we can actually read and write from Markdown into the software, then we’re helping people to be able to produce the end products that they need for repeatable research.

And that’s something which is so important, it’s very much in the line we’re wanting to work. So I’m really glad that question came up and it’s on the radar of the things that we want to enable. So it’s very exciting, well done.

[00:20:23] Lily: Thank you. So what about going forward? We’ve touched on some things going forward such as Linux and Markdown, but I know that you have some bigger ideas going forward.

[00:20:32] David: I always have bigger ideas. Let me break down a few of the key points. Enabling better analysis of data is at the heart of a lot of what we care about and want to do. And some of that we believe is held back by the tools, and I think the simple observation there is that there is a much wider audience who can work and interpret data than the audience who can work and interpret data by coding.

Now you could argue that the large language models are now producing code for people. But I don’t think that is going to solve that problem, because actually, if you get code produced by a large language model, it works really well for someone like you who reads it and edits that code, it’s a time save. But it is not a time save for the person who isn’t already an expert coder, because they don’t know if it’s doing exactly what they need or not, it’s like a black box.

And it’s not a black box you can trust. And it can never be a black box that you can just trust, because that’s not what large language models are set up to do. They’re set up very well to help people who are already good at reading code, who can then check what it does and make sure that’s what you want. And it’s an incredible time save, I know you use it all the time. But it’s not a replacement for someone who doesn’t want to engage with code in my mind.

[00:22:12] Lily: I mean, that’s true for anything when using like large language models, you need to have a little bit of context, but you need to have a little bit of the knowledge to be able to use it. Otherwise, how do you know to trust the output? We’ve said it in lots of other podcasts.

[00:22:25] David: It’s the simple fact that large language models are really powerful for people who know what they’re doing and how, and therefore can use them. They are a lot more dangerous as a tool for people to replace what somebody would otherwise have to be an expert at. So I don’t need to learn how to code ’cause a large language model would code for me, that sort of statement really worries me because, actually, the large language model will help you to code, but I’m really nervous if it replaces the actual need for expertise in coding. Because then, you know, you are not responsible for what it does. So I think there’s still this need to be able to understand where we can use AI responsibly in these contexts.

[00:23:18] Lily: Absolutely. But hopefully, coming back to R-Instat and Tricots on that, hopefully a benefit that R-Instat can give is it could give you that code.

[00:23:30] David: Yes.

[00:23:31] Lily: Then you can use that to get your outputs in a way that is more responsible.

[00:23:37] David: Absolutely. Well, this is the whole point. This is a curated, expert built system. So it is something you can trust if it does what you want to do. This is the thing, the large language models can do anything. But they’re also potentially there to help you to learn what you’re actually doing so you can learn the R language and so on, a whole different discussion.

So I think that’s really important. It’s not that one is better than the other, they serve different roles. And I think that’s the important thing. So I believe there is a real role for these deterministic, trustworthy systems, which are built to support specific types of need by specific sets of users.

The Tricots experiments, this is a niche group, it’s an incredibly important niche group because the work they’re doing is so important. So to be able to build for that need and to be able to make sure that they’re supported in that, this is great, and this is, I think, really what’s so exciting.

Now, in general that idea of building software to support niche groups, particularly in things like data analysis, this is at the heart of what we want to do. We want to be able to have software which is reliable, extendable, adaptable to these different groups, and which can be built in such a way that specific user groups can have something to serve them. This is at the heart of what we’re doing.

And the key is, it’s not that this niche group needs something which is totally for them. What they’re using quite a lot of the time is the same as what other people would be using. So it has this real overlap. But certain things need to be tailored to the specific design that they’ve chosen, and that gives very powerful tools for them. That’s the approach, which I feel is so important, that we’re not throwing away the general, we are putting the niche inside the general to give them a better experience.

[00:25:36] Lily: I think that’s very well put.

[00:25:38] David: And I will just say that the next step there, in many ways, is that they have their own software, and better integration with and into that is something which I think will help more people using their software to just use this combined ecosystem. And more generally, this idea of building this ecosystem of interoperable tools, which are open source, which are tailored to needs of specific audiences, is I think what is needed here.

I’ve just come from a workshop in California where we were discussing this for education, mathematical education, and this idea that again, there’s an ecosystem of tools which are needed, which need to be interoperable and working well with one another. And it’s the same core idea, that there’s a sort of niche audience, which is building a particular type of textbook, but that idea that it’s not a single tool, there’s a set of tools which are needed, and we need to have these ecosystems, digital ecosystems of some form bringing those together is the same concept, which is what we’re seeing with the Tricots.

[00:26:51] Lily: Excellent. And I think that that’s probably a good place for us to stop, to finish on this.

I’m very excited going forward on these kind of obviously very grand ideas that you always have. But on even the smaller, more manageable ones that we can do more immediately.

[00:27:08] David: This is what’s so exciting to me, is that what you’ve taken is an element of the grand idea and said, okay, forget about the grand idea, practically what can we do to help this small group now with their needs that they have, which is very simple? They have people who are coming in, the cognitive load of learning how to code as well as learning this specific type of analysis was too much. How can we help this so that they can better do the training and the researchers they’re working with can better analyse their data? Here we go. Here’s their way forward. And that’s been great to see.

[00:27:41] Lily: Yeah. And we do also meanwhile have people working on these grander ideas.

[00:27:46] David: Yes. Always.

[00:27:48] Lily: Thank you very much, David.

[00:27:49] David: Thanks.