INDIVIDUAL PLANT DETECTION – CAMERA SYSTEMS WITH A FUTURE?
Camera-based plant protection – the technology already exists but where do we stand when it comes to using it in practice? HORSCH LEEB has been carrying out tests in this respect for quite some time. Theo Leeb tells us about the challenges and the chances.
What is the state of technology with regard to individual plant detection?
‘Theodor Leeb: At the Agritechnica 2019, some startup companies already presented camera systems for spot spraying weeds. This made the expectations of customers, manufacturers and also of political decision-makers rise.
In the past few years, we tried to shed light on this topic and to test where we actually stand. To begin with, spot spraying with optical sensors or cameras is not basically new. This method has already been used for about 20 years in the typical no-till regions with low rainfall like Australia, Russia or Kazakhstan – namely in the “Green on Brown” sector. Another method is “Green in Green”. So the technology already exists. The question is which system really makes sense when and where.
What does “Green on Brown” and “Green in Green” mean?
Theodor Leeb: There are two principles: “Green on Brown” and “Green in Green”. Brown corresponds to the arable soil and green to the plant – whether cultivated plant or weed. The topic Green on Brown has been existing for quite some time. Among others, some manufacturers offer systems for the application of glyphosate before sowing. This method is mainly used in no-till regions. With the Green in Green method, you have to differentiate between cultivated plant and weed. And partly you also get the information what kind of weed it is. With regard to the question how far we have come with this topic: We carried out some tests this season and also last year. With regard to Green in Green we for example tried to spray thistles in wheat. Thistles normally appear in nests and not on the whole field. This would be a typical application for spot spraying. With the test, we wanted to find out how exactly the system detects the thistles and what hit rate we would achieve. We basically can only say that the system works. The thistles were detected but only partly. The hit rate ranged between 40 and 60 %. The question, of course, is if this is enough. In my opinion, it still is far from being field-ready. Moreover, as a farmer you wonder about the weeds that still are on the field – are they tolerable or not? This, of course, also depends on the kind of weed, but it should be clarified.
Were the thistles not detected or did the system not react fast enough and the thistles,
thus, were not hit?
Theodor Leeb: In the test, we differentiated between “detected but not hit” or simply “not detected” and ogically not hit. But this also is a question of the calibration of the system. AT a 36-metre boom there are a total of 12 cameras which are mounted every three metres facing forward with a slanting angle. And the nozzles are assigned to each camera according to the spatial arrangement. It is a rather complex procedure to calibrate the individual camera positions to make sure that the appropriate nozzle opens at exactly the right time.
But the real problem rather is that the thistles indeed are not detected by the system. The biggest challenge are the different lighting conditions. I.e. there is a difference if it is cloudy or sunny, if you have to work with or against the sun etc. And the weather conditions in turn influence the shape of the thistle. The leaves for example roll up a little bit in case of a high solar radiation resulting in a significantly lower detection rate. So what we found out was that there still is need for optimisation.
How could the system Green in Green be improved to make it work?
Theodor Leeb: You have to know that there is an AI behind it. An incredible amount of training data is required to make sure that the system always detects the thistle. You need photos and data from thistles in all shapes, in all lighting conditions, growing stages, from all different thistle varieties etc. We are talking about thousands of photos that have to be analysed and labelled manually. Every pixel has to be assigned correctly. This is an enormous manual effort and finally the crux of the matter. The more labelled photos there are, the more exactly and reliably the system will work.
And a thistle is rather easy to detect compared to other plants.
Theodor Leeb: Exactly. The human eye can detect it rather easily and a human can also differentiate it. The difference between monocotyledonous and dicotyledonous plants is quite obvious. But if you want to differentiate for example black-grass from wheat it is getting difficult. This is where we possibly reach the limits of what is feasible.
But there are further technical restrictions. An important point is the spot size, i.e. the smallest possible area that can be sprayed. In theory, the biggest savings potential would be if we treated every little weed with an effective spraying area of for example 5×5 cm. However, as we work with sprayers where the nozzles are mounted with a spacing of 50 cm or 25 cm, there is minimum spot width of approx. 60 cm respectively 35 cm depending on the nozzle layout. As the nozzles cannot switch infinitely fast, the spots in the direction of travel are about 50 cm long. If the distance between the weeds is smaller than 50 cm, the system will not turn off. Thus, for the savings potential the ratio of spot size and weed infestation is essential.
Physics respectively optics are two other limiting factors. Let’s take beets as an example: It is important to detect weeds at an early stage, i.e. when the size is one centimetre or even smaller. In theory, it is possible the detect this tiny plant with the system if you drive very slowly and can really look at it from all sides. But in practice, operational speeds of 10 km/h and more are common. To achieve a sufficient reaction time, the cameras face forward at a slanting angle. But if there is a large clod in front of the small weed or if another bigger plant hides the weed, the camera will not be able to detect it. So you cannot achieve a hit rate of 100 %. Now the question is what is acceptable. Are 90 % enough? At the moment, we simply do not know.
As thistles do not grow in the whole field but in nests, they are ideal to carry out tests with spot spraying. In the test, we analysed the hit rate of the system and if the thistles were detected.
So at the moment the topic is limited by the training data and physics.
Theodor Leeb: Yes, but there is another exciting question we have to clarify. In many row crops it is good professional practice to apply a soil herbicide after sowing. This guarantees a basic protection for a certain time. The weeds that emerge after two to three weeks then are treated with foliaceous agents. If you decide to do without the soil herbicide you logically have to wait until the weed has grown so that a camera can detect it. Let’s assume that we use the spot spraying method to apply a foliaceous agent on the emerged weeds: The problem is that the foliaceous agents affect the development of the cultivated plant. You cannot avoid spraying on it if the weed for example is close to the beet. Moreover, new weeds are constantly emerging all along. So the question is: How often do we have to spot a certain area to make sure that for example a beet field remains clean? We have not yet tried to do without the soil herbicide. But in my opinion, for beets, it does not make sense to do without it.
A reasonable solution would be a combination, i.e. a soil herbicide as the first measure and the other treatments with camera-based spot spraying systems. Another rather exciting idea is to accept a certain economic threshold respectively to tolerate certain weeds that are classified by the camera as they do not pose a problem in the next season due to a cleverly chosen rotation or because they are easy to treat. I think this might provide the biggest savings potential. However, it still requires a lot of development as the weeds do not only have to be detected but also have to be classified.
You have just been talking about herbicides. Can you image other sectors where this
method could be used?
Theodor Leeb: In case of plant illnesses for example you could apply fungicides or even growth regulators in cereals in a site-specific way. However, this does not require such a finely itemised spot spraying system as in this case we are talking about a larger area. For this application we use our pulse system PrecisionSpray with a variable application rate per 3 metre boom section. But there are approaches to use the cameras to detect illnesses. The question rather is if it is not too late at that time. In my opinion, the approach via biomass and weather models would be more productive.Â
How does “Green in Brown” work?
Theodor Leeb: Together with a manufacturer from France we carried out tests. The method is based on a mere differentiation of colours, i.e. you have a camera image and analyse which pixels are green or brown, thus plant or field. The green areas are sprayed. This worked quite well, however this system is not of major importance in Central Europe as we usually carry out tillage and as the conditions are wet.
Could you explain this in more detail?
Theodor Leeb: On the high-yield sites stubble cultivation is usually carried out after the harvest to mix in the straw. After a few days or weeks volunteer crops and weeds emerge. I.e. the field more or less is green all-over. Spotting does not make sense as the plants are too close to each other. So you would have to treat the whole area and could not rely on point application. In dry regions where no-till farming is very common this is different. There is no tillage after the harvest. As it is very dry there are little weeds or catch crops. And in this case, you can – instead of spraying all over – work with a camera system in a targeted way for example to save costs when using glyphosate for spraying the individual plants.
In addition to Green on Brown and Green in Green there is another differentiation: offline and online methods. What I have described so far are online methods, i.e. the camera is mounted on the boom and while driving the system decides whether to spray or not. With offline methods you get the information by means of a previous scanning. You normally fly over the field with a drone that is equipped with a high-resolution camera and scan the field from a height of approx. 20 m. At the moment, an algorithm is used to differentiate weed from crop in the high-resolution photo. This system provides an application map with the sections that are to be sprayed. This information then is loaded into the terminal of the machine and the field is treated. It works similar to the application maps for fertilisation.
Together with a start-up company we also have made tests with offline system for quite some time. The system basically works, but there also are a few obstacles. For example if you want to spray you have to have current data. There is little point in flying over the field with the drone 14 days in advance as in the meantime the weed infestation will change. The other obstacle is a physical one. Because of the offset method the spots have to be larger to hit the weeds as the GPS tolerances of the drone and the sprayers add up. Larger spots in turn mean that the sprayed area is larger, too, and thus there is less savings potential.
The extremely high data volume is an additional challenge. Endless gigabytes per hectare are generated that are sent to a server for calculation. This often takes the current internet connections to their limits. On the other hand, the application maps have to be sent back to the terminal of the farmer. Depending on the number of polygons (spots) the current ISOBUS terminals only allow for applications maps with a size of less than 5 hectares.
This means: From a technical or technological point of view the offline method can be displayed. For practical use, however, some more time is required to optimise the processes. And first of all, we need solutions for the high data volume. We possibly even have to find a parallel solution to ISOBUS.
What is your summary?
Theodor Leeb: In my opinion, spot spraying is the next logical step to meet the future requirements with regard to Green Deal, environmental protection and sustainability. Consequently, we come from an all-over treatment over band application to small-area spot spraying. The objective always is to apply the agent only where it is really required. In this respect, a camera-based system – whether online or offline – can make a valuable contribution.
We are intensely working on an optimisation of these systems and we will continue to carry out tests to gather more experiences. Our task is to transfer everything that is possible from a technological point of view into practice in such a way that the farmer can safely and simply use these methods in his everyday working like. Thus, spot spraying can become another component with regard to the optimisation of conventional crop care. But I also see the limits of what is possible as in the field we do not have standard, industrial and constant conditions.
My summary is: nature still will be nature. And nature cannot be constrained by industrial or digital standards.