Wageningen University on a quest to develop the best robotic grippers

During the recent Agro Food Robotics parcourse, the Agro Food Robotics unit of Wageningen University & Research (WUR) showcased 14 different precision agriculture and robotics projects they’re working on. With a legacy going back some 25 years in among others image recognition and processing and robotics, they have (co) developed solutions such as the Sweeper sweet pepper harvesting robot, the Plantalyzer yield prognosing robot and the Steketee IC-Weeder.

Currently, WUR, within the framework of several public private projects (PPS), is developing robotic platforms and grippers for harvesting fruits such as apples and pears and for pruning and thinning pear trees and red currant bushes.

_{Text continues below image}

Multifunctional platform development

Dr. Jochen Hemming, senior researcher computer vision & robotics in horticulture at Wageningen University & Research lead the development of the Sweeper harvesting robot and is currently working on a PPS project called ‘The Next Fruit 4.0’. “This project of WUR, ABB, Munckhof, NFO, RIWO and the Washington Tree Fruit Research Commission looks into various precision agriculture and robotic developments for orchards and one of them is a multifunctional robot platform”, says Hemming.

“Multifunctional because the past years and decades we saw a lot of projects, both from research institutes and startups, focussing on a specific action or operation in open field production such as robotic harvesting and that proves to be very challenging. First of all because of the short harvest season. In The Netherlands, apples for instance generally have to be harvested within 6 to 8 weeks. And secondly because of the seasonal pattern. You can only test your concept locally once a year. Another aspect is that investors, especially venture capitalists, demand a very short-term success and payback time.”

These developments involve amounts of money even Amazon, Apple, Facebook, Google and Microsoft are not willing to spend

Significant risks

“These developments involve significant risks and very, very deep pockets. Massive amounts of money that even big tech companies like Amazon, Apple, Facebook, Google and Microsoft are not willing to spend. It has led to bankruptcies such as those of Abundant Robotics (despite it had raised a total of $12 million of venture capital for development) and Dutch Cerescon. We therefore develop a multifunctional robot platform which’ autonomous platform, robotic arms and sensors can be used to make it economically feasible for more circumstances. With specialised end-effectors and grippers we make it suitable for different operations such as harvesting, pruning, disease detection and yield forecasting.”

_{Text continues below image}

Challenging circumstances

Hemming sees that outdoor farming circumstances are particularly challenging for commercially available industrial components. “That not only relates to light, shade, rain, dust and wind (moving objects) but also to natural variations in open field crops and trees. In the end, an outdoor robot is compiled of different components such as a platform, the robotic arm, the sensors/cameras and the gripper. Integrating these into affordable solutions that can cope with natural variations along with the complex software and AI developments remains to be challenging, also for us as WUR. Yet I believe we have the benefit of know-how in agriculture and horticulture compared to other universities and institutions.”

He illustrates the challenges in agriculture by means of an example: “The sensor and camera market is developing at an incredibly rapid pace as a result of the automotive industry. However, ‘their’ sensors such as radar, LiDAR and visual cameras are required to work within a 10 to like 30 metres range whereas farming, especially robotic harvesting, requires a 5 cm to 1 metre range.”

“Another challenge is the combination of information and detection we require such as the location, the colour and the ripeness of fruits. Or the difference between a one year or two-year-old branch or twine to prune. If I look back though, I see the tremendous improvement and advantages of current day deep learning algorithms compared to classical vision technology for image recognition and processing of especially ripe cucumbers around the year 2000.”

_{Text continues below video}

‘Pruning robot is the holy grail’

The researchers still consider (robotic) hand eye coordination as particularly challenging. In one of the projects they focus on developing a gripper for picking apples and pears and pruning trees and bushes. “Picking an apple or pear differs a lot. An apple can be located and picked – pulled – to harvest but for harvesting a pear, you need to localise the centre of the pear and its stem and then kind of bend the fruit upward. You can’t just pull it. Very apparent for a human being, but very challenging for a robot!”

The bottleneck isn’t the speed of sensing but the moving speed of the robotic arm

The researchers currently manage to pick a single apple in 10 seconds (and in one go/try) compared to 20 or even 30 seconds and multiple goes/tries a year ago. “The bottleneck isn’t the speed of sensing but the moving speed of the robotic arm and the speed with which it can create the right amount of suction power (vacuum). By finetuning the operation of the robotic arm, we can now operate the gripper and the arm much, much faster.”

Hemming sees that commonly used Intel RealSense stereo depth cameras are technically and financially feasible for many robotic operations but fail to be successful in detecting and localising rather thin objects. “They have trouble detecting thin objects such as twines to prune when there’s clutter or background noise from other trees in a next row with the same colours and structures. We are currently developing an end-effector for pruning red currant bushes because of the involvement of the Nederlandse Fruittelers Organisatie/Dutch Fruit Growers Association (NFO) and red currant growers. I see this as a prelude for pruning larger (pear) trees in future.”

I have to admit that I underestimated the complexity of robotic pruning

“While working in the second year of this four-year project, I have to admit that I underestimated the complexity of robotic pruning. Everything comes together in this application: 3D sensing of small structures, accessibility of the target with a robot arm and powerful tool in a very dense environment, complex rules of where and what to prune. For that reason, I consider developing a pruning robot as the holy grail in agro-robotics.”

“Depending on the availability of funding and the involvement and dedication of big multinational tech companies, the developments could accelerate and result in commercial availability in 5 to 10 years’ time. Companies such as Japanese car component manufacturer Denso (who acquired a minority stake in Dutch horticultural company Certhon in 2020) and robot manufacturers ABB and KUKA are all anxiously following the developments and trends in agriculture and horticulture!”

_{Text continues below images}

Gripper for picking soft fruits

While Jochen Hemming is working on a multifunctional robotic platform, his colleague Dr. Ali Leylavi Shoushtari, researcher at the Wageningen University & Research, Farm Technology Group, is working on a soft gripper as part of the Dutch Soft Robotics program. The soft agro-food robotics project is aimed to design versatile robots, grippers made out of soft materials in order to deal with the natural variability in food product such as shape, size and softness.

During the WUR Agro Food Robotics parcourse, Leylavi Shoushtari showcased a first design of a soft gripper capable of picking crops with high variations such as apples, oranges, mandarin, melons, cucumbers, tomatoes, peppers, broccoli, and lettuce.

We have to develop less powerful and less bulky solutions that can operate in crops with a lot of branches

Leylavi Shoushtari: “Industrial grippers are not suitable for fruit and vegetable picking because they are designed for a fixed and predictable product. We have to address the uncertainty in agro food products and the high variability. To achieve this, we need to have grippers that are not only adaptive to various shapes and sizes, but also powerful enough to handle products with various weights and also able to handle dynamic loads during detachment of apples or pears while minimising the risk of crop damage. This means that we have to develop less powerful and less bulky solutions that can operate in crops with a lot of branches. In the ‘Universal Soft Robotic Harvester for Intercropping System (USOROH)’ project, we are trying to address these typical variability and safety challenges.”

“Adding adhesion to the outer surface of these soft grippers where it contacts with objects, would enhance the safety of the grasping. Imagine, if you could make your fingers and the palm of your hand sticky then you can pick up objects almost without squeezing them. But of course you need to make it non-sticky to release the object which means practically we need a controlled-adhesion surface.”

“Suction cups are one of the practical ways to provide controlled adhesion in industry. But having a single or an array of suction cups for instance are less suitable because they can apply quite big forces on a small surface that can lead to high mechanical stress on the fruit. We aim to develop a gripper with a programmable and controllable adhesion for which we can also look at an octopus-like solution: small suction cups on a soft pad.”

Sensorise the gripper and control the actuatio

The next step is to automate the grasping for which we need to sensorise the gripper and control the actuation, Leylavi Shoushtari says. “Sensors will tell the robot how much force is applied on the crops and the control of the air pressure (vacuum) allows the robot to regulate this force (according to the weight of the object). I believe our developments stand out because we target versatile and mechanically reliable multi-crop solutions for an affordable pricing.”

His project ends by the end of this year. “If we manage to find a company interested in commercialising our solution, then it could become commercially available within a year.”

Jochen Hemming adds that soft grippers are challenging when it comes to sufficient operating speeds and crop hygiene and health. “Soft grippers offer a lot of possibilities but are a bit more difficult to (keep) clean and to prevent them from transferring diseases from one plant or tree to another.”