Fast detection and location of longan fruits using UAV images

Abstract In agriculture, fruit picking robots on the ground have difficulty adapting to the terrain conditions of mountain orchards and cannot pick longan fruit from tall longan trees. In this paper, aiming to allow picking of longan fruit by unmanned aerial vehicles (UAVs), a deep learning-based scheme to quickly and accurately detect and locate suitable picking points on fruit branches is proposed. The scheme includes a UAV fuzzy image preprocessing method, longan detection based on a convolutional neural network (CNN), red, green, blue and depth (RGB-D) information fusion and an accurate target location strategy. First, the UAV is equipped with an Intel Realsense D455 camera, which collects longan images from the front for training and testing the model. Second, the lightweight MobileNet backbone network is used to improve the performance of the You Only Look Once version 4 (YOLOv4) model in feature extraction. The results for the test set show that compared with the classical feature pyramid network (FPN), YOLOv3 and YOLOv4 models, this model reduces the computation, parameters and detection time of the model. Compared with MobileNet single-shot multibox detector (MobileNet-SSD) and YOLOv4-tiny, this model exhibits improved detection accuracy. Third, according to the target detection result map, a strategy is formulated to accurately determine the suitable picking point on the main branch of the result. Finally, the performance of the improved model and picking platform in the harvest scene is evaluated by performing picking experiments in a longan orchard. In summary, we fully exploit the advantages of the combination of UAVs, RGB-D cameras and CNNs to improve the speed and accuracy of target detection and location for longan picking by UAVs based on vision.