Tracking and recognition algorithm for a robot harvesting oscillating apples

Apple fruits on trees tend to swing because of wind or other natural causes, therefore reducing the accuracy of apple picking by robots. To increase the accuracy and to speed up the apple tracking and identifying process, tracking and recognition method combined with an affine transformation was proposed. The method can be divided into three steps. First, the initial image was segmented by Otsu’s thresholding method based on the two times Red minus Green minus Blue (2R-G-B) color feature; after improving the binary image, the apples were recognized with a local parameter adaptive Hough circle transformation method, thus improving the accuracy of recognition and avoiding the long, time-consuming process and excessive fitted circles in traditional Hough circle transformation. The process and results were verified experimentally. Second, the Shi-Tomasi corners detected and extracted from the first frame image were tracked, and the corners with large positive and negative optical flow errors were removed. The affine transformation matrix between the two frames was calculated based on the Random Sampling Consistency algorithm (RANSAC) to correct the scale of the template image and predict the apple positions. Third, the best positions of the target apples within 1.2 times of the prediction area were searched with a de-mean normalized cross-correlation template matching algorithm. The test results showed that the running time of each frame was 25 ms and 130 ms and the tracking error was more than 8% and 20% in the absence of template correction and apple position prediction, respectively. In comparison, the running time of our algorithm was 25 ms, and the tracking error was less than 4%. Therefore, test results indicate that speed and efficiency can be greatly improved by using our method, and this strategy can also provide a reference for tracking and recognizing other oscillatory fruits. Keywords: apple picking robot, tracking and recognition algorithm, oscillating apple, Hough transform, pyramid LK optical flow algorithm, affine transform, template matching DOI: 10.25165/j.ijabe.20201305.5520 Citation: Yang Q H, Chen C, Dai J Y, Xun Y, Bao G J. Tracking and recognition algorithm for a robot harvesting oscillating apples. Int J Agric & Biol Eng, 2020; 13(5): 163–170.