PM1-8 : Development of a Sensor System for Agricultural Machines using Stereo Vision and Deep Learning

Autonomous navigation of agricultural machines by employing the global navigation satellite system (GNSS) has developed rapidly in recent times. However, a machine that is only based on GNSS cannot detect obstacles such as humans, which will cause an increased risk of collision with such obstacles. Further, conventional distance sensors cannot accurately determine the distances to various obstacles because of grasses or crops that may lie between the sensors and obstacles, blocking the sight of the sensor. To overcome this problem, we have developed a sensor system that can precisely determine the distances from the sensors to humans even in the aforementioned circumstances. We combine human detection, based on deep learning, with distance detection by means of a stereo camera. Human detection with deep learning can be used to obtain an RGB image from the stereo camera in order to classify obstacles and to detect their locations in the image. When humans are detected, the detection image is compared with a depth image, and the location in the distance image is determined. Further, the median of the distance values corresponding to the pixels at the detected location is calculated. Using this sensor system, we measure the distances from the sensors to a human who was standing in a vegetated region. The errors are 2.2, 4.9, and 14.5 cm, respectively, for distances of 2, 3, and 4 m from the camera. The results depicted that this sensor system exhibits sufficient accuracy in case of agricultural machines.