Towards simulating semantic onboard UAV navigation

In recent years the field of robotic navigation has increasingly harnessed semantic information in order to facilitate the planning and execution of robotic tasks. The use of semantic information focuses on employing representations more understandable by humans to accomplish tasks with robustness against environmental change, limiting memory requirements and improving scalability. Contemporary computer vision algorithms extracting semantic information have continuously improved their performance on benchmark datasets, however, most computations are expensive, limiting their use for robotic platforms constrained by size, weight and power such as unmanned aerial vehicles (UAVs). Recent advances have demonstrated the potential for navigation systems based on semantic information to be included into real-time operation of UAVs. This paper describes the development of a processing pipeline to incorporate the use of semantic information into a UAV navigation system. A navigation framework that uses the Robot Operating System (ROS) and semantic information is being developed, with simulations as a primary evaluation mechanism, preceeding deployment on hardware. The proposed system takes inputs from RGB images generated on-board the UAV and processes them in real-time to generate a semantic representation of its environment. The UAV executes subsequent actions autonomously by reasoning about the semantic content of the environment in order to accomplish a goal. Results from simulation indicated that the system is capable of extracting semantic information from camera images alone and infer plausible motion inputs for the flight controller to execute. The results also show that the system is capable of processing data in real-time and is able to enhance navigational capabilities to drive UAVs towards a higher level of autonomy.