49 Sight ++: Prototyping a computer vision guided assistive technology

Introduction The visually impaired in society are amongst the most impacted by social isolation restrictions of the COVID-19 pandemic, fueling research into assistive technologies, including devices primed for computer vision task-orientated image recognition. We present a Proof of Concept prototype modular system that uses Intel RealSense depth cameras connected to a modular ML inference platform to construct near field object information that guides and encourages exploration for users. Methods Early engagement from experts in the field of global disability enabled us to better appreciate orientation, mobility-related considerations, sensory components and meaningful voice instructions. We subsequently designed a novel modular, extensible platform that runs inference classification and depth detection on camera input, then uses heuristic AI Priortiser to analyse and identify essential guidance output for the users. Results Sight ++ uses object recognition to accurately inform users on near field objects, including data on proximity. By parsing the items through a series of environment rules, which results in objects having more or less relative importance, the system can output qualified audio guidance for obstacle avoidance and awareness. The use of OpenVINO resulted in a 2-fold increase in performance of our inference classifiers. We anticipate that a miniaturised depth camera would be fitted to a backpack adjustable strap to offer real-time object recognition and meaningful notification of artefacts of interest at waist height and above. Conclusion We have produced a robust, foundational assistive system which rather than replace recognised and trusted methods of navigation, introduces a new dimension of intelligence. Future versions will implement motion tracking of the objects, haptic feedback and a teleassistance function connecting a sighted volunteer to enhance guidance. We aspire that future improvements offer a seamless understanding of new environments and create a novel user experience for exploration that are designed for personalisability, social acceptability and social inclusion.