SikuliBot: automating physical interface using images

We present SikuliBot, an image-based approach to automating user interface. SikuliBot extends the visual programming concept of Sikuli Script[2] from the graphical UIs to the real world of physical UIs, such as mobile devices' touch-screens and hardware buttons. The key to our approach is using a physical robot to see an interface, identify a target, and perform an action on the target using the robot's actuators. We demonstrate working examples on MakerBot 3D printer that could move a stylus to perform multi-touch gestures on touchscreen to automate tasks such as swipe-to unlock, playing a virtual piano, and playing the Angry Bird game. A wide range of automation possibilities are made viable using a simple scripting language based on images of UI components. The benefits of our approach are: generalizability, instrumentation-free, and high-level programming abstraction.