FloraVision: A Spatial Crowd-based Learning System for California Native Plants

With the availability of massive amounts of visual data covering wide geographical regions, various image learning applications have emerged, including classifying the street cleanliness level, detecting forest fires or road hazards. Such applications share similar characteristics as they need to 1) detect specific objects or events (what), 2) associate the detected object with a location (where), and 3) know the time that the event happened (when). Advancements in image-based machine learning (ML) benefit these applications as they can automate the detection of objects of interest. Along with the edge computing (EC) paradigm, the processing cost is offloaded to the devices, hence reducing latency and communication cost. Moreover, sensors on the edge devices (e.g., GPS) enrich the collected data with metadata. However, a shortcoming of existing approaches is that they rely on pre-trained "static" models. Nonetheless, crowdsourced data at diverse locations can be leveraged to iteratively improve the robustness of a model. We refer to the aforementioned strategy as "spatial crowd-based learning".To showcase this class of applications, we present FloraVision, an end-to-end system that integrates ML, crowdsourcing, and EC to automate the detection, mapping, and exploration of California Native Plants. FloraVision implements a pipeline to collect and clean publicly available image data, train a lightweight MobileNet-based classification model, and then deploy the model on mobile devices. It leverages spatial crowd-based learning to iteratively evolve the initial model from crowdsourced data. Its mobile application facilitates detecting plants and mapping their geolocations. Finally, it allows end-users to submit ad hoc spatio-temporal nearest neighbor queries and visualizes the results in an augmented reality user interface. Although our application focuses on plants, several other applications follow similar architectural patterns.