A scalable realtime analytics pipeline and storage architecture for physiological monitoring big data

Abstract Physiological monitors produce data essential to patient care. While the data is routinely observed and used at the bedside, it is rarely recorded permanently. Majority of data, especially high-resolution EKG waveforms, is discarded immediately. As the roles of big data and analytics in medicine are evolving, the streaming output of physiological monitors offers a potential source of highly relevant data for decision making and research. The inherently high volume and velocity of physiological data pose unique practical challenges for collection, storage, and analysis. A successful solution has to enable consistent and constant connectivity for streaming and to provide adequate storage and access capabilities. The solution also needs to meet security and privacy requirements for medical data. We propose a scalable, distributed architecture that leverages open-source stream processing software to connect raw monitoring output to a file storage system and an integration with existing data warehouse and data retrieval systems. Analytics pipeline has a potential to provide real-time feedback to the clinicians, including critical event detection or even prediction. The combination of real-time analysis and distributed storage of physiological big data, previously discarded, now opens up possibilities for future applications relevant to both clinicians and researchers alike. We have built and tested the first version of the continuously streaming data pipeline, from bedside physiological monitors to the storage. The results are promising, with raw or analyzed and enriched data already finding their place in supporting a variety of research use cases.