A Collaborative Effort to Improve Lossy Compression Methods for Climate Data

Climate model simulations produce large volumes of data, and reducing the storage burden with data compression is increasingly of interest to climate scientists. A key concern to the climate community, though, is ensuring that any data loss due to compression does not in any way affect their scientific analysis. For this reason, the climate community is taking a cautious approach to adopting lossy compression by carefully investigating the potential existence of artifacts due to compression in a wide variety of analysis settings. Spatio-temporal statistical analysis in particular can highlight compression-induced features that would go unnoticed by the standard metrics common to the data compression community. Communicating such findings to the algorithm developers in the context of a collaborative improvement cycle is one – in our view productive – way to foster trust within the climate community and pave the way for eventual adoption of lossy compression. In this work, we report on the initial results of a successful and mutually beneficial collaboration between the two communities that led to improvements in a well regarded compression algorithm and more effective compression of climate simulation data.