InterFS: An Interplanted Distributed File System to Improve Storage Utilization

Resource under-utilization is a common problem in modern data centers. Though researchers have proposed consolidation techniques to improve utilization of computing resources, there still lacks an approach to mitigate particularly low utilization of storage capacity in clusters for online services. A potential solution is to "interplant" a distributed storage system together with the services on these clusters to leverage the unused storage. However, avoiding performance interference with existing services is an essential prerequisite for interplanting. Thus, we propose InterFS, a POSIX-compliant distributed file system aiming at fully exploiting the storage resource on data center clusters. We adopt intelligent resource isolation, peak load dodging, and region-based replica placement schemes in InterFS. Therefore, it can be interplanted with other resource-intensive services without interfering with them, and amply fulfill the storage requirements of small-scale applications in the data center. Currently InterFS is deployed in 20,000+ servers at Baidu, providing 80 PB storage space to 200+ long-tailed services.