Collaborative energy demand response with decentralized actor and centralized critic

The ongoing industrialization and rising technology adoption around the world are leading to ever higher energy consumption. The benefits of electrification are enormous, but the growing demand also comes with challenges with respect to associated greenhouse gas emissions. Although continuing progress in energy research has brought up new technologies in energy generation, storage, and distribution, most of those technologies focus on increasing efficiency of individual components. Work on integration and coordination abilities between individual components in micro-grids will lead to further improvements and gains in efficiency that are necessary to reduce carbon footprints and slow down climate change. To this end, the CityLearn environment provides a simulation framework that allows the control of energy components in buildings that are organized in districts. In this paper, we propose an energy management system based on the decentralized actor-critic reinforcement learning algorithm MARLISA but integrate a centralized critic and call it MARLISADSCC. In this way, we are training a model to autonomously control the energy storage of individual buildings in a CityLearn district to improve demand response guided by a better informed training signal. We show performance increases over baseline control techniques for a district but also discuss the resulting action selection for individual buildings.