An Efficient Parallel Reinforcement Learning Approach to Cross-Layer Defense Mechanism in Industrial Control Systems

The ongoing digitalization enables stable control processes and smooth operations of Industrial Control Systems (ICSs). A direct consequence of the highly interconnected architecture of ICSs is the introduced cyber vulnerability and increasing cyber security threats to ICSs. Numerous researches pay attention to the security problem of ICSs. However, most current studies face two challenges. First, the interaction problem between the cyber layer and the physical layer of ICSs may result in incorrect attack response strategies. Second, ICSs are real-time systems, but existing defense decision algorithms based on game theory or reinforcement learning techniques have high computational complexity, which prevents them from making decisions quickly. In this paper, we design a new multi-attribute based method for quantifying rewards and propose a multi-attribute based Q-learning algorithm to resolve the interaction problem. In addition, to overcome the limitation of slow convergence, we develop an effective parallel Q-learning (PQL) algorithm to quickly find the optimal strategy. The experimental results show the effectiveness of the PQL algorithm. Compared with the Q-learning algorithm (QL) and the deep Q-network (DQN) algorithm, our proposed solution can reduce the average completion time by 12.5 to 37 percent.