Co-optimizing battery storage for energy arbitrage and frequency regulation in real-time markets using deep reinforcement learning

Miao, Y., Chen, T. , Bu, S., Liang, H. and Han, Z. (2021) Co-optimizing battery storage for energy arbitrage and frequency regulation in real-time markets using deep reinforcement learning. Energies, 14(24), e8365. (doi: 10.3390/en14248365)

Abstract

Battery energy storage systems (BESSs) play a critical role in eliminating uncertainties associated with renewable energy generation, to maintain stability and improve flexibility of power networks. In this paper, a BESS is used to provide energy arbitrage (EA) and frequency regulation (FR) services simultaneously to maximize its total revenue within the physical constraints. The EA and FR actions are taken at different timescales. The multitimescale problem is formulated as two nested Markov decision process (MDP) submodels. The problem is a complex decision-making problem with enormous high-dimensional data and uncertainty (e.g., the price of the electricity). Therefore, a novel co-optimization scheme is proposed to handle the multitimescale problem, and also coordinate EA and FR services. A triplet deep deterministic policy gradient with exploration noise decay (TDD−ND) approach is used to obtain the optimal policy at each timescale. Simulations are conducted with real-time electricity prices and regulation signals data from the American PJM regulation market. The simulation results show that the proposed approach performs better than other studied policies in literature.

• Item Type: Articles
• Additional Information: .
• Keywords: Battery energy storage, energy arbitrage, frequency regulation, real-time market, deep reinforcement learning.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Miao, Yushen and Chen, Tianyi
• Authors: Miao, Y., Chen, T., Bu, S., Liang, H., and Han, Z.
• College/School: College of Science and Engineering > School of Engineering
• Journal Name: Energies
• Publisher: MDPI
• ISSN: 1996-1073
• ISSN (Online): 1996-1073
• Copyright Holders: Copyright © 2021 The Authors
• First Published: First published in Energies 14(24):e8365
• Publisher Policy: Reproduced under a Creative Commons licence