Deep Reinforcement Learning Based Grid-Forming Inverter

Abstract

This research work introduces Deep Deterministic Policy Gradient (DDPG), a type of Reinforcement Learning (RL), for grid modeling, estimating voltage and phase angle, and control method for grid-forming inverters. The aim is to develop a grid-forming inverter that sets the voltage level and frequency of the grid and mitigates voltage dips originating from faults and frequency deviation fluctuations. Unlike conventional methods for estimating setpoints for the controller loops, we do not need several chains of estimation tools such as Fast Fourier Transform (FFT), Synchronous Reference Frame (SRF), or lowpass filters. With the DDPG, we also optimize the phase lock-loop (PLL) and accurately deliver the angle for the actuation part of the inverter to generate the given reference signal. The developed method does not need exhaustive tuning of parameters such as coefficients of PID controllers and lowpass filters. We observe that the proposed method has a faster response time than the PID-based control unit (15ms compared to 50ms) for the grid-forming inverter in the case of compensating voltage dips. We also observed that the DDPG-based grid-forming inverter is more efficient in compensating continuous voltage variations and frequency deviations than a trivial PID-based version.