Hierarchical Multi-Agent Deep Reinforcement Learning Based Energy Management System for Active Distribution Networks

Abstract

The integration of distributed energy resources (DERs), such as photovoltaic (PV) systems and wind turbines, has transformed modern power systems. This shift to renewable energy requires advanced energy management systems (EMS) to handle DER complexities. Traditional unbalanced radial distribution systems, designed for unidirectional power flow, now face challenges in maintaining voltage profiles and economic efficiency. Few studies address low voltage (LV) residential distribution networks (DNs), particularly for multi-home energy management. Additionally, assumptions that all consumers can respond to dynamic pricing signals may not reflect reality. This thesis addresses these challenges through three key contributions. Firstly, it develops a bi-level optimal energy management (OEM) framework that simultaneously optimizes the operation of distribution system operators (DSOs) and smart home (SH) consumers in an unbalanced residential DN. Secondly, this research explores the impact of varying levels of home energy management system (HEMS) penetration on DSO profitability, consumer energy costs, and voltage regulation, with a focus on mitigating both undervoltage and overvoltage deviations. Thirdly, a novel hybrid control approach is also proposed for the bi-level OEM framework where the DSO exercises both direct control over non-smart home (NSH) consumers’ DERs and indirect control over SH consumers through dynamic pricing. This thesis discusses the impact of dynamic pricing signals by upper-level agent on lower-level EMS agents and assesses assumptions about agent capabilities. A comparative analysis of deep reinforcement learning (DRL) based local marginal price (LMP) generation versus direct dynamic pricing by the DSO examines their effects on energy consumption, cost efficiency, and voltage profile. The influence of different DRL agent-based EMS penetration levels within LVDNs on DSO agents’ techno-economic objectives is also explored. The IEEE European LV test feeder is used to validate the proposed models and strategies. The proposed framework integrates DSO-level and SH-level optimization, providing a coordinated approach to managing DR and DERs in LVDNs. This strategy enhances the economic performance of DSOs while maintaining grid stability. The analysis reveals that increasing HEMS penetration generally leads to better DSO profitability and consumer cost savings, with optimal performance observed at moderate penetration levels. These findings are critical for guiding the deployment of smart grid technologies. The thesis introduces and evaluates coordinated DR and active power curtailment techniques, demonstrating their effectiveness in mitigating both undervoltage and overvoltage issues in LVDNs. These strategies offer a more adaptive solution compared to traditional voltage regulation methods. The research highlights the relationship between price variability and pricing caps, showing that lower caps result in more stable energy costs for consumers and predictable revenue for DSOs. This insight is essential for designing effective dynamic pricing models that balance grid stability with economic fairness. The hybrid control method allows DSOs to directly manage NSH consumers’ DERs while using dynamic pricing to influence SH consumers. The hybrid control further enhances the DSO’s ability to optimize voltage profiles, offering a flexible solution adaptable to different levels of consumer participation. Overall, this thesis contributes to the advancement of energy management strategies in LVDNs, providing practical solutions for the challenges posed by increasing renewable energy integration. The methodologies developed here lay the groundwork for future research and practical applications aimed at achieving a reliable, efficient, and economically viable smart grid.