A deep reinforcement learning approach for voltage and frequency control of microgrids with power hardware-in-the-loop validation

• Ano de defesa: 2025
• Autor(a) principal: Barbalho, Pedro Inácio de Nascimento e
• Tipo de documento: Tese
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Biblioteca Digitais de Teses e Dissertações da USP
• Palavras-chave em Português: Aprendizado profundo por reforço; controle hierárquico; controle de microrredes; Deep reinforcement learning; hardware-in-the-loop; hierarchical control; microgrid control
• Link de acesso: https://www.teses.usp.br/teses/disponiveis/18/18154/tde-10032025-155823/

Throughout the modernisation of the distribution system, distributed energy resources appeared, characterised by their proximity to loads and local control. In this context, microgrids (MGs) emerged to coordinate these resources with local loads, improving energy efficiency and the reliability of the grid. The MG controllers must adapt to different operating conditions without losing performance. However, due to the MGs non-linear dynamics, classical controllers cannot operate correctly if the system moves far from equilibrium. Therefore, researchers have explored adaptive algorithms, such as those based on deep reinforcement learning (DRL). This Thesis proposed an approach based on DRL to control the voltage and frequency of islanded MGs. Existing solutions tackled this problem by tuning the parameters of traditional controllers, usually with a large deep neural network, which demands a large computational effort. Thus, the present Thesis proposed a framework for optimised training of the DRL-based controller, which directly operated the MG. The proposed framework is based on i) MG modelling, ii) environment agent interaction, iii) reward function design, iv) DRL algorithm parametrisation, v) simulation tests, vi) controller hardware-in-the-loop (HIL) tests, and vii) power HIL validation. The MG was modelled using the discrete phasor models of the MG components. The discrete phasor model allows the agent to be trained with the MG model, preserving essential dynamics for the DRL algorithm and requiring low computational effort during training. The environment-agent interactions allow us to select proper DRL-based controller input/output and which scenarios should be used for training. Afterwards, designing the reward function is also critical, as the DRL-based controllers learning relies on receiving proper feedback from training in a way that learns the best actions for each operating condition. This Thesis also proposes a strategy for DRL algorithm parametrisation and simulation tests to ensure that the controller generalises its learning. The proposed approach enables DRL-based voltage and frequency control of MGs, feasible for real-world implementations with low computational cost and generalise their learning for critical operating scenarios.