Evaluating LLMs for multimodal GUI test generation in Android applications

• Ano de defesa: 2025
• Autor(a) principal: FAGUNDES, Nayse da Silva
• Orientador(a): TEIXEIRA, Leopoldo Motta
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Universidade Federal de Pernambuco
• Programa de Pós-Graduação: Programa de Pos Graduacao em Ciencia da Computacao
• País: Brasil
• Palavras-chave em Português: GUI; Testes; LLMs
• Link de acesso: https://repositorio.ufpe.br/handle/123456789/68466

Graphical User Interface (GUI) testing is a fundamental task in mobile application development, as it ensures that the user interface of any mobile application behaves cor rectly and meets user expectations. However, when performed manually, GUI testing remains time-consuming. With the rise of Large Language Models (LLMs), there is in creasing interest in exploring their potential to automate software development tasks, including the generation of GUI tests. This study investigates how LLMs can generate GUI test intentions and scripts for Android applications using multimodal inputs, such as screenshots and structured UI data, which provide both visual and semantic informa tion about the interface. This work present an approach that combines these inputs from open-source Android apps and evaluate the performance of four LLMs, including three proprietary models and one open-source model. The results show significant differences among the models, where the Claude 3 Sonnet model produced the most complete results, GPT-4o generated smaller tests focusing on essential flows, while Gemini 2.5 Pro and the open-source Gemma 3 model presented similar results, limiting themselves to basic in teractions. Overall, the results demonstrate that LLMs models show potential to reduce manual effort and increase productivity in GUI test creation, offering distinct benefits according to the model used.