Saliency-based methods for automated video cropping in sidewalk footage

• Ano de defesa: 2025
• Autor(a) principal: Costa, Suayder Milhomem
• Tipo de documento: Dissertação
• Tipo de acesso: Acesso aberto
• Idioma: eng
• Instituição de defesa: Biblioteca Digitais de Teses e Dissertações da USP
• Palavras-chave em Português: Infraestrutura urbana; Pavimento tátil; Predição de saliência; Recorte de vídeos; Salience prediction; Tactile paving; Urban infrastructure; Video cropping
• Link de acesso: https://www.teses.usp.br/teses/disponiveis/45/45134/tde-13092025-192333/

The condition of urban infrastructure is an important aspect in ensuring the safety and well-being of pedestrians. This is especially important when considering accessibility for individuals with mobility challenges, such as the elderly and visually impaired, who are especially vulnerable to poorly maintained sidewalks. Areas surrounding hospitals are of particular concernnot only due to the high volume of pedestrian and vehicle traffic, but also because they serve individuals in vulnerable health conditions who require safe and reliable access to medical services. In this context, many computational tools have already demonstrated their potential for urban infrastructure analysis, including surface material classification and obstacle detection; however, most solutions require labeled data, which is costly and time-consuming. To address this gap, two strategies for salience prediction in videos are proposed, aiming to reduce the dependence on manual labeling and contribute to sidewalk analysis. Both strategies ultimately serve to train saliency predictors tailored to specific sidewalk features. The first strategy leverages human visual attention by converting user clicks into attention maps through post-processing. This approach is particularly effective for identifying general sidewalk obstacles, such as cracks and surface defects. The second employs the \\acf model, enhanced with additional processing, to generate labeled video data more efficiently for specialized tactile features. This enables the training of saliency predictors that can recognize key elements of tactile paving, including directional changes and broken tiles. An important advantage of this strategy is its scalability -- it can potentially be extended to detect a wider range of features in the urban environment. These saliency models serve as the foundation for a proposed video cropping method designed to automatically trim frames and highlight the most relevant regions based on saliency map outputs. This approach enables the identification of key areas within each frame and supports applications such as content-aware video retargeting, object-focused attention, and sidewalk condition analysis by emphasizing defects and potential hazards. This research compiles our previous studies \\citep{costa2024videocropping, costa2024tactile, costa2025salience} presenting the following main contributions: (1) development of a click-based video annotation tool, (2) an annotated dataset of egocentric videos in sidewalk footage tailored for saliency prediction, (3) implementation of two saliency prediction strategies for sidewalk video cropping, (4) training and evaluation of saliency models for sidewalk structure analysis, and (5) integration of these models into a video cropping framework. Experimental results demonstrate that the saliency models effectively highlight relevant information in urban environments, achieving an AUC of 0.582 for human-based attention and 0.914 for tactile-based attention, thereby enhancing assistive technologies for visually impaired individuals.