Deep learning-based reconstruction of shredded documents

• Ano de defesa: 2022
• Autor(a) principal: Paixão, Thiago Meireles
• Orientador(a): Santos, Thiago Oliveira dos
• Banca de defesa: Boldt, Francisco de Assis , Hirata Junior, Roberto , Britto Junior, Alceu de Souza
• Tipo de documento: Tese
• Tipo de acesso: Acesso aberto
• Idioma: por
• Instituição de defesa: Universidade Federal do Espírito Santo; Doutorado em Ciência da Computação
• Programa de Pós-Graduação: Programa de Pós-Graduação em Informática
• Departamento: Centro Tecnológico
• País: BR
• Palavras-chave em Português: Reconstrução de documentos fragmentados; Avaliação de compatibilidade; Problema de quebra cabeça; Otimização combinatorial
• Área do conhecimento CNPq: Ciência da Computação
• Link de acesso: http://repositorio.ufes.br/handle/10/15982

The reconstruction of shredded documents is a relevant task in various domains, such as forensic investigation and history reconstruction. As an alternative for the manual reconstruction, researchers have been investigating ways to perform (semi-)automatic digital reconstruction. Despite the several works on this topic, dealing with real-shredded data is a very sensitive issue in the current literature. Two research directions are addressed in this thesis to face this scenario: properly evaluating the fitting of shreds (the bulk of this work) and integrating the human into the reconstruction process. Regarding the fitting (compatibility) evaluation, it was verified that traditional pixel based approaches are not robust to real shredding, while more sophisticated techniques compromise significantly time performance. This thesis presents two deep learning self supervised approaches that have achieved state-of-the-art accuracy in more realistic/complex scenarios involving several real-shredded documents where the shreds are mixed (multi-page reconstruction or multi-reconstruction). The first approach models the compatibility evaluation as a two-class (valid or invalid) pattern recognition problem. The second approach, based on deep metric learning, proposes decoupling feature extraction from compatibility evaluation to improve scalability (time performance) for large reconstruction instances. Human interaction is explored to improve the accuracy of automatic methods. A critical issue regarding this topic is that the proposed methods do not scale well for large instances (real scenario), either because the user has the entire responsibility of arranging the shreds, or because he/she has to visualize the reconstruction and designate the shreds to be analyzed. In face of this challenge, we propose a human-in-the-loop framework that automatically selects potential mistakes (wrong pairings) in the solution for user analysis.