Fault tolerance for high-level parallel and distributed stream processing in C++ - Alf, Lucas Machado (2025)
Stream processing is a computing paradigm that addresses the gathering, processing, and analysis of a high-volume heterogeneous continuous data stream, aiming to extract valuable information in real-time. Considering the need for these systems to run for long periods, possibly indefinitely, reprocessing all data in case of failure can be highly costly or even unfeasible. Thus, it is essential for a stream processing system not only to recover after a failure but also to ensure that the generated results are correct. SPar is an annotation-based C++ domain-specific language designed to simplify the development of stream processing applications for multiple parallel architectures. In distributed architectures, SPar generates code using DSParLib, which does not provide fault tolerance mechanisms or strong message delivery guarantees. Other parallel programming alternatives in the literature, such as MPI, are low-level APIs with restrictive support for implementing resilient parallel applications. Most state-of-the-art distributed parallel programming tools for streaming applications with resilience support are available in Java. Given these factors, the main objective of this master’s thesis is to investigate fault tolerance and exactly-once semantics for streaming systems so that it is possible to support it on the SPar software ecosystem without rewriting the user source code. To this end, we created ResiPipe, a C++ library for fault-tolerant distributed stream processing that became a runtime system from the SPar ecosystem to execute parallel code. The results demonstrate that ResiPipe delivers performance comparable to, and in some cases superior o, other analyzed stream processing libraries, in addition to requiring fewer source lines of code (SLOC) per application and presenting a reduced estimated development time, according to Halstead’s method