Data processing with cross-application interference control via system-level instrumentation - Xavier, Miguel Gomes (2019)
World’s gigantic data collection is reaching a crucial point for significant technological changes to deal with the immense variety and performance needs. While the complexities of data have been increasing, the real impact depends on the ability of extraction and transformation of these massive and varied raw data sets to uncover valuable information. Gaining insights into this information has led to the area of Big Data analytics. Data analysis has represented a major challenge in designing highly scalable resource management systems to integrate, extract and transform data into information, while maintaining users’ experience and business’ expectation. Resource management systems for Big Data generally consolidate applications and use system-level virtualization (containers) to enable resource sharing and improve efficiency, but performance still vary unpredictably due to the competition in access to shared resources like CPU, memory, disk and network. The initial intuition motivating our work is that the system-level information availability could be used to classify the interference emanate from containerized applications. We therefore conjecture that interference-aware clusters may speed up applications to accelerate Big Data analytics and improve resource-efficiency, while maintaining users’ experiences and business’ expectations. To confirm our thesis, we first studied Big Data performance needs and existing container system performance isolation weaknesses. Gaining insight from these studies to propose an container placement with interference recognition, we put all these together in an interference-aware scheduler prototype, which resulted in gains of up to 35% in scheduling performance and 42% in resource efficiency, thus, confirming the thesis.