Liu, ZhaoleiNg, T. S. Eugene2017-08-022017-08-022016-03-25Liu, Zhaolei and Ng, T. S. Eugene. "Leaky Buffer: A Novel Abstraction for Relieving Memory Pressure form Cluster Data Processing Frameworks." (2016) https://hdl.handle.net/1911/96417.https://hdl.handle.net/1911/96417The shift to the in-memory data processing paradigm has had a major influence on the development of cluster data processing frameworks. Numerous frameworks from the industry, open source community and academia are adopting the in-memory paradigm to achieve functionalities and performance breakthroughs. However, despite the advantages of these in memory frameworks, in practice they are susceptible to memorypressure related performance collapse and failures. The contributions of this paper are two-fold. Firstly, we conduct a detailed diagnosis of the memory pressure problem and identify three preconditions for the performance collapse. These preconditions not only explain the problem but also shed light on the possible solution strategies. Secondly, we propose a novel programming abstraction called the leaky buffer that eliminates one of the preconditions, thereby addressing the underlying problem. We have implemented the leaky buffer abstraction in Spark. Experiments on a range of memory intensive aggregation operations show that the leaky buffer abstraction can drastically reduce the occurrence of memory-related failures, improve performance by up to 507% and reduce memory usage by up to 87.5%.13 ppengYou are granted permission for the noncommercial reproduction, distribution, display, and performance of this technical report in any format, but this permission is only for a period of forty-five (45) days from the most recent time that you verified that this technical report is still available from the Computer Science Department of Rice University under terms that include this permission. All other rights are reserved by the author(s).Technical reportTR15-03