Oblivious yet High Performance Task Scheduling for Large Shared Clusters

Date
2018-06-11
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract

Data analytics in large scale clusters are gradually shifting from monolithic and centralized scheduling frameworks to distributed or hybrid scheduling frameworks. In these distributed or hybrid frameworks, task queues on workers have widely been adopted to reconcile the conflict of task placements by different cluster schedulers. While a lot of task scheduling policies are available for each worker, the impact of each policy on the task performance and the ultimate job performance is not well understood. Consequently, the choice of scheduling policy for task is usually quite \textit{ad hoc}, especially when the task runtime information is not available beforehand. This thesis explores the task queuing effect by examining and comparing different scheduling policies for workers. We present the design and implementation of a worker-level task scheduler, Runway, that is oblivious to the individual task runtime information while still provides high performance and fairness. We demonstrate Runway's effectiveness in reducing average task completion time while guaranteeing starvation-freedom through extensive evaluations. Results show that Runway can provide 5× task performance improvement, and 42% job performance improvement under high load compared to the state-of-art solution.

Description
Degree
Master of Science
Type
Thesis
Keywords
Cluster Scheduling, Task Scheduling, Big data frameworks, Non-clairvoyant Scheduling
Citation

Wu, Dingming. "Oblivious yet High Performance Task Scheduling for Large Shared Clusters." (2018) Master’s Thesis, Rice University. https://hdl.handle.net/1911/105585.

Has part(s)
Forms part of
Published Version
Rights
Copyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.
Link to license
Citable link to this page