Oblivious yet High Performance Task Scheduling for Large Shared Clusters
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Data analytics in large scale clusters are gradually shifting from monolithic and centralized scheduling frameworks to distributed or hybrid scheduling frameworks. In these distributed or hybrid frameworks, task queues on workers have widely been adopted to reconcile the conflict of task placements by different cluster schedulers. While a lot of task scheduling policies are available for each worker, the impact of each policy on the task performance and the ultimate job performance is not well understood. Consequently, the choice of scheduling policy for task is usually quite \textit{ad hoc}, especially when the task runtime information is not available beforehand. This thesis explores the task queuing effect by examining and comparing different scheduling policies for workers. We present the design and implementation of a worker-level task scheduler, Runway, that is oblivious to the individual task runtime information while still provides high performance and fairness. We demonstrate Runway's effectiveness in reducing average task completion time while guaranteeing starvation-freedom through extensive evaluations. Results show that Runway can provide 5
Description
Advisor
Degree
Type
Keywords
Citation
Wu, Dingming. "Oblivious yet High Performance Task Scheduling for Large Shared Clusters." (2018) Master’s Thesis, Rice University. https://hdl.handle.net/1911/105585.