Browsing by Author "Wu, Dingming"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Designing Hybrid Data Center Networks for High Performance and Fault Tolerance
    (2020-01-08) Wu, Dingming; Ng, T. S. Eugene
    This thesis explores the design space of hybrid electrical and optical network architectures for modern data centers. It tries to reach a delicate balance between performance, fault-tolerance, scalability and cost through coordinated use of both electrical and optical components in the network. We have developed several approaches to achieving these goals from different angles. First, we used optical splitters as key building blocks to improve multicast transmission performance. We built an unconventional optical multicast architecture, called HyperOptics, that provides orders of magnitude of throughput improvement for multicast transmissions. Second, we developed a failure tolerant network, called ShareBackup, by embedding optical switches into the Clos networks. Sharebackup, for the first time, achieves network-wide full-capacity failure recovery in milliseconds. Third, we proposed to enable programmable network topology at runtime by inserting optical switches at the network edge. Our system, called RDC, breaks the bandwidth boundaries between servers and dynamically optimizes its topology according to traffic patterns. Through these three works, we demonstrate the high potential of hybrid datacenter network architectures for high performance and fault-tolerance.
  • Thumbnail Image
    Item
    Oblivious yet High Performance Task Scheduling for Large Shared Clusters
    (2018-06-11) Wu, Dingming; Ng, T. S. Eugene
    Data analytics in large scale clusters are gradually shifting from monolithic and centralized scheduling frameworks to distributed or hybrid scheduling frameworks. In these distributed or hybrid frameworks, task queues on workers have widely been adopted to reconcile the conflict of task placements by different cluster schedulers. While a lot of task scheduling policies are available for each worker, the impact of each policy on the task performance and the ultimate job performance is not well understood. Consequently, the choice of scheduling policy for task is usually quite \textit{ad hoc}, especially when the task runtime information is not available beforehand. This thesis explores the task queuing effect by examining and comparing different scheduling policies for workers. We present the design and implementation of a worker-level task scheduler, Runway, that is oblivious to the individual task runtime information while still provides high performance and fairness. We demonstrate Runway's effectiveness in reducing average task completion time while guaranteeing starvation-freedom through extensive evaluations. Results show that Runway can provide 5$\times$ task performance improvement, and 42\% job performance improvement under high load compared to the state-of-art solution.