Browsing by Author "Dinu, Florin"
Now showing 1 - 6 of 6
Results Per Page
Sort Options
Item Analysis of Hadoop’s Performance under Failures(2011-08-11) Dinu, Florin; Ng, T. S. EugeneFailures are common in today’s data center environment and can significantly impact the performance of important jobs running on top of large scale computing frameworks. In this paper we analyze Hadoop’s behavior under compute node and process failures. Surprisingly, we find that even a single failure can have a large detrimental effect on job running times. We uncover several important design decisions underlying this distressing behavior: the inefficiency of Hadoop’s statistical speculative execution algorithm, the lack of sharing failure information and the overloading of TCP failure semantics. We hope that our study will add new dimensions to the pursuit of robust large scale computing framework designs.Item Gleaning network wide congestion information from packet markings(2010) Dinu, Florin; Ng, T. S. EugeneCongestion information can greatly benefit network level decisions. For example, fast-reroute algorithms should leverage congestion information when computing backup paths. They could also use the information to monitor if the re-routing decision itself causes congestion in the network. Today, most solutions for inferring congestion work at the end-host level and relay end-to-end congestion information to transport protocols. Network level decisions, on the other hand, may need link level congestion information. Unfortunately, the mechanisms that routers can use to infer link level congestion information are insufficient. Such information could potentially be obtained by periodically sharing estimates between routers. However, this solution increases the traffic load on the network and has difficulty in reliably delivering the estimates during periods of congestion. In this thesis we show that routers inside an autonomous system can easily and accurately infer congestion information about each other. Routers first measure path level congestion information only from the congestion markings in the traffic that they forward. Next, we propose that routers combine routing information with the path level congestion information to obtain a more detailed description of the congestion in the network. Link level congestion information can be computed using this approach. Our techniques never add supplementary traffic into the network and use little router resources. They can be deployed incrementally or in heterogeneous environments. We show that the accuracy of the inference is good using experiments with multiple traffic patterns and various congestion levels.Item Gleaning Network-Wide Congestion Information from Packet Markings(2010-06-29) Dinu, Florin; Ng, T. S. EugeneDistributed control protocols routinely have to operate oblivious of dynamic network information for scalability or complexity reasons. However, more informed protocols are likely to make more intelligent decisions. We argue that protocols can leverage dynamic congestion information without suffering the mentioned penalties. In this paper we show that routers can readily exchange congestion information in a purely passive fashion using congestion markings from existing traffic. As a result, each router can locally infer a congestion map of the network. Moreover, the maps are continuously updated with near real-time information. Our solution for building the congestion maps leverages standardized and widely used congestion management protocols and does not require changes to end hosts. We find that 90% of the time, the inference accuracy is usually within 10% even for environments with multiple congestion points and sudden changes in the traffic pattern.Item RCMP: A System Enabling Efficient Re-computation Based Failure Resilience for Big Data Analytics(2013-04-30) Dinu, Florin; Ng, T. S. EugeneMulti-job I/O-intensive big-data computations can suffer a significant performance hit due to relying on data replication as the main failure resilience strategy. Data replication is inherently an expensive operation for I/O-intensive jobs because the datasets to be replicated are very large. Moreover, since the failure resilience guarantees provided by replication are fundamentally limited by the number of available replicas, jobs may fail when all replicas are lost. In this paper we argue that job re-computation should also be a first-order failure resilience strategy for big data analytics. Recomputation support is especially important for multi-job computations because they can require cascading re-computations to deal with the data loss caused by failures. We propose RCMP, a system that performs efficient job re-computation. RCMP improves on state-of-the-art big data processing systems which rely on data replication and consequently lack any dedicated support for recomputation. RCMP can speed-up a job’s re-computation by leveraging outputs that it stored during that job’s successful run. During re-computation, RCMP can efficiently utilize the available compute node parallelism by switching to a finer-grained task scheduling granularity. Furthermore, RCMP can mitigate hot-spots specific to re-computation runs. Our experiments on a moderate-sized cluster show that compared to using replication, RCMP can provide significant benefits during failure-free periods while still finishing multijob computations in comparable or better time when impacted by single and double data loss events.Item The Preliminary Design and Implementation of the Maestro Network Control Platform(2008-10-01) Cai, Zheng; Cox, Alan L.; Dinu, Florin; Ng, T. S. Eugene; Zheng, JieNetwork operation is inherently complex because it consists of many functions such as routing, firewalling, VPN provisioning, traffic load-balancing, network maintenance, etc. To cope with this, network designers have created modular components to handle each function. Un fortunately, in reality, unavoidable dependencies exist between some of the components and they may interact accidentally. At the same time, some policies are realized by compositions of different components, but the methods of composition are ad hoc and fragile. In other words, there is no single mechanism for systematically governing the interactions between the various components. To address these problems, we propose a clean-late system called Maestro. Maestro is an “operating system” that orchestrates the network control applications that govern the behavior of a network, and directly controls the underlying network devices. Maestro provides abstractions for the modular implementation of network control applications, and is the first system to address the fundamental problems originating from the concurrent operations of network control applications, namely communication between applications, scheduling of application executions, feedback management, concurrency management, and network state transition management. As the networking industry moves towards building directly controllable devices like the OpenFlow Switch, we believe Maestro can become a common platform.Item Understanding and Improving the Efficiency of Failure Resilience for Big Data Frameworks(2013-10-30) Dinu, Florin; Ng, T. S. Eugene; Cox, Alan L.; Knightly, Edward W.; Gkantsidis, ChristosBig data processing frameworks (MapReduce, Hadoop, Dryad) are hugely popular today because they greatly simplify the management and deployment of big data analysis jobs requiring the use of many machines in parallel. A strong selling point is their built-in failure resilience support. Big data frameworks can run computations to completion despite occasional failures in the system. However, an important but overlooked point has been the efficiency of their failure resilience. The vision of this thesis is that big data frameworks should not only be failure resilient but that they should provide the resilience in an efficient manner with minimum impact on computations both under failures as well as during failure-free periods. To this end, the first part of the thesis presents the first in-depth analysis of the efficiency of the failure resilience provided by the popular Hadoop framework under failures. The results show that even single machine failures can lead to large, variable and unpredictable job running times. This thesis determines the causes behind this inefficient behavior and points out the responsible Hadoop mechanisms and their limitations. The second part of the thesis focuses on providing efficient failure resilience for the case of computations comprised of multiple jobs. We present the design, implementation and evaluation of RCMP, a MapReduce system based on the fundamental insight that using data replication to enable failure resilience oftentimes leads to significant and unnecessary increases in computation running time. In contrast, RCMP is designed to use job re-computation as a first-order failure resilience strategy. Re-computations under RCMP are efficient. Specifically, RCMP re-computes the minimum amount of work and uniquely it ensure this minimum re-computation work is performed efficiently. In particular, RCMP mitigates hot-spots that affect data transfers during re-computations and ensures that the available compute node parallelism is well leveraged.