Improving the Efficiency of Map-Reduce Task Engine
| dc.contributor.advisor | Cox, Alan L. | en_US |
| dc.contributor.committeeMember | Rixner, Scott | en_US |
| dc.contributor.committeeMember | Sarkar, Vivek | en_US |
| dc.creator | Chadha, Mehul | en_US |
| dc.date.accessioned | 2016-01-06T21:00:20Z | en_US |
| dc.date.available | 2016-01-06T21:00:20Z | en_US |
| dc.date.created | 2014-12 | en_US |
| dc.date.issued | 2014-10-03 | en_US |
| dc.date.submitted | December 2014 | en_US |
| dc.date.updated | 2016-01-06T21:00:20Z | en_US |
| dc.description.abstract | Map-Reduce is a popular distributed programming framework for parallelizing computation on huge datasets over a large number of compute nodes. This year completes a decade since it was invented by Google in 2004. Hadoop, a popular open source implementation of Map-Reduce was introduced by Yahoo in 2005. Over these years many researchers have worked on various problems related to Map-Reduce and similar distributed programming models. Hadoop itself has been the subject of various research projects. The prior work in this field is focussed on making Map- Reduce more efficient for iterative processing, or making it more pipelined across different jobs. This has resulted in an improvement of performance for iterative applications. However, little focus was given to the task engine which carries out the Map-Reduce computation itself. Our analysis of applications running on Hadoop shows that more than 50% of the time is spent in the framework in doing tasks such as sorting, serialization and deserialization . We solve this problem introducing an extension to the Map-Reduce programming model. This extension allows us to use more efficient data structures like hash tables. It also allows us to lower the cost of serialization and deserialization of the key value pairs. With these efforts we have been able to lower the overheads of the framework, and the performance of certain important applications such as Pagerank and Join has improved by 1.5 to 2.5 times. | en_US |
| dc.format.mimetype | application/pdf | en_US |
| dc.identifier.citation | Chadha, Mehul. "Improving the Efficiency of Map-Reduce Task Engine." (2014) Master’s Thesis, Rice University. <a href="https://hdl.handle.net/1911/87731">https://hdl.handle.net/1911/87731</a>. | en_US |
| dc.identifier.uri | https://hdl.handle.net/1911/87731 | en_US |
| dc.language.iso | eng | en_US |
| dc.rights | Copyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder. | en_US |
| dc.subject | Hadoop | en_US |
| dc.subject | Map-Reduce | en_US |
| dc.subject | Pagerank | en_US |
| dc.subject | Join | en_US |
| dc.subject | Barrier free | en_US |
| dc.title | Improving the Efficiency of Map-Reduce Task Engine | en_US |
| dc.type | Thesis | en_US |
| dc.type.material | Text | en_US |
| thesis.degree.department | Computer Science | en_US |
| thesis.degree.discipline | Engineering | en_US |
| thesis.degree.grantor | Rice University | en_US |
| thesis.degree.level | Masters | en_US |
| thesis.degree.name | Master of Science | en_US |
Files
Original bundle
1 - 1 of 1