A MPI library performs two tasks---computation on behalf of the application, and communication in the form of sending and receiving messages among processes forming the application. Efficient communication is key to a high-performance MPI library, and the use of specialized interconnect technologies has been a common way to achieve this goal. However, these custom technologies lack the portability and simplicity of a generic communication solution like TCP over Ethernet. This thesis first shows that even though TCP is a higher overhead protocol than UDP, as a messaging medium it performs better than the latter, because of library-level reliability overheads with UDP. Then, the thesis presents a technique to separate computation and communication aspects of a MPI library, and handle each with the most efficient mechanism. The results show a significant improvement in performance of MPI libraries with this technique; bringing Ethernet closer to the specialized networks.