Software overhead in messaging layers

Abstract

Despite improvements in network interfaces and software messaging layers, software communication overhead still dominates the hardware routing cost in most systems. In this study, we identify the sources of this overhead by an- alyzing software costs of typical communication protocols built atop the active messages layer on the CM-5. We show that up to 50-70% of the software messaging costs are a direct consequence of the gap between specific network fea- tures such as arbitrary delivery order, finite buffering, and limited fault-handling, and the user communication require- ments of in-order delivery, end-to-end flow control, and reli- able transmission. However, virtually all of these costs can be eliminated if routing networks provide higher-level ser- vices such as in-order delivery, end-to-end flow control, and packet-level fault-tolerance. We conclude that significant cost reductions require changing the constraints on messag- ing layers: we propose designing networks and network in- terfaces which simplify or replace software for implementing user communication requirements.