Proactive fault tolerance for HPC with Xen virtualization
Top Cited Papers
- 17 June 2007
- proceedings article
- Published by Association for Computing Machinery (ACM)
Abstract
Large-scale parallel computing is relying increasingly on clusters with thousands of processors. At such large counts of compute nodes, faults are becoming common place. Current techniques to tolerate faults focus on reactive schemes to recover from faults and generally rely on a checkpoint/restart mechanism. Yet, in today's systems, node failures can often be anticipated by detecting a deteriorating health status. Instead of a reactive scheme for fault tolerance (FT), we are promoting a proactive one where processes automatically migrate from "unhealthy" nodes to healthy ones. Our approach relies on operating system virtualization techniques exemplified by but not limited to Xen. This paper contributes an automatic and transparent mechanism for proactive FT for arbitrary MPI applications. It leverages virtualization techniques combined with health monitoring and load-based migration. We exploit Xen's live migration mechanism for a guest operating system (OS) to migrate an MPI task from a health-deteriorating node to a healthy one without stopping the MPI task during most of the migration. Our proactive FT daemon orchestrates the tasks of health monitoring, load determination and initiation of guest OS migration. Experimental results demonstrate that live migration hides migration costs and limits the overhead to only a few seconds making it an attractive approach to realize FT in HPC systems. Overall, our enhancements make proactive FT a valuable asset for long-running MPI application that is complementary to reactive FT using full checkpoint/restart schemes since checkpoint frequencies can be reduced as fewer unanticipated failures are encountered. In the context of OS virtualization, we believe that this is the first comprehensive study of proactive fault tolerance where live migration is actually triggered by health monitoring.Keywords
This publication has 20 references indexed in Scilit:
- Cooperative checkpointingPublished by Association for Computing Machinery (ACM) ,2006
- Proactive Fault Tolerance in MPI Applications Via Task MigrationLecture Notes in Computer Science, 2006
- Self-migration of operating systemsPublished by Association for Computing Machinery (ACM) ,2004
- Xen and the art of virtualizationPublished by Association for Computing Machinery (ACM) ,2003
- The Google file systemPublished by Association for Computing Machinery (ACM) ,2003
- Critical event prediction for proactive management in large-scale computer clustersPublished by Association for Computing Machinery (ACM) ,2003
- The performance of μ-kernel-based systemsPublished by Association for Computing Machinery (ACM) ,1997
- Manetho: transparent roll back-recovery with low overhead, limited rollback, and fast output commitIEEE Transactions on Computers, 1992
- Transparent process migration: Design alternatives and the sprite implementationSoftware: Practice and Experience, 1991
- Fine-grained mobility in the Emerald systemACM Transactions on Computer Systems, 1988