Fault-tolerance for task-based runtime systems
Nicolas Ducarton va répéter son exposé Compas'26.
Speaker
Nicolas Ducarton
When
Thursday, June 26, 11:00
Where
room Alan Turing
Title
Fault-tolerance for task-based runtime systems
Abstract
Massively parallel and heterogeneous architectures can efficiently be exploited using task-based runtime systems. They are used for ever longer-running prob- lems, and on ever larger supercomputers. Using more nodes for longuer leads to a heightened risk of failure, making fault-tolerance capabilities for such systems a desirable addition. To this end, StarPU, a state of the art runtime system, has started including checkpoint-restart mechanisms. StarPU checkpoints use a buddy mechanism to save data to volatile memory. To tolerate failures, the runtime system must survive and replace failed processes.
The MPI Forum has recently started including the User-Level Failure Mit- igation (ULFM) extension in the MPI Standard to enable the writing of fault- tolerant applications. ULFM defines a set of routines that help support opera- tions after failures, and changes and the behavior of MPI when failures occur. Using ULFM, StarPU can survive failures. It must thus be adapted to this spec- ification.
In this work we present this adaptation, and our implementation of the restart mechanism in StarPU. StarPU performs a local restart: only the failed process restarts from the last checkpoint. To restore the missing messages to the new process, a message logging mechanism is needed. The messages are replayed from the checkpoint to the moment the failure happened. MPI communicators are replaced, processes are reordered and the work can continue normaly.
These operations are transparent to the application running on top of StarPU, which continues submission as the failure is repaired. The runtime system also continues working as long as the dependencies for submitted tasks are satisfied. We then present implementation challenges raised by this mechanism, notably on the process of handling requests on old communicators, and switching to re- placements while still accepting new requests submitted by the application on its original communicators. We validate our approach with a working implemen- tation in StarPU.