• INRIA PhD Student: Pierre Riteau

The goal of this project was to study the scalability and dynamic extension of sky computing infrastructures using large-scale testbeds. We leveraged the FutureGrid test bed in the USA and the Grid'5000 test bed in France to create large MapReduce (Hadoop) virtual clusters. We experimented with virtual clusters consisting of more than a thousand processor cores. We used the Nimbus cloud toolkit to manage virtual machines, and leveraged the dynamic extension feature of Hadoop to grow a virtual cluster with new resources.

We enhanced the Nimbus middleware with fast propagation mechanisms based on the TakTuk/Kastafior broadcast tool and Xen Copy-on-Write images. This allowed us to drastically increase propagation performance and thus make dynamic extension possible in a short time.