Accueil > FORUM TERATEC > Programme > Atelier 1

Forum TERATEC 2014
Atelier 1 - Mercredi 2 juillet 9h00 à 12h30
Evolutions des applications et des architectures HPC & Big Data Influences, confluences

Running Hadoop Map Reduce Jobs on Lustre

EMEA Lustre Solution Architect INTEL
Zhiqi TAO
ASMO Lustre Solution Architect INTEL

Abstract : This talk presents a tutorial of how to run Hadoop Map Reduce jobs on the Lustre file system and some results from Terasort and DFSIO benchmarks.

Hadoop is becoming increasingly popular on data analytics. The Map Reduce frame is an effective way to process lots of data and aggregate the results. Many researchers in the HPC field are interested in leveraging the Hadoop Map Reduce frame to solve the problems that also are involved with lots of data processing.

However, the default Hadoop deployment mode uses Hadoop File System (HDFS) and therefore requires local storage, which present a challenge in the HPC data centers as many HPC computing infrastructure has little or none local storage, e.g. blade systems.

The Intel Lustre team developed a Hadoop Lustre adapter to solve this challenge. In 2013 the Intel Engineer Omkar Kulkarni presented the first prototype of the Hadoop Adapter for Lustre and the early work.

This year Intel has integrated and expanded this effort in the HPC distribution for Hadoop. We like to present the tutorial of how to make Hadoop Map Reduce Jobs running on Lustre in details. Not only will we present the benefits, but we will also discuss the areas that the Hadoop Lustre Adapter might be useful for HPC customers.


Biographie : Gabriele PACIUCCI
Ing. Gabriele Paciucci is a solution architect in the High Performance Data Division at Intel. In this role, Ing. Paciucci provides technical consultations to partners and customers and evangelizes the Lustre technology worldwide. Gabriele joined Intel in 2013. Previously, Gabriele was a senior software engineer specialized in HPC and Cloud solution based on Open Source Software. He has architected a number of high performance computing storage solutions based on Lustre on a variety of hardware platforms since 2006. Gabriele is involved in several Open Source projects and has promoted the adoption of Open Source software and Linux in the Enterprise since 2000 when he worked as software engineer at Red Hat. Ing. Paciucci received his Master Degree in Chemical Engineering from “Università degli Studi di Roma La Sapienza” in 1999.


Biographie : Zhiqi TAO


fleche La participation à cet atelier est gratuite sous réserve des places disponibles
fleche L'enregistrement en ligne est obligatoire pour y assister. Cliquer ICI pour vous inscrire.  



© Ter@tec - Tous droits réservés - Mentions légales