x

Project Title: Scalable I/O Middleware and File System Optimizations for High-Performance Computing

Supported by National Science Foundation’s HECURA program Award No: CCF-0621443

University: Northwestern University and Penn State University

Investigator= s: (PI) Alok Choudhary (NWU), (Co-PI) Rajeev Thakur (NWU, ANL) and (Co-PI) Mahmut Kandemir (PSU)

<= /o:p>

Publications: http://www.ece.northwestern.edu/~choudhar/publications/<= /span>

<= /span>

<= span style=3D'mso-bidi-font-size:11.0pt'>Project Summary

Ever increasing research and developmen= t in science and engineering is based on simulations and/or on analysis of observational data requiring the use of high-performance computers. High-end systems with 10s to 100s of thousands of processors are now availabl= e, enabling very large-scale applications. Most of such applications are extre= mely data intensive. Data-intensive applications come from all types of domains including astrophysics, climate and operational weather modeling, nanoscien= ce, materials, combustion, cosmology, bioinformatics and drug design, nuclear simulations, sensor data processing, security, and others. Data that is produced, consumed, ha= ndled, visualized, analyzed and mined is reaching hundreds of terabytes to petabyt= es range (raw data, derived images, movies, streams, metadata, mined data etc.= ) for most application domains. Thus, along with computational capabilities, scalable software for I/O, and storage for the required capacity and performance must be developed in order to address the= data intensive nature of applications and reap benefits in performance and productivity of these large-scale systems. For data-intensive applications, I/O an= d storage layers are extremely critical, and are often overlooked, but they become a bottleneck in not only obtaining scalable performance but also in utilizati= on and productivity of systems and application scientists.

Technical Mer= it: This proposal entails research and development to address several parallel I/O problems in the HECURA initiative. In particular, the main goals of this proposal are to design and implement novel I/O middleware techniques and optimizations, parallel file system techniques that scale to ultra-scale systems, design and development of techniques that efficiently enable newer APIs including suggested extensions to POSIX for parallelism, and flexible = I/O benchmarks that mimic real and dynamic I/O behavior of science and engineer= ing applications. The fundamental premise is that, to achieve extreme scalability, incremental changes or adaptation of traditional techniques for scaling data accesses and I/O will= not succeed because they are based on pessimistic and conservative assumptions = of parallelism and interactions. We propose innovative techniques to optimize data accesses that utilize the understand= ing of high-level access patterns (“intent”), and use that informat= ion through middleware and file systems to enable optimizations. Specifically, = the objectives are to (1) design = and develop middleware I/O optimizations and cache system that are able to capt= ure small, unaligned, irregular I/O accesses from large number of processors and uses access pattern information to optimize for I/O; (2) incorporate these optimizations in MPICH2’s MPI-IO implementation to make them availabl= e to a large number of users; (3) design scalable parallel file system techniques and optimizations including a versioning parallel file system, programmable= and adaptable consistency semantics (which can be invoked by the middleware bas= ed on its understanding of applications’ I/O access patterns), layout optimizations, and self-tuning capabilities; (4) design and evaluate enhanc= ed APIs for file system scalability, particularly for recently proposed enhancement= s to the POSIX interface (API) to enable high-performance parallel I/O; and (5) = develop flexible, execution oriented and scalable I/O benchmarks that mimic = the I/O behavior of real science, engineering and bioinformatics applications..=

Broader Impact: In addition to the above descri= bed technical contributions, we expec= t to have significant impact in several ways. We have developed close collaborat= ions with several DOE laboratories (e.g., ANL, SNL, LBNL, etc.) and industries s= uch as SUN and IBM (letters are attached).This collaboration has enabled us not= only to identify important problems faced on real systems for scalable applicati= ons, but we have regularly provided our collaborators with the software we have developed in our current and prior research. We have identified specific wa= ys of technology transfer to these organizations of the results from the propo= sed research including internships for graduate students and incorporation of s= oftware techniques in public domain releases. Co-PI Thakur is the primary architect= and developer of ROMIO implementation of MPI-IO and one of the key developers of MPICH2, (widely used communication and I/O software in parallel systems). We will directly impact and enable application scientists to benefit from the results of our research by incorporating our results into the MPICH2 implementation. We will also provide the software to companies and obtain feedback. Most of our graduate students go to DOE labs or industry during summer for such technology trans= fer, and we will continue with this model. The PI, Choudhary, has graduated four women PhDs, and Co-PI Kandemir has graduated two women and one minority graduate students. We will continue our commitment to promote underrepresen= ted groups. The PIs regularly give tutorials at major conferences. For example, Choudhary and Thakur have given tutorials on I/O at many SCxx conferences. Thakur also regularly gives tutorials at other conferences and supercomputi= ng centers, to train scientists and practitioners. We will continue this pract= ice and incorporate results from this research for our future tutorials. <= /o:p>

<= /o:p>

Recent Highlights:

<= /o:p>

<= o:p>

<= /o:p>

<= /span>

This figure demonstrates the effectiveness a= nd performance enhancements provided by Middleware Caching Schemes proposed in this HECURA project. Initial results show that Caching performance provides a factor of two or more performance improvement in I/O as compared to traditional middleware optimizations. Performance is shown using the BTIO benchmark.

Project Description:

The objectives of this project are to (1) design and develop middleware I/O optimizations and cache system that are able to capture small, unaligned, irregular I/O accesses from large number of processors and uses access patt= ern information to optimize for I/O; (2) incorporate these optimizations in MPI= CH2’s MPI-IO implementation to make them available to a large number of users; and (3) develop flexible, execution oriented and scalable I/O benchmarks that mimic= the I/O behavior of real science, engineering and bioinformatics applications <= /p>

Discovery:

Ever increasing research and development in science and engineering is based on simulations and/or on analysis of observational data requiring the use of high-performance computers. High-end systems with 10s = to 100s of thousands of processors are now available, enabling very large-scale applications. Most of such applications are extremely data intensive. Data-intensive applications come from all types of domains including astrophysics, climate and operational weather modeling, nanoscience, materi= als, combustion, cosmology, bioinformatics and drug design, nuclear simulations, sensor data processing, security, and others. Data that is produced, consumed, handled, visualized, analyzed and mined is reaching hundreds of terabytes to petabytes range (raw data, derived images, movies, streams, metadata, mined data etc.) for most application domains. Thus, along with computational capabilities, scalable software for I/O, and stora= ge for the required capacity and performance must be developed in order to add= ress the data intensive nature of applications and reap benefits in performance = and productivity of these large-scale systems. For data-intensive applications, I/O and storage layers are extremely critical, and are often overlooked, but they become a bottleneck = in not only obtaining scalable performance but also in utilization and productivity of systems and application scientists.

So far, through out interactions with scientists we ha= ve discovered that their productivity is adversely affected due to the poor performance of I/O systems and software. We have also shown, using prelimin= ary results from this recently started project that significant improvements ca= n be made in this performance by providing the right interface and optimizations= for parallel I/O. One of the scientists’ application is already starting = to use the initial results from this project showing tremendous performance ga= ins.

Learning

Although it has been only six months since the incepti= on of this project, the project currently supports a graduate student who is work= ing towards her PhD. She has already started developing techniques and software optimizations as outlined in the project description. She has also started interacting with Application scientists in national laboratories and plans = to go for a summer internships to one of the national laboratories. In additio= n, the project supports part of a senior personnel’s time who has been evaluating performance of the techniques developed in this project on production machines in various supercomputing centers using production applications.

<= o:p>

Research Infrastructure:

Although it has been only six months since the incepti= on of this project, we have developed MPI-IO caching software which has been prov= ided to Argonne National Laboratories to be incorporated into the MPI-IO which is part of the MPICH-2 distribution. This is the most widely used MPI software= in the world. Our software is available for Beta testing and will be available= in the full release over the next few months.