loading...
Efficient Structured Data Access in Parallel File Systems
Hong Kong December 01-December 04
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/CLUSTR.2003.1253331Fifth IEEE International Conference o ...
 This Article 
 
PDF
HTML
 
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Avery Ching, Northwestern University
Alok Choudhary, Northwestern University
Wei-keng Liao, Northwestern University
Robert Ross, Argonne National Laboratory
William Gropp, Argonne National Laboratory

Parallel scientific applications store and retrieve very large, structured datasets. Directly supporting these structured accesses is an important step in providing high-performance I/O solutions for these applications. High-level interfaces such as HDF5 and Parallel netCDF provide convenient APIs for accessing structured datasets, and the MPI-IO interface also supports efficient access to structured data. However, parallel file systems do not traditionally support such access.

In this work we present an implementation of structured data access support in the context of the Parallel Virtual File System (PVFS). We call this support "datatype I/O" because of its similarity to MPI datatypes. This support is built by using a reusable datatype-processing component from the MPICH2 MPI implementation. We describe how this component is leveraged to efficiently process structured data representations resulting from MPI-IO operations. We quantitatively assess the solution using three test applications. We also point to further optimizations in the processing path that could be leveraged for even more efficient operation.

Citation:
Avery Ching, Alok Choudhary, Wei-keng Liao, Robert Ross, William Gropp, "Efficient Structured Data Access in Parallel File Systems," cluster, pp.326, Fifth IEEE International Conference on Cluster Computing (CLUSTER'03), 2003
Usage of this product signifies your acceptance of the Terms of Use.