The Evolution of Lustre File System and the Perspective of Application to the Meteorology Filed

Very high performance computer (HPC) systems are needed to run modern Numerical Weather Prediction (NWP) model. More and more meteorological applications, especially NWP Models have already been and will be run in these large scale cluster systems. As the HPC development has stepped into the mature phase, computing power is never a big problem any more. However, the data processing and data services are becoming a conspicuous issue, since a more powerful HPC would demand and generate much more data. One of the key elements in a HPCs environment to address the issue is the Cluster File System Technology.To improve the HPC's comprehensive utilization and the operational efficiency for meteorological applications, when data processing becomes a restrictive factor, several problems must be solved, such as, how the data can be efficiently moved into or out of a HPC, how a large application can be input or large amount of output data be generated fast; how the data can be exchanged or shared effectively among multiple HPCs? Cluster File System is the answer, and Lustre is one of the best Cluster File System solutions currently in the market. The advantages are as follows. First, Lustre is designed to be a very flexible, scalable and stable file system. In practice, it can be configured with a large variety of machines, as well as different network technologies; the number of nodes can range from several to tens of thousands. Second, Lustre software experiences for over 6 years in many important HPC environments, including the largest lab in the Department of Energy (DOE) in America. It has been widely developed, tested and then put into production for some highest mission-critical applications. During the latest one to two years, it has been recognized by the HPC world and successfully adopted by a majority of Linux based Clusters. One of the core technologies of Lustre is Object Storage which is usually implemented as Object Based Storage Devices (OSD) that aims at achieving both high performance and cross-platform features by offering an entirely new way of abstracting storage-objects. The concept of Object Storage is implemented by Lustre by introducing the Metadata Server (MDS) which is both the hardware and the software component in a Lustre Cluster.A couple of HPCs are currently maintained by National Meteorological Information Center for China Meteorological Administration users. A concept framework is proposed that is designed to establish a globally unified parallel file system by which multiple Linux clusters can be spaned in our environment, hence the operational workflow is optimized and the utilization of the storage resources among the clusters is improved. The global Business Continuity of mulitple HPC clusters can also be greatly improved with the help of this framework.