A Fine-grained, Real Time HPC Resource Management System

In contrast to the rapid development of capability construction, resource management of national meteorological high performance computers is left behind. Absence of operational software in resource management keeps system administrators from having a detailed knowledge of what's going on in national meteorological high performance computers and exerting effective control over resource allocations. Regarding existing problems, a fine grained, real time high performance computer resource management system is proposed. The system is designed to be a real time, fine grained one with cross cluster (Grid) support. The system works closely with CPU hours resources under keen competition. With the introduction of GCU (General Computing Unit), a resource virtualization unit, to measure computing resources, diversities of computing resources in different high performance computer systems are shielded and fine grained uniform quantitative management is enabled by the system. The target users of the system include resource users, leaders of user organizations, resource system administrators, decision makers etc. The system comprises three layers, namely, user interfaces, resource management, and high performance computer systems. Resource management layer, the primary layer, can be divided into resource accounting and allocation manager, Grid plat form, and resource information database. With open source software from supercomputing centers abroad, Grid project funded by MOST, and RDBMS employed, the system has seen an implementation, deployment and experimental running in National Meteorological Information Center. Fundamental functions of resource accounting and allocation management have been implemented, including cluster system job accounting, resource accounts management, management, allocation and query of user and organizations, providing command line interface for users. PostgreSQL database technology is adopted as the resource information database, on which accounts, users, organizations, computer systems, job records, accounting and allocation relation tables are created. The software system has been deployed into the three partitions of IBM high performance computer system, Sunway 32I cluster, Sunway 32P cluster, IBM SP system, working with LoadLeveler, PBS. Information of users on national meteorological high performance computer systems have been sorted and updated, resulting in uniform UID and GID, and inserted into databases. Two layers of management, organizations (projects) and individuals, are established. Computing resources are evenly allocated to user organizations according to 200 percent of the total available resource in terms of GCUs. Only resources allocated to their department can be used by individual users. The validity of resources are set to a season. Overdraft is allowed. Based on partial data collected during experimental run, initial statistical analyses are made to probe resource usage by user organizations and individuals. At present, the high performance computer resource system has been put into operational run and successfully applied to operation management.