一个精细粒度实时计算资源管理系统

A Fine-grained, Real Time HPC Resource Management System

  • 摘要: 由于相应业务系统软件的缺乏,国家级气象高性能计算机的资源管理措施落后于能力建设的发展。对此,该文提出了一个精细粒度实时计算资源管理系统。系统设计紧密围绕着目前竞争最为激烈的计算资源,采用资源虚拟单元GCU作为资源使用的计量单位,屏蔽了不同高性能计算机系统的体系结构差异,实现了计算资源细粒度的统一量化统计。系统可分为用户接口层、资源管理层、HPC系统层等3个层次,根据与网格平台软件不同结合方式以两种方式运行。在国家气象信息中心完成了系统的研发、部署和试验运行,根据试验运行的部分数据进行了用户单位和用户个人的计算资源使用的统计分析。目前,计算资源管理系统成果已成功应用到国家级气象高性能计算机计算资源的业务管理工作中。

     

    Abstract: In contrast to the rapid development of capability construction, resource management of national meteorological high performance computers is left behind. Absence of operational software in resource management keeps system administrators from having a detailed knowledge of what's going on in national meteorological high performance computers and exerting effective control over resource allocations. Regarding existing problems, a fine grained, real time high performance computer resource management system is proposed. The system is designed to be a real time, fine grained one with cross cluster (Grid) support. The system works closely with CPU hours resources under keen competition. With the introduction of GCU (General Computing Unit), a resource virtualization unit, to measure computing resources, diversities of computing resources in different high performance computer systems are shielded and fine grained uniform quantitative management is enabled by the system. The target users of the system include resource users, leaders of user organizations, resource system administrators, decision makers etc. The system comprises three layers, namely, user interfaces, resource management, and high performance computer systems. Resource management layer, the primary layer, can be divided into resource accounting and allocation manager, Grid plat form, and resource information database. With open source software from supercomputing centers abroad, Grid project funded by MOST, and RDBMS employed, the system has seen an implementation, deployment and experimental running in National Meteorological Information Center. Fundamental functions of resource accounting and allocation management have been implemented, including cluster system job accounting, resource accounts management, management, allocation and query of user and organizations, providing command line interface for users. PostgreSQL database technology is adopted as the resource information database, on which accounts, users, organizations, computer systems, job records, accounting and allocation relation tables are created. The software system has been deployed into the three partitions of IBM high performance computer system, Sunway 32I cluster, Sunway 32P cluster, IBM SP system, working with LoadLeveler, PBS. Information of users on national meteorological high performance computer systems have been sorted and updated, resulting in uniform UID and GID, and inserted into databases. Two layers of management, organizations (projects) and individuals, are established. Computing resources are evenly allocated to user organizations according to 200 percent of the total available resource in terms of GCUs. Only resources allocated to their department can be used by individual users. The validity of resources are set to a season. Overdraft is allowed. Based on partial data collected during experimental run, initial statistical analyses are made to probe resource usage by user organizations and individuals. At present, the high performance computer resource system has been put into operational run and successfully applied to operation management.

     

/

返回文章
返回