气象大数据云平台设计与实现

Design and Implementation of CMA Big Data and Cloud Platform

  • 摘要: 面向国家统筹集约化建设气象信息系统的要求和气象大数据服务的业务需求, 国家气象信息中心设计并建设了气象大数据云平台, 全面提升气象数据处理、加工与存储服务能力。气象大数据云平台采用流式数据处理、分布式异构数据存储、基于容器的算法调度和基于网关的统一接口服务等技术, 实现气象数据的快速标准化解码入库处理、高效存储与服务及算法高效集约调度运行。建立中试仿真环境, 构建气象业务软件云原生开发测试环境, 提供丰富的测试数据、开发环境和测试准入评估功能, 显著提升应用系统的云化效率与规范性。2021年气象大数据云平台在国省业务化运行, 支撑全国气象业务实时运行, 数据年访问次数为5.27×1010, 数据年访问量为135 PB, 实现8109个业务算法的稳定调度运行。基于气象大数据云平台, 数据孤岛逐步消除, 应用系统集约化程度明显提高, 数据应用时效提升2~10倍, 对提升业务系统的运行效率和协同性、推进“云+端”新型气象业务技术体制改革和实现气象业务集约化发展发挥了关键支撑作用。

     

    Abstract: The intensive development of meteorological operational systems and the reform of "cloud + end" technology framework are important measures to achieve high-quality development of meteorological services. In 2020, China Meteorological Administration proposed the establishment of a "cloud + end" operational technology system, designating CMA Big Data and Cloud Platform as the cloud and existing meteorological business system as the end components, clarifying the positioning of CMA Big Data and Cloud Platform as a key foundational technological infrastructure. The Platform leverages technologies such as stream data processing, distributed heterogeneous data storage, container-based algorithm scheduling, and gateway-based unified interface services. These enable rapid standardized decoding, storage, and servicing of meteorological data, efficient and centralized algorithm scheduling of computational tasks. The Platform consists of a data processing system, storage management system, product processing system, data interface system, and a pilot-test simulation environment. Among these, the data processing system utilizes technologies such as Storm stream data processing to achieve high-efficiency decoding and processing of meteorological data. The storage management system employs distributed heterogeneous storage technology to address different data formats and application scenarios, thereby facilitating efficient storage and application of massive meteorological data. The product processing system, which is structured around an algorithm library and processing pipeline, allows for unified management of meteorological algorithms and efficient container-based scheduling and execution of algorithms. The data interface system adopts unified interface service technology based on an API gateway, providing efficient, stable, and secure sharing services for meteorological data. The pilot-test simulation environment serves as a cloud-native development and testing environment for meteorological business software, offering abundant test data, development environments, and test admission evaluation functions, significantly improving the cloudification efficiency and standardization of application systems.The Platform was put into operational use at national and provincial levels in 2021, supporting real-time meteorological operations across the country. It handles 52.7 billion data access requests annually, with an annual data access volume of 135 PB, and ensures the stable scheduling and operation of 6287 business algorithms. The Platform has progressively eliminated data silos, significantly enhanced the integration of application systems, and improved data application efficiency by 2 to 10 times. It plays a vital supporting role in boosting the operational efficiency of meteorological systems, strengthening their collaboration, accelerating the reform of the "cloud + end" operational technology system, and promoting the intensive development of meteorological operations.The continuous advancement of new technologies, including artificial intelligence and digital twins, is expected to drive architectural innovation, technological upgrades, and functional expansion of the Platform. This evolution will establish the next-generation digital foundation for meteorological operations, namely Earth System Data Platform, to comprehensively support the digital transformation, intelligent upgrading, and smart services of meteorological operations.

     

/

返回文章
返回