北京全球信息系统中心全球数据缓存设计与应用

Design and Application of GISC Beijing Global Cache

  • 摘要: 随着气象数据量呈指数级增长,世界气象组织信息系统(World Meteorological Organization Information System,WIS)/全球电信系统(Global Telecommunication System,GTS)架构难以契合未来全球气象数据实时交互需求,2023年气象大会开启以世界气象组织信息系统2.0(World Meteorological Organization Information System 2.0,WIS2.0)取代WIS/GTS的进程,中国气象局向世界气象组织(World Meteorological Organization,WMO)承诺建设WIS2.0的四大全球服务系统,全球数据缓存(global cache)作为其中之一,负责实时收集汇聚全球数据,并提供数据共享服务,应用数据主动发现技术,构建高性能、低时延的消息收集传输模型;采用最优路径汇集和基于特征信息的识别清洗技术,实现数据最短时间内的汇聚;构建标准化的主题层次结构,发布全球数据和元数据到达的消息通知,为全球用户提供高吞吐、高并发的数据下载服务。全球数据缓存2024年10月通过WMO测试,2025年业务运行,截至2025年5月从全球82个世界气象组织信息系统2.0节点(简称WIS2节点)获取数据,缓存数据量约为2 TB·d-1,支撑全球WIS2节点日平均数据下载次数超过2×106 d-1,为中国气象局“全球监测、全球预报、全球服务”提供地球系统多圈层数据的支撑保障。

     

    Abstract: The exponential growth of meteorological data poses significant challenges, rendering the existing WIS(World Meteorological Organization Information System)/GTS(Global Telecommunication System) framework increasingly inadequate to meet the future demands of real-time global meteorological data interaction. In response, World Meteorological Organization (WMO) has initiated the crucial process of replacing WIS/GTS with the advanced WMO Information System 2.0 (WIS2.0) in 2023. CMA(China Meteorological Administration) has committed to constructing 4 essential global service systems that support WIS2.0. The key to this transformation is the global data cache system, which has been meticulously developed by Beijing GISC (Global Information System Center). Its primary objective is to significantly enhance the global capacity for collecting meteorological data. This sophisticated system integrates essential functionalities, including proactive data discovery, streamlined and efficient downloading mechanisms, and intelligent cache processing. A comprehensive analysis of the construction background offers valuable insights into the system’s design philosophy. The global data cache’s overall architecture is introduced, including its core functionalities and the underlying technical implementation. The global cache leverages the message notification-driven global data active discovery technology inherent in MQTT (message queuing telemetry transport) protocol to establish a high-performance, low-latency, and highly available model for message collection and transmission. It analyzes the global meteorological data sharing mechanism, extracts core fields for feature value analysis to perform data cleansing, identifies and eliminates duplicates through sorting and comparison of similar elements, and employs file storage technology to ensure data integrity while providing a data access interface for the back-end system. Additionally, a standardized thematic hierarchy is established, handling notifications for the arrival of global data and metadata. It offers global users with high-throughput, high-concurrency, and high-timeliness data services. In October 2024, after extensive tests conducted by the expert group of WMO, the global data cache established by Beijing GISC successfully passed all functional and performance evaluations. Making China the first country among the six global data caches to fully meet the requirements of the global data cache assessments. Currently, the system actively ingests data from 82 WIS2 nodes distributed across the globe. It is a volume of approximately 2 TB of cached data daily, while supporting an average of over 2 million data download requests per day from 20 WIS2 nodes worldwide. In the future, the global data cache is expected to gradually take on the essential functions that were previously managed by GTS. By leveraging the powerful big data cloud platform Tianqing. The system will deliver enhanced services to the national meteorological department, providing robust, full-spectrum support for the CMA’s strategic vision of “global monitoring, global forecasting, global service” Concurrently, efforts will be intensified to actively expand global user services, further elevate the international influence of Beijing GISC, and provide dedicated assistance to countries within its area of responsibility, ensuring a smooth and successful migration to the WIS2.0 framework.

     

/

返回文章
返回