地面自动气象站数据流式处理设计与实现

The Design and Implementation of Stream Processing for Data of Ground Automatic Weather Stations

  • 摘要: 针对观测密度和频次日益增加的海量地面自动气象站数据, 在气象大数据云平台(天擎)中设计了基于Storm的实时流式处理, 利用大规模并行处理的优势提高地面自动气象站数据的处理时效。在流式处理中, 设计处理拓扑直接解码标准格式的数据消息;消息确认采用手工确认的方式, 将数据解码组件锚定数据接入组件, 实现每条数据的可靠处理;数据解码时进行字节校验和时间检查等, 过滤异常数据;应用批量加定时的发送策略, 解决海量监控信息发送气象综合业务实时监控系统(天镜)的问题;集群部署时保留部分剩余资源, 有效应对单节点异常。应用效果表明:国家气象站小时数据的服务时效由全国综合气象信息共享系统(CIMISS)的175 s提高至天擎的78 s, 约6×104个区域气象站小时数据的服务时效由CIMISS的5 min提高至天警的2 min, 实况分析系统将数据源切换至天擎后, 相同时间检索可获取的站点数量较CIMISS增加1倍。2021年12月基于Storm的流式处理与天擎一同在国省业务化运行, 实现了长期稳定运行, 为MICAPS4、SWAN2.0、实况分析系统等用户提供高效稳定的地面自动气象站数据。

     

    Abstract: To process the high-density and high-frequency mass data generated by ground automatic weather stations, a real-time stream processing system based on Storm is designed and implemented in the Meteorological Big Data Cloud Platform (Tianqing). It leverages the advantages of large-scale parallel computing to enhance processing speed. For BUFR messages, a Storm topology is designed to process the standardized BUFR format data transmitted by RabbitMQ directly on the service, reducing the intermediate steps from transmission to processing of observations. In the spout design, the manual confirmation mode of RabbitMQ messages is adopted to ensure that each message is effectively processed. In the decoding process, bolt is anchored to the spout using message identification (ID) to ensure reliable processing of each message. Format and time checks are performed during data decoding to filter out abnormal data. A batch timing monitoring strategy is applied to address the issue of data ingestion loss caused by port occupancy during extensive monitoring data transmission. A startup strategy with a configurable number of spout and bolt is designed for quick optimization and adjustment based on system resources. During cluster deployment, some resources are reserved to enable automatic task migration without disrupting business operations in case of node corruption within the cluster. System design involves automatically reconnecting message queues and databases to enhance system stability and enable self-healing capabilities. Application results show that the service efficiency of 2442 national stations has decreased from 175 s with CIMISS to 78 s with Tianqing. Additionally, the service efficiency of hourly data from over 60000 regional stations has decreased from 5 min with CIMISS to 2 min with Tianqing. After switching the data source of the ART (analysis of real time) system to Tianqing, the number of stations that can be retrieved simultaneously is doubled compared to CIMISS. It can effectively improve the quality of ART live products while keeping other conditions unchanged. By implementing specialized stream processing, it can effectively handle various business scenarios where data access process of the provincial Tianqing ground automatic weather stations differ from that of other provinces. It enables the provincial Tianqing to quickly process nationwide data from ground automatic weather stations. In December 2021, Storm-based stream processing is implemented in the national and provincial meteorological information departments alongside Tianqing. It has been running smoothly over two years, delivering reliable ground automatic weather station data to users, including MICAPS4, SWAN2.0, ART systems and others.

     

/

返回文章
返回