The Design and Implementation of Stream Processing for Data of Ground Automatic Weather Stations

Xiao Weiqing; Xue Lei; Liu Zhen; Luo Bing; Wang Ying; Zhang Laien; Guo Ping; Huo Qing; Han Shuli; He Wenchun

doi:10.11898/1001-7313.20240310

Vol.35, NO.3, 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Applied Meteorological Science > 2024 > 35(3): 373-384

Xiao Weiqing, Xue Lei, Liu Zhen, et al. The design and implementation of stream processing for data of ground automatic weather stations. J Appl Meteor Sci, 2024, 35(3): 373-384. DOI: 10.11898/1001-7313.20240310.

Citation:

Xiao Weiqing, Xue Lei, Liu Zhen, et al. The design and implementation of stream processing for data of ground automatic weather stations. J Appl Meteor Sci, 2024, 35(3): 373-384. DOI: 10.11898/1001-7313.20240310.

Citation:

PDF( 858 KB)

The Design and Implementation of Stream Processing for Data of Ground Automatic Weather Stations

DOI: 10.11898/1001-7313.20240310

National Meteorological Information Center, Beijing 100081

Received Date: 2023-11-29
Rev Recd Date: 2024-02-29
Publish Date: 2024-05-31

Abstract

Abstract

To process the high-density and high-frequency mass data generated by ground automatic weather stations, a real-time stream processing system based on Storm is designed and implemented in the Meteorological Big Data Cloud Platform (Tianqing). It leverages the advantages of large-scale parallel computing to enhance processing speed. For BUFR messages, a Storm topology is designed to process the standardized BUFR format data transmitted by RabbitMQ directly on the service, reducing the intermediate steps from transmission to processing of observations. In the spout design, the manual confirmation mode of RabbitMQ messages is adopted to ensure that each message is effectively processed. In the decoding process, bolt is anchored to the spout using message identification (ID) to ensure reliable processing of each message. Format and time checks are performed during data decoding to filter out abnormal data. A batch timing monitoring strategy is applied to address the issue of data ingestion loss caused by port occupancy during extensive monitoring data transmission. A startup strategy with a configurable number of spout and bolt is designed for quick optimization and adjustment based on system resources. During cluster deployment, some resources are reserved to enable automatic task migration without disrupting business operations in case of node corruption within the cluster. System design involves automatically reconnecting message queues and databases to enhance system stability and enable self-healing capabilities. Application results show that the service efficiency of 2442 national stations has decreased from 175 s with CIMISS to 78 s with Tianqing. Additionally, the service efficiency of hourly data from over 60000 regional stations has decreased from 5 min with CIMISS to 2 min with Tianqing. After switching the data source of the ART (analysis of real time) system to Tianqing, the number of stations that can be retrieved simultaneously is doubled compared to CIMISS. It can effectively improve the quality of ART live products while keeping other conditions unchanged. By implementing specialized stream processing, it can effectively handle various business scenarios where data access process of the provincial Tianqing ground automatic weather stations differ from that of other provinces. It enables the provincial Tianqing to quickly process nationwide data from ground automatic weather stations. In December 2021, Storm-based stream processing is implemented in the national and provincial meteorological information departments alongside Tianqing. It has been running smoothly over two years, delivering reliable ground automatic weather station data to users, including MICAPS4, SWAN2.0, ART systems and others.
- meteorological big data cloud platform;
- ground automatic weather station;
- Storm;
- RabbitMQ;
- stream processing;
- BUFR

FullText(HTML)

References(40)

Figures and Tables

Fig. 1 Message number of observations per minute from ground automatic weather stations from 0050 UTC to 0210 UTC on 1 Mar 2023

DownLoad: Full-Size Img PowerPoint

Fig. 1 Message number of observations per minute from ground automatic weather stations from 0050 UTC to 0210 UTC on 1 Mar 2023

DownLoad: Full-Size Img PowerPoint

Fig. 2 Relationship between processing speed of ground automatic weather station data and bolt quantity

DownLoad: Full-Size Img PowerPoint

Fig. 2 Relationship between processing speed of ground automatic weather station data and bolt quantity

DownLoad: Full-Size Img PowerPoint

Fig. 3 Relationship between regional station hourly data and minutely data simultaneous processing speed and bolt quantity

DownLoad: Full-Size Img PowerPoint

Fig. 3 Relationship between regional station hourly data and minutely data simultaneous processing speed and bolt quantity

DownLoad: Full-Size Img PowerPoint

Fig. 4 Comparison of data insert rate changes between Tianqing and CIMISS for 24 samples of national station hourly data on 29 Dec 2021

DownLoad: Full-Size Img PowerPoint

Fig. 4 Comparison of data insert rate changes between Tianqing and CIMISS for 24 samples of national station hourly data on 29 Dec 2021

DownLoad: Full-Size Img PowerPoint

Fig. 5 Comparison of data insert rate changes between Tianqing and CIMISS for 24 samples of regional station hourly data on 23 May 2021

DownLoad: Full-Size Img PowerPoint

Fig. 5 Comparison of data insert rate changes between Tianqing and CIMISS for 24 samples of regional station hourly data on 23 May 2021

DownLoad: Full-Size Img PowerPoint

Table 1 Description of data insert tables for ground automatic weather station data

数据名称	天擎入库表名
国家气象站小时数据	中国地面小时原始报告表
	中国地面小时表
	全球地面小时表
	中国地面日值表
	中国地面日照表
	重要天气表
国家气象站分钟数据	中国地面分钟原始报告表
	中国地面分钟降水表
	地面分钟全要素表
区域气象站小时数据	中国地面小时原始报告表
区域气象站小时数据	中国地面小时表
区域气象站分钟数据	中国地面分钟原始报告表
	中国地面分钟降水表
	地面分钟全要素表

DownLoad: Download CSV

Table 1 Description of data insert tables for ground automatic weather station data

数据名称	天擎入库表名
国家气象站小时数据	中国地面小时原始报告表
	中国地面小时表
	全球地面小时表
	中国地面日值表
	中国地面日照表
	重要天气表
国家气象站分钟数据	中国地面分钟原始报告表
	中国地面分钟降水表
	地面分钟全要素表
区域气象站小时数据	中国地面小时原始报告表
区域气象站小时数据	中国地面小时表
区域气象站分钟数据	中国地面分钟原始报告表
	中国地面分钟降水表
	地面分钟全要素表

DownLoad: Download CSV

Table 2 Topology description of national Tianqing ground automatic weather station processing

数据类型	每小时上传数据次数	工作进程数量	数据接入组件数量	解码入库组件数量
国家气象站小时数据	1	12	12	36
国家气象站分钟数据	60	6	6	36
区域气象站小时数据	1	24	24	240
区域气象站分钟数据	12	24	20	300

DownLoad: Download CSV

Table 2 Topology description of national Tianqing ground automatic weather station processing

数据类型	每小时上传数据次数	工作进程数量	数据接入组件数量	解码入库组件数量
国家气象站小时数据	1	12	12	36
国家气象站分钟数据	60	6	6	36
区域气象站小时数据	1	24	24	240
区域气象站分钟数据	12	24	20	300

DownLoad: Download CSV

Table 3 Summary of ground automatic weather stations hourly data from systems by various institutions of China Meteorological Administration in Mar 2023

单位	系统名称	访问次数	数据量/GB
国家气象信息中心	天擎实况	17844182	22739.5
中国气象科学研究院	东亚区域再分析及智能预报竞赛系统	753707	9400.8
国家气象中心	智能网格预报处理系统	603944	3305.1
气象探测中心	综合气象观测数据质量控制系统_天衡天衍	372840	350.6
气象干部培训学院	短临预警技能与素质综合训练系统	247207	1580.8
人工影响天气中心	人影效果评估系统	113491	1695.5
国家气候中心	气候变化影响评估与服务系统	42591	8.5
国家卫星气象中心	卫星天气应用平台(SWAP)	22425	54.4
公共气象服务中心	国家级交通气象服务业务	13647	229.5
地球系统数值预报中心	GRAPES数值预报业务系统	5517	44.8

DownLoad: Download CSV

Table 3 Summary of ground automatic weather stations hourly data from systems by various institutions of China Meteorological Administration in Mar 2023

单位	系统名称	访问次数	数据量/GB
国家气象信息中心	天擎实况	17844182	22739.5
中国气象科学研究院	东亚区域再分析及智能预报竞赛系统	753707	9400.8
国家气象中心	智能网格预报处理系统	603944	3305.1
气象探测中心	综合气象观测数据质量控制系统_天衡天衍	372840	350.6
气象干部培训学院	短临预警技能与素质综合训练系统	247207	1580.8
人工影响天气中心	人影效果评估系统	113491	1695.5
国家气候中心	气候变化影响评估与服务系统	42591	8.5
国家卫星气象中心	卫星天气应用平台(SWAP)	22425	54.4
公共气象服务中心	国家级交通气象服务业务	13647	229.5
地球系统数值预报中心	GRAPES数值预报业务系统	5517	44.8

DownLoad: Download CSV

References

[1]	Yan Y. Ground Meteorological Observation. Beijing: China Meteorological Press, 2014.
[2]	Gao S, Bi B G, Li Y A, et al. Implementation and development plan of MICAPS4. J Appl Meteor Sci, 2017, 28(5): 513-531. doi: 10.11898/1001-7313.20170501
[3]	Wang R T, Wang J M, Huang X D, et al. The architecture design of MICAPS4 server system. J Appl Meteor Sci, 2018, 29(1): 1-12. doi: 10.11898/1001-7313.20180101
[4]	Huang L P, Deng L T, Wang R C, et al. Key technologies of CMA-MESO and application to operational forecast. J Appl Meteor Sci, 2022, 33(6): 641-654. doi: 10.11898/1001-7313.20220601
[5]	Han F, Wo W F. Design and Implementation of SWAN2.0 Platform. J Appl Meteor Sci, 2018, 29(1): 25-34. doi: 10.11898/1001-7313.20180103
[6]	Liu H Z, Xu H, Bao H J, et al. Application of machine learning classification algorithm to precipitation-induced landslides forecasting. J Appl Meteor Sci, 2022, 33(3): 282-292. doi: 10.11898/1001-7313.20220303
[7]	Rostanski M, Grochla K, Seman A. Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ. IEEE, 2014: 879-884.
[8]	Zhang L E, Wang P, Han X Q. Design and practice of CTS2.0 message encapsulation and exchange control strategy. Adv Meteor Sci Tech, 2018, 8(1): 271-273. doi: 10.3969/j.issn.2095-1973.2018.01.054
[9]	Hu Y M, Wang F D, Tan X H, et al. Application of stateful message queue technology in national meteorological communication system. Comput Syst Appl, 2020, 29(3): 121-126. https://www.cnki.com.cn/Article/CJFDTOTAL-XTYY202003017.htm
[10]	Deng X, Wang Z X, Yang Y K, et al. Research on data transmission in meteorological standard format based on RabbitMQ technology. Tech Autom Appl, 2021, 40(5): 182-185. https://www.cnki.com.cn/Article/CJFDTOTAL-ZDHJ202105043.htm
[11]	Wang Y, Xue L, Zhao F, et al. Progress in standardization design and implementation of meteorological data format. Adv Meteor Sci Tech, 2018, 8(1): 252-255. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKZ201801056.htm
[12]	WMO. Manual on Codes(2019 Ed). 2021.
[13]	Wang S J, Cui P, Zheng X D, et al. Representing atmospheric motion vectors of meteorological satellites in BUFR. Meteor Sci Technol, 2011, 39(3): 339-343. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201103014.htm
[14]	Zhang E H, Yin H Y. Decoding analysis of meteorological satellite data based on compressed format BUFR code. Guangdong Meteor, 2021, 43(5): 70-74. https://www.cnki.com.cn/Article/CJFDTOTAL-GDCX202105018.htm
[15]	Xiong A Y, Zhao F, Wang Y, et al. Design and Implementation of China Integrated Meteorological Information Sharing System(CIMISS). J Appl Meteor Sci, 2015, 26(4): 500-512. doi: 10.11898/1001-7313.20150412
[16]	Ji Y H, Sun C, Liu Y M, et al. A method for optimizing storage efficiency of meteorological data in CIMISS. Meteor Sci Technol, 2017, 45(1): 29-34. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201701005.htm
[17]	Yang R Z, Shen W H, Xiao W Q, et al. A set of MapReduce tuning experiments based on meteorological operations. J Appl Meteor Sci, 2014, 25(5): 618-628. http://qikan.camscma.cn/article/id/20140511
[18]	Li Y S, Zeng Q, Xu M H, et al. Design and implementation of NWP data service platform based on Hadoop framework. J Appl Meteor Sci, 2015, 26(1): 122-128. doi: 10.11898/1001-7313.20150113
[19]	Xiao W Q, Yang R Z, Hu K X, et al. Application of Hadoop in data-intensive processing of meteorological data. Meteor Sci Technol, 2015, 43(5): 823-828. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201505009.htm
[20]	Toshniwal A, Taneja S, Shukla A, et al. Storm@twitter//Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. ACM, 2014: 147-156.
[21]	Chen M M, Wang X C, Huang F X. Storm Technology Insider and Big Data Practice. Beijing: Posts & Telecom Press, 2015.
[22]	Sun X J, Shi T, Hu Y X, et al. Real-time processing of space science satellite data based on stream computing. J Comput Appl, 2019, 39(6): 1563-1568. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJY201906003.htm
[23]	Qiao T, Zhao Z F, Ding W L. Stream computing system for monitoring copy plate vehicles. J Comput Appl, 2017, 37(1): 153-158. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJY201701028.htm
[24]	Sun C, Huo Q, Ren Z H, et al. Design and implementation of surface meteorological data statistical processing system. J Appl Meteor Sci, 2018, 29(5): 630-640. doi: 10.11898/1001-7313.20180511
[25]	Xu D, Zeng L, Wang Y J. Optimization of calculation function of "the Mirror" whole process index. Comput Technol Dev, 2023, 33(7): 20-26. https://www.cnki.com.cn/Article/CJFDTOTAL-WJFZ202307003.htm
[26]	Huo Q, He W C, He L, et al. Design and application of algorithm intensive environment for CMA big data and cloud platform. J Appl Meteor Sci, 2024, 35(1): 80-89. doi: 10.11898/1001-7313.20240107
[27]	Wang S, Xiao Y Q, Liu D W, et al. Research of main memory database. J Comput Appl, 2007, 27(10): 2353-2357. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJY200710004.htm
[28]	Yang R Z, Ma Q, Li D Q, et al. Application of memory forwarding model to data transmission system of CIMISS. J Appl Meteor Sci, 2012, 23(3): 377-384. http://qikan.camscma.cn/article/id/20120315
[29]	Dai C X. Research and application of connection pool access to database. Comput Era, 2017(11): 20-22. https://www.cnki.com.cn/Article/CJFDTOTAL-JSJS201711007.htm
[30]	Richardson L, Ruby S. RESTfulWebServices. Xu H, Li H J, Hu W, Trans. Beijing: Publishing House of Electronics Industry, 2008.
[31]	Stevens W R. TCP/IP Illustrated. Fan J H, Trans. Beijing: China Machine Press, 2000.
[32]	Lu L, Yu J, Bian C, et al. Task migration strategy of Storm, a big data streaming computing framework. J Comput Res Dev, 2018, 55(1): 71-92. https://www.cnki.com.cn/Article/CJFDTOTAL-JFYZ201801005.htm
[33]	Xing N, Zhong J Q, Lei L, et al. A probabilistic forecast experiment of short-duration heavy rainfall in Beijing based on CMA-BJ. J Appl Meteor Sci, 2023, 34(6): 641-654. doi: 10.11898/1001-7313.20230601
[34]	Li Y, Wang G F. Design and implementation of Meteorological Disaster Risk Management System. J Appl Meteor Sci, 2022, 33(5): 628-640. doi: 10.11898/1001-7313.20220510
[35]	Chang Y, Wen J W, Yang X F, et al. Rainstorm inspection in Nenjiang River Basin based on CMA-TYM and SCMOC. J Appl Meteor Sci, 2023, 34(2): 154-165. doi: 10.11898/1001-7313.20230203
[36]	Yang H P, Zhang Q, Luo B, et al. Construction and application of Meteorological Integrated Command Platform. J Appl Meteor Sci, 2023, 34(1): 117-128. doi: 10.11898/1001-7313.20230110
[37]	Shi C X, Pan Y, Gu J X, et al. A review of multi-source meteorological data fusion products. Acta Meteor Sinica, 2019, 77(4): 774-783. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB201904013.htm
[38]	Zheng Y G, Zhang X L, Zhou Q L, et al. Review on severe convective weather short-term forecasting and nowcasting. Meteor Mon, 2010, 36(7): 33-42. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX201007009.htm
[39]	Zhang M L. The National Meteorological Center Takes the Lead in Transforming the Short-term Forecasting and Nowcasting Business Process. China Meteorological News, 2023-03-29(003).
[40]	Han F, Tang W Y, Zhou C X, et al. Improving a precipitation nowcasting algorithm based on the SWAN system and related application assessment. Acta Meteor Sinica, 2023, 81(2): 304-315. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB202302008.htm