设为首页
加入收藏
Indentification of Weather Radar Abnormal Data Based on Deep Learning
-
摘要: 天气雷达基数据中因观测设备故障或标定问题而产生的异常数据, 直接影响天气雷达数据质量、定量估测降水及天气系统的分析和判断。目前在中国气象局气象探测中心实时业务中, 通过人工勘误环节对异常数据进行处理。针对2020—2022年业务中勘误较多的、大面积故障异常和易与降水数据混合的局部电磁干扰或故障的两类异常数据, 分别构建和训练R-ResNet和R-LinkNet两种模型, 提取雷达硬件故障、电磁干扰等特征, 实现异常数据的识别和处理。评估结果表明:两种模型在提取异常数据特征方面均具有很强的学习能力, R-ResNet在分类判识异常数据与正常数据的准确率超过99%, R-LinkNet在分离电磁干扰杂波和降水回波的准确率超过98%。两种模型可用于实时业务中监控和勘误电磁干扰、故障等异常数据, 实现异常数据的自动勘误处理。Abstract: The world's largest weather radar observation network which consists of 236 weather radars is built up in China. The quality control of weather radar data becomes an indispensable part in operation as data grow. In real-time operation of CMA Meterological Observation Center, the abnormal data caused by radar hardware fault or calibration problem usually leads to a bad image, and the problem directly affects the quality of weather radar data, quantitative estimation of precipitation and analysis of the weather system. At present, the abnormal data are processed by artificial corrigendum in real-time operation, so that it does not affect the application of follow-up data. In recent years, artificial intelligence technology has developed rapidly, and deep learning algorithms are used to build convolutional neural network models to extract image features from abnormal data such as radar hardware failures and electromagnetic interference. According to characteristics of the abnormal data, two kinds of abnormal dataset are established. The first kind is labeled data for those samples with whole abnormal picture or precipitation. The second kind is the electromagnetic interference in fixed direction, which is easily mixed with precipitation, and it is used to label the samples for pixels. Based on the convolution pre-training network model (ResNet), the R-ResNet model is built with the first kind of abnormal data. The model evaluation shows that the R-ResNet model achieves more than 99% accuracy in both training and test datasets, and the prediction results of the model in test datasets are all consistent with the label data. For the second kind of abnormal data, based on the image semantic segmentation network model LinkNet, a group of encoders and decoders are extended, and a hop-link structure is added, and ResNet50 is used as the encoder structure. The R-LinkNet model is constructed and the accuracy is over 98% on the training and test datasets, with intersection over union of 83.4% and 83.2%, respectively. Two models can be used to monitor the abnormal data of the national radar in real-time operation, so that abnormal data can be corrected automatically and the workload of the manual on duty can be reduced greatly.
-
Key words:
- deep learning;
- neural network;
- abnormal data;
- model
-
图 1 2020年全国布网运行天气雷达数据质量问题占比
Fig. 1 Proportion of weather radar data quality problems in China in 2020
图 2 全国各型号天气雷达电磁干扰占比及个例
(a)各型号天气雷达电磁干扰占比，(b)2022年7月16日01:25:58山东滨州雷达电磁干扰(相邻距离圈间隔为50 km，下同)
Fig. 2 Proportion of electromagnetic interference of various types of weather radar
(a)proportion of electromagnetic interference, (b)a case of electromagnetic interference of Binzhou radar in Shandong at 012558 BT 16 Jul 2022 (the distance between adjacent rings is 50 km, similarly hereinafter)
图 3 全国各型号天气雷达设备故障、标定异常数据占比及个例
(a)各型号天气雷达设备故障、标定异常数据占比，(b)2022年4月11日11:21:10新疆图木舒克雷达异常数据
Fig. 3 Percentage of abnormal data of equipment failure and calibration of various types of weather radar in China
(a)proportion of abnormal data of equipment failure and calibration, (b)a case of abnormal data of Tumxuk radar in Xinjiang at 112110 BT 11 Apr 2022
图 4 第2类异常数据原始图像与标签数据
(a)2022年7月20日08:15:02黑龙江黑瞎子岛雷达观测图像，(b)标签数据
Fig. 4 Observation image and label data of the second kind of abnormal data
(a)observation image of Heixiazi Island radar in Heilongjiang at 081502 BT 20 Jul 2022, (b)label data
图 5 ResNet50在第1类异常数据的准确率和损失率
Fig. 5 Accuracy and loss of ResNet50 model on the first kind of abnormal data
图 6 R-LinkNet在第2类异常数据的准确率和损失率
Fig. 6 Accuracy and loss of R-LinkNet model on the second kind of abnormal data
图 7 R-LinkNet分离黑龙江黑瞎子岛雷达观测电磁干扰和降水回波
(a)2022年7月8日20:03:34观测图像，(b)2022年7月8日20:03:34质量控制后图像，(c)2022年8月5日10:03:35观测图像，(d)2022年8月5日10:03:35质量控制后图像，(e)2022年9月8日09:02:33观测图像，(f)2022年9月8日09:02:33质量控制后图像
Fig. 7 R-LinkNet model discriminate electromagnetic interference and precipitation of Heixiazi Island radar in Heilongjiang
(a)observation image at 200334 BT 8 Jul 2022, (b)image after quality control by R-LinkNet model at 200334 BT 8 Jul 2022, (c)observation image at 100335 BT 5 Aug 2022, (d)image after quality control by R-LinkNet model at 100335 BT 5 Aug 2022, (e)observation image at 090233 BT 8 Sep 2022, (f)image after quality control by R-LinkNet model at 090233 BT 8 Sep 2022
表 1 ResNet基础模型准确率(单位：%)
Table 1 Accuracy of ResNet models (unit:%)
指标 数据 ResNet18 ResNet50 ResNet101 Acc 训练集 99.92 99.98 99.99 测试集 100 100 100 Los 训练集 0.1 0.1 0.1 测试集 0.1 0.1 0.1 
下载: 导出CSV
表 2 模型准确率、损失率和交并比评估(单位：%)
Table 2 Accuracy, loss and intersection over union of model (unit:%)
数据 Acc Los Iou LinkNet R-LinkNet LinkNet R-LinkNet LinkNet R-LinkNet 训练集 97.8 98.6 0.7 0.4 73.8 83.4 测试集 97.8 98.6 0.7 0.4 73.5 83.2
下载: 导出CSV
表 3 R-LinkNet模型准确率检验评估(单位：%)
Table 3 Evaluation of R-LinkNet model accuracy (unit:%)
测试时间 R-LinkNet模型 7月 8月 9月 7月 98.6 98.5 98.3 8月 98.2 98.6 98.5 9月 98.0 98.3 98.6
下载: 导出CSV
-
[1] 赵瑞金,刘黎平,张进.硬件故障导致雷达回波错误数据质量控制方法.应用气象学报, 2015, 26(5):578-589. doi: 10.11898/1001-7313.20150507Zhao R J, Liu L P, Zhang J. The quality control method of erroneous radar echo data generated by hardware fault. J Appl Meteor Sci, 2015, 26(5): 578-589. doi: 10.11898/1001-7313.20150507 [2] 邵楠, 裴翀, 刘传才, 等. 基于图像处理技术自动判别雷达异常产品. 气象科技, 2013, 41(6): 993-997. doi: 10.3969/j.issn.1671-6345.2013.06.004Shao N, Pei C, Liu C C, et al. Automatic idenfication system of abnormal radar echoes based on image processing technology. Meteor Sci Technol, 2013, 41(6): 993-997. doi: 10.3969/j.issn.1671-6345.2013.06.004 [3] 江源, 刘黎平, 庄薇. 多普勒天气雷达地物回波特征及其识别方法改进. 应用气象学报, 2009, 20(2): 203-213. http://qikan.camscma.cn/article/id/20090210Jiang Y, Liu L P, Zhuang W. Statistical characteristics of clutter and improvements of ground clutter identification technique with Doppler weather radar. J Appl Meteor Sci, 2009, 20(2): 203-213. http://qikan.camscma.cn/article/id/20090210 [4] 李丰, 刘黎平, 王红艳, 等. C波段多普勒天气雷达地物识别方法. 应用气象学报, 2014, 25(2): 158-167. http://qikan.camscma.cn/article/id/20140205Li F, Liu L P, Wang H Y, et al. Identification of ground clutter with C-band Doppler weather radar. J Appl Meteor Sci, 2014, 25(2): 158-167. http://qikan.camscma.cn/article/id/20140205 [5] 刘黎平, 吴林林, 杨引明. 基于模糊逻辑的分步式超折射地物回波识别方法的建立和效果分析. 气象学报, 2007, 65(2): 252-260. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB200702010.htmLiu L P, Wu L L, Yang Y M. Development of fuzzy-logical two-step ground clutter detection algorithm. Acta Meteor Sinica, 2007, 65(2): 252-260. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB200702010.htm [6] 文浩, 刘黎平, 张扬. 多普勒天气雷达地物回波识别算法改进. 高原气象, 2017, 36(3): 736-749. https://www.cnki.com.cn/Article/CJFDTOTAL-GYQX201703014.htmWen H, Liu L P, Zhang Y. Improvements of ground clutter identification algorithm for Doppler weather radar. Plateau Meteor, 2017, 36(3): 736-749. https://www.cnki.com.cn/Article/CJFDTOTAL-GYQX201703014.htm [7] 文浩, 张乐坚, 梁海河, 等. 基于模糊逻辑的新一代天气雷达径向干扰回波识别算法. 气象学报, 2020, 78(1): 116-127. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB202001010.htmWen H, Zhang L J, Liang H H, et al. Radial interference echo identification algorithm based on fuzzy logic for weather radar. Acta Meteor Sinica, 2020, 78(1): 116-127. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB202001010.htm [8] 谭学, 刘黎平, 范思睿. 新一代天气雷达海浪回波特征分析和识别方法研究. 气象学报, 2013, 71(5): 962-975. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB201305015.htmTan X, Liu L P, Fan S R. Statistical characteristics of sea clutter and its identification with the CINRAD. Acta Meteor Sinica, 2013, 71(5): 962-975. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB201305015.htm [9] 冷亮, 黄兴友, 杨洪平, 等. 多普勒雷达晴空回波识别与应用. 气象科技, 2012, 40(4): 534-541. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201204005.htmLeng L, Huang X Y, Yang H P, et al. Recognition and application of Doppler weather radar clear air echoes. Meteor Sci Technol, 2012, 40(4): 534-541. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201204005.htm [10] 陈亚军, 梁海河, 张乐坚, 等. 新一代天气雷达晴空回波反射率因子特征分析. 气象科技, 2022, 50(3): 303-313. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ202203001.htmChen Y J, Liang H H, Zhang L J. Characteristic analysis reflectivity factor of clear air echoes of Doppler weather radars. Meteor Sci Technol, 2022, 50(3): 303-313. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ202203001.htm [11] 叶飞, 梁海河, 文浩, 等. 新一代天气雷达均一性评估. 气象科技, 2020, 48(3): 322-330. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ202003002.htmYe F, Liang H H, Wen H, et al. Evaluation of homogeneity of new generation weather radar. Meteor Sci Technol, 2020, 50(3): 322-330. https://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ202003002.htm [12] 李丰, 刘黎平, 王红艳, 等. S波段多普勒天气雷达非降水气象回波识别. 应用气象学报, 2012, 23(2): 147-158. http://qikan.camscma.cn/article/id/20120203Li F, Liu L P, Wang H Y, et al. Identification of non-precipitation meteorological echoes with Doppler weather radar. J Appl Meteor Sci, 2012, 23(2): 147-158. http://qikan.camscma.cn/article/id/20120203 [13] 肖艳姣, 万玉发, 王珏, 等. 一种自动多普勒雷达速度退模糊算法研究. 高原气象, 2012, 31(4): 1119-1128. https://www.cnki.com.cn/Article/CJFDTOTAL-GYQX201204028.htmXiao Y J, Wan Y F, Wang J, et al. Study of an automated Doppler radar velocity dealiasing algorithm. Plateau Meteor, 2012, 31(4): 1119-1128. https://www.cnki.com.cn/Article/CJFDTOTAL-GYQX201204028.htm [14] 杨川, 刘黎平, 胡志群, 等. C波段多普勒雷达双PRF模式速度混淆区识别和处理方法研究. 气象学报, 2013, 70(4): 875-886. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB201204026.htmYang C, Liu L P, Hu Z Q, et al. An algorithm for chaos radial velocity identifying and processing in C-band Doppler radars running in the dual PRF mode. Acta Meteor Sinica, 2013, 70(4): 875-886. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB201204026.htm [15] 吴翀, 刘黎平, 仰美霖, 等. X波段双偏振雷达相态识别与拼图的关键技术. 应用气象学报, 2021, 32(2): 200-216. doi: 10.11898/1001-7313.20210206Wu C, Liu L P, Yang M L, et al. Key technologies of hydrometeor classification and mosaic algorithm for X-band polarimetric radar. J Appl Meteor Sci, 2021, 32(2): 200-216. doi: 10.11898/1001-7313.20210206 [16] Tang L, Zhang L, Langston C, et al. A physically based precipitation-nonprecipitation radar classifier using polarimetric and environmental data in a real-time national system. Wea Forecasting, 2014, 29(5): 1106-1119. [17] Park H S, Ryzhkov A V, Zrnić D S, et al. The hydrometeor classification algorithm for the polarimetric WSR-88D: Description and application to an MCS. Wea Forecasting, 2009, 24(3): 730-748. [18] Jiang Y, Xu Q, Zhang P F, et al. Using WSR-88D polarimetric data to identify bird-contaminated Doppler velocities. Adv Meteor, 2013, 1: 1-13. [19] 朱轶明, 马舒庆, 杨玲, 等. 上海南汇WSR-88D双偏振天气雷达的生物回波识别与分析. 气象与环境科学, 2019, 42(3): 118-128. https://www.cnki.com.cn/Article/CJFDTOTAL-HNQX201903016.htmZhu Y M, Ma S Q, Yang L, et al. Recognition and analysis of biological echo using WSR-88D dual-polarization weather radar in Nanhui of Shanghai. Meteor Environ Sci, 2019, 42(3): 118-128. https://www.cnki.com.cn/Article/CJFDTOTAL-HNQX201903016.htm [20] 张林, 杨洪平. S波段WSR-88D双偏振雷达观测非降水回波识别方法研究. 气象, 2018, 44(5): 665-675. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX201805007.htmZhang L, Yang H P. Non-precipitation identification technique on S-band WSR-88D polarization weather radar. Meteor Month, 2018, 44(5): 665-675. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX201805007.htm [21] 张林, 李峰, 吴蕾, 等. CINRAD/SAD双偏振雷达非降水回波识别技术. 应用气象学报, 2022, 33(6): 724-735. doi: 10.11898/1001-7313.20220607Zhang L, Li F, Wu L, et al. Non-precipitation identification technique for CINRAD/SAD dual polarimetric weather radar. J Appl Meteor Sci, 2022, 33(6): 724-735. doi: 10.11898/1001-7313.20220607 [22] 张林, 李峰, 冯婉悦, 等. 移动X波段双线偏振雷达数据质量分析及偏差订正. 气象, 2021, 47(3): 337-347. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103007.htmZhang L, Li F, Feng W Y, et al. Research of data quality analysis and bias correction on mobile X-band dual-polarization weather radar. Meteor Mon, 2021, 47(3): 337-347. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103007.htm [23] 尹晓燕, 胡志群, 郑佳锋, 等. 利用深度学习填补双偏振雷达回波遮挡. 应用气象学报, 2022, 33(5): 581-593. doi: 10.11898/1001-7313.20220506Yin X Y, Hu Z Q, Zheng J F, et al. Filling in the dual polarization radar echo occlusion based on deep learning. J Appl Meteor Sci, 2022, 33(5): 581-593. doi: 10.11898/1001-7313.20220506 [24] 李颖, 陈怀亮. 机器学习技术在现在农业气象中的应用. 应用气象学报, 2020, 31(3): 257-266. doi: 10.11898/1001-7313.20200301Li Y, Chen H L. Review of machine learning approaches for modern agrometeorology. J Appl Meteor Sci, 2020, 31(3): 257-266. doi: 10.11898/1001-7313.20200301 [25] 张烨方, 冯真祯, 刘冰. 基于卷积神经网络的雷电临近预警模型. 气象, 2021, 47(3): 373-380. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103010.htmZhang Y F, Feng Z Z, Liu B. Lightning nowcasting early warning model based on convolutional neural network. Meteor Mon, 2021, 47(3): 373-380. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103010.htm [26] 周康辉, 郑永光, 韩雷, 等. 机器学习在强对流监测预报中的应用进展. 气象, 2021, 47(3): 274-289. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103002.htmZhou K H, Zheng Y G, Han L, et al. Advances in application of machine learning to severe convective weather monitoring and forecasting. Meteor Mon, 2021, 47(3): 274-289. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103002.htm [27] 黄骄文, 蔡荣辉, 姚蓉, 等. 深度学习网络在降水相态判识和预报中的应用. 气象, 2021, 47(3): 317-326. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103005.htmHuang J W, Cai R H, Yao R, et al. Application of deep learning method to discrimination and forecasting of precipitation type. Meteor Mon, 2021, 47(3): 317-326. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXX202103005.htm [28] 金子琪, 王新敏, 鲍艳松, 等. 基于卷积神经网络的飑线识别算法. 应用气象学报, 2021, 32(5): 580-591. doi: 10.11898/1001-7313.20210506Jin Z Q, Wang X M, Bao Y S, et al. Squall line identification method based on convolution neural network. J Appl Meteor Sci, 2021, 32(5): 580-591. doi: 10.11898/1001-7313.20210506 [29] Nawal H, Timothy D, Sebastian T, et al. A neural network quality-control scheme for improved quantitative precipitation estimation accuracy on the UK weather radar network. J Atmos Ocean Technol, 2021, 38(6): 1157-1172. [30] 皇甫江, 胡志群, 郑佳锋, 等. 利用深度学习开展偏振雷达定量降水估测研究. 气象学报, 2022, 80(4): 565-577. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB202204006.htmHuangfu J, Hu Z Q, Zheng J F, et al. A study on polarization radar quantitative precipitation estimation using deep learning. Acta Meteor Sinica, 2022, 80(4): 565-577. https://www.cnki.com.cn/Article/CJFDTOTAL-QXXB202204006.htm [31] 赵琳娜, 卢姝, 齐丹, 等. 基于全连接神经网络方法的日最高气温预报. 应用气象学报, 2022, 33(3): 257-269. doi: 10.11898/1001-7313.20220301Zhao L N, Lu S, Qi D, et al. Daily maximum air temperature forecast based on fully connected neural network. J Appl Meteor Sci, 2022, 33(3): 257-269. doi: 10.11898/1001-7313.20220301 [32] 刘海知, 徐辉, 包红军, 等. 机器学习分类算法在降雨型滑坡预报中的应用. 应用气象学报, 2022, 33(3): 282-292. doi: 10.11898/1001-7313.20220303Liu H Z, Xu H, Bao H J, et al. Application of machine learning classification algorithm to precipitation-induced landslides forecasting. J Appl Meteor Sci, 2022, 33(3): 282-292. doi: 10.11898/1001-7313.20220303 [33] 张方言, 赵梦, 周弈志, 等. 基于ResNet50和迁移学习的红鳍东方鲀病鱼检测方法. 渔业现代化, 2021, 48(4): 51-60. https://www.cnki.com.cn/Article/CJFDTOTAL-HDXY202104007.htmZhang F Y, Zhao M, Zhou Y Z, et al. Detection of diseased takifugu rubripes based on ResNet50 and transfer learning. Fishery Modern, 2021, 48(4): 51-60. https://www.cnki.com.cn/Article/CJFDTOTAL-HDXY202104007.htm [34] 日月光华. PyTorch深度学习简明实战. 北京: 清华大学出版社, 2022.Riyue G H. PyTorch Deep Learning Concise Actual Combat. Beijing: Tsinghua University Press, 2022. [35] 郑泽宇, 梁博文, 顾思宇. TensorFlow: 实战Google深度学习框架(第2版). 北京: 电子工业出版社, 2018.Zheng Z Y, Liang B W, Gu S Y. TensorFlow: A Framework for Google Deep Learning in Actual Combat. 2nd Ed. Beijing: Publishing House of Electronics Industry, 2018. [36] 陈勇, 党淑雯, 聂铃. 基于ResNet模型的回环检测算法. 智能计算机与应用, 2022, 12(8): 196-199. https://www.cnki.com.cn/Article/CJFDTOTAL-DLXZ202208039.htmChen Y, Dang S W, Nie L, Loop detection algorithm based on resnet model. Intelligent Computer Appl, 2022, 12(8): 196-199. https://www.cnki.com.cn/Article/CJFDTOTAL-DLXZ202208039.htm [37] Abhishek C, Culurciello E. LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation. arXiv Preprint arXiv, 2017. DOI: 10.1109/VCIP.2017.8305148. [38] 刘海涛. 基于改进LinkNet的遥感道路提取方法研究. 哈尔滨: 哈尔滨工程大学, 2021.Liu H T. Research on Remote Sensing Road Extraction Method Based on Improved Linknet. Harbin: Harbin Engineering University, 2021. [39] 张颖. 基于深度学习的图像语义分割算法研究. 成都: 西南科技大学, 2021.Zhang Y. Research on Image Semantic Segmentation Algorithm Based on Deep Learning. Chengdu: Southwest University of Science and Technology, 2021. [40] 杨旭勃. 基于语义分割的卫星影像中道路和小建筑物提取方法研究. 武汉: 华中科技大学, 2019.Yang X B. Road and Small Buliding Extraction Methods Based on Semantic Segmentation in Remote Sensing Image. Wuhan: Huazhong University of Science and Technology, 2019. -
图(7) / 表(3)
计量
- 摘要浏览量: 601
- HTML全文浏览量: 103
- PDF下载量: 149
- 被引次数: 0
