设为首页
加入收藏
Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform
-
摘要: 气象业务系统集约化发展和“云+端”业务技术体制改革是实现气象业务高质量发展的重要措施。2020年中国气象局提出构建以气象大数据云平台为云、气象业务系统为端的“云+端”业务技术体制, 明确气象大数据云平台作为关键基础技术平台的定位。加工流水线作为气象算法的集约化环境, 应用数算一体、高效任务调度、可视化流程编排、容器等技术, 实现气象算法的统一管理与高效集约调度运行。2021年加工流水线业务运行, 支撑全国202个业务系统的实时运行, 业务系统性能提升1~10倍, 集约化程度显著提高, 对提升业务系统的运行效率、增强业务系统的协同性、加速“云+端”业务技术体制改革进程和推进气象业务集约发展发挥了重要支撑作用。Abstract: With the advancement of meteorological services, various product processing systems and supporting data management systems have been developed for different business systems. However, this has led to the problem of system non-intensification. The lack of intensification in meteorological services can result in inconsistent data standards and make the operation and maintenance more challenging, which can lead to significant waste in investment due to duplicate data storage and inconsistent data caused by untimely synchronization. Moreover, the lack of information and technology hinders the integration of upstream and downstream businesses. The intensive development of meteorological business systems and the reform of "cloud+end" technology system are important measures to achieve high-quality development of meteorological business. China Meteorological Administration proposed to build a "cloud+end" technology system in 2020. CMA Big Data and Cloud Platform serves as the cloud, while the meteorological business system is the end, clarifying the positioning of CMA Big Data and Cloud Platform as a key foundational technology platform. The data processing line (DPL) is an intensive environment for meteorological algorithms. It addresses business needs such as efficient and stable processing of data products, data sharing and collaboration among business systems, and efficient and intensive business applications. To achieve this goal, algorithm libraries and task control functions are established by utilizing technologies such as integrated digital and computing, efficient task scheduling, visual process arrangement, and containers. The algorithm library facilitates the standardization, unified management and sharing of algorithms. Task control supports multiple scheduling strategies with high reliability and fault tolerance, enabling efficient and stable scheduling operations for algorithms. All functions mentioned above are in the form of interfaces for the application frontend. At the same time, based on the meteorological business comprehensive monitoring system (referred to as Tianjing) enables automatic collection of algorithm operation status and detection of abnormal alarms. Since its operation in 2021, the processing assembly line has facilitated the real-time operation of 202 business systems nationwide, resulting in a performance improvement of 1-10 times and a significant increase in efficiency. It plays an important supporting role in improving the operational efficiency of business systems, enhancing their collaboration, accelerating the process of "cloud+end" business technology system reform, and promoting the intensive development of meteorological business.
-
图 5 基于加工流水线功能开放接口构建“云+端”业务架构
Fig. 5 "cloud+end" business architecture based on open functional interface of data processing line
图 7 加工流水线天气雷达产品加工任务流程示意图
Fig. 7 Task flow of weather radar products on data processing line
图 8 截至2023年11月底国省融入加工流水线的算法数量(省级取前10名)
Fig. 8 Number of meteorological algorithms integrated into data processing line from national-level and provincial-level by the end of Nov 2023(only the top 10 provinces are shown)
-
[1] 郑国光.以信息化推进气象现代化.浙江气象, 2015, 36(2): 1-4.Zheng G G. Promoting meteorological modernization by informatization. J Zhejiang Meteor, 2015, 36(2): 1-4. [2] 沈文海, 赵芳, 高华云, 等. 国家级气象资料存储检索系统的建立. 应用气象学报, 2004, 15(6): 727-736. http://qikan.camscma.cn/article/id/20040690Shen W H, Zhao F, Gao H Y, et al. The construction of national meteorological archival and retrieval system. J Appl Meteor Sci, 2004, 15(6): 727-736. http://qikan.camscma.cn/article/id/20040690 [3] 李集明, 沈文海, 王国复. 气象信息共享平台及其关键技术研究. 应用气象学报, 2006, 17(5): 621-628. http://qikan.camscma.cn/article/id/200605105Li J M, Shen W H, Wang G F. Meteorological information sharing service platform and its key technologies. J Appl Meteor Sci, 2006, 17(5): 621-628. http://qikan.camscma.cn/article/id/200605105 [4] 熊安元, 赵芳, 王颖, 等. 全国综合气象信息共享系统的设计与实现. 应用气象学报, 2015, 26(4): 500-512. doi: 10.11898/1001-7313.20150412Xiong A Y, Zhao F, Wang Y, et al. Design and implementation of China Integrated Meteorological Information Sharing System(CIMISS). J Appl Meteor Sci, 2015, 26(4): 500-512. doi: 10.11898/1001-7313.20150412 [5] 赵芳, 熊安元, 张小缨, 等. 全国综合气象信息共享平台架构设计技术特征. 应用气象学报, 2017, 28(6): 750-758. doi: 10.11898/1001-7313.20170610Zhao F, Xiong A Y, Zhang X Y, et al. Technical characteristics of the architecture design of China Integrated Meteorological Information Sharing System. J Appl Meteor Sci, 2017, 28(6): 750-758. doi: 10.11898/1001-7313.20170610 [6] 孙超, 霍庆, 任芝花, 等. 地面气象资料统计处理系统设计与实现. 应用气象学报, 2018, 29(5): 630-640. doi: 10.11898/1001-7313.20180511Sun C, Huo Q, Ren Z H, et al. Design and implementation of surface meteorological data statistical processing system. J Appl Meteor Sci, 2018, 29(5): 630-640. doi: 10.11898/1001-7313.20180511 [7] 吴焕萍, 张永强, 孙家民, 等. 气候信息交互显示与分析平台(CIPAS)设计与实现. 应用气象学报, 2013, 24(5): 631-640. http://qikan.camscma.cn/article/id/20130513Wu H P, Zhang Y Q, Sun J M, et al. Designing and implementation of Climate Interactive Plotting and Analysis System. J Appl Meteor Sci, 2013, 24(5): 631-640. http://qikan.camscma.cn/article/id/20130513 [8] 王若曈, 王建民, 黄向东, 等. MICAPS4服务端系统架构设计. 应用气象学报, 2018, 29(1): 1-12. doi: 10.11898/1001-7313.20180101Wang R T, Wang J M, Huang X D, et al. The architecture design of MICAPS4 server system. J Appl Meteor Sci, 2018, 29(1): 1-12. doi: 10.11898/1001-7313.20180101 [9] 韩丰, 沃伟峰. SWAN2.0系统的设计与实现. 应用气象学报, 2018, 29(1): 25-34. doi: 10.11898/1001-7313.20180103Han F, Wo W F. Design and implementation of SWAN2.0 platform. J Appl Meteor Sci, 2018, 29(1): 25-34. doi: 10.11898/1001-7313.20180103 [10] 吴门新, 庄立伟, 侯英雨, 等. 中国农业气象业务系统(CAgMSS)设计与实现. 应用气象学报, 2019, 30(5): 513-527. doi: 10.11898/1001-7313.20190501Wu M X, Zhuang L W, Hou Y Y, et al. The design and implementation of China Agricultural Meteorological Service System(CAgMSS). J Appl Meteor Sci, 2019, 30(5): 513-527. doi: 10.11898/1001-7313.20190501 [11] 李德泉, 李抗抗, 李宏宇, 等. 飞机作业监测移动应用系统的设计与实现. 应用气象学报, 2019, 30(6): 745-758. doi: 10.11898/1001-7313.20190610Li D Q, Li K K, Li H Y, et al. Design and implementation of mobile application for real-time monitoring of weather-modification aircraft operations. J Appl Meteor Sci, 2019, 30(6): 745-758. doi: 10.11898/1001-7313.20190610 [12] 马强, 颜京辉, 魏敏, 等. 北京气候中心CMIP6数据共享平台及应用. 应用气象学报, 2022, 33(5): 617-627. doi: 10.11898/1001-7313.20220509Ma Q, Yan J H, Wei M, et al. Implementation and application of BCC CMIP6 Experimental Data Sharing Platform. J Appl Meteor Sci, 2022, 33(5): 617-627. doi: 10.11898/1001-7313.20220509 [13] 李莹, 王国复. 气象灾害风险管理系统设计与应用. 应用气象学报, 2022, 33(5): 628-640. doi: 10.11898/1001-7313.20220510Li Y, Wang G F. Design and implementation of Meteorological Disaster Risk Management System. J Appl Meteor Sci, 2022, 33(5): 628-640. doi: 10.11898/1001-7313.20220510 [14] 张进, 孙健, 沈学顺, 等. CMA-GFS V4.0模式关键技术研发和业务化. 应用气象学报, 2023, 34(5): 513-526. doi: 10.11898/1001-7313.20230501Zhang J, Sun J, Shen X S, et al. Key model technologies of CMA-GFS V4.0 and application to operational forecast. J Appl Meteor Sci, 2023, 34(5): 513-526. doi: 10.11898/1001-7313.20230501 [15] 杨和平, 张强, 罗兵, 等. 气象综合指挥平台建设与应用. 应用气象学报, 2023, 34(1): 117-128. doi: 10.11898/1001-7313.20230110Yang H P, Zhang Q, Luo B, et al. Construction and application of Meteorological Integrated Command Platform. J Appl Meteor Sci, 2023, 34(1): 117-128. doi: 10.11898/1001-7313.20230110 [16] 梁丽, 马舒庆, 滕玉鹏, 等. 天气雷达空中生态监测系统建设和应用. 应用气象学报, 2023, 34(5): 630-640. doi: 10.11898/1001-7313.20230511Liang L, Ma S Q, Teng Y P, et al. Construction and application of Weather Radar Aerial Ecological Monitoring System. J Appl Meteor Sci, 2023, 34(5): 630-640. doi: 10.11898/1001-7313.20230511 [17] 沈文海. 气象信息化进程中云计算的意义. 中国信息化, 2015(3): 80-88.Shen W H. Significance of cloud computing in the process of meteorological informatization. Chinese Informatization, 2015(3): 80-88. [18] 沈文海. 气象业务信息系统未来基础架构探讨——"云计算"和"大数据"在气象信息化中的作用. 气象科技进展, 2015, 5(3): 64-66.Shen W H. Discussion on the future infrastructure of meteorological business information system-The role of "cloud computing" and "big data" in meteorological informatization. Adv Meteor Sci Tech, 2015, 5(3): 64-66. [19] 章国材. 气象云建设的研究与思考. 气象与环境科学, 2015, 38(4): 1-11. doi: 10.3969/j.issn.1673-7148.2015.04.001Zhang G C. Study and thinking on the construction of meteorological cloud. Meteor Environ Sci, 2015, 38(4): 1-11. doi: 10.3969/j.issn.1673-7148.2015.04.001 [20] 聂峰英. 大数据资源技术服务协同研究——以气象数据为例. 信息化研究, 2016, 42(1): 6-11.Nie F Y. The research of the big data resources technical services collaboration-A case study of meteorological data. Informatization Res, 2016, 42(1): 6-11. [21] 沈文海. 对气象信息化的理解和再认识. 气象科技进展, 2013, 3(5): 56-62.Shen W H. Understanding and re-understanding of meteorological informatization. Adv Meteor Sci Tech, 2013, 3(5): 56-62. [22] 周勇, 刘东君, 马锋波. 气象信息化标准体系框架研究. 中国信息化, 2016(4): 74-80.Zhou Y, Liu D J, Ma F B. Research on the framework of meteorological informatization standard system. Chinese Informatization, 2016(4): 74-80. [23] 中国气象局. 气象大数据行动计划(2017—2020年). 2017.China Meteorological Administration. Plan of the Meteorological Big Data Action(2017-2020). 2017. [24] 中国气象局. 气象信息化发展规划(2018—2022年). 2017.China Meteorological Administration. Plan of the Meteorological Informatization Development(2018-2022). 2017. [25] 中国气象局. 中国气象局关于推进气象业务技术体制重点改革的意见. 2020.China Meteorological Administration. Opinions of the China Meteorological Administration on Promoting the Key Reform of Meteorological Technology System. 2020. [26] 中国气象局. 新型气象业务技术体制改革方案(2022—2025年). 2022.China Meteorological Administration. Plan for the Reform of Meteorological Technology System(2022-2025). 2022. [27] 中国气象局. 气象大数据云平台设计方案. 2018.China Meteorological Administration. Design of the CMA Big Data and Cloud Platform. 2018. [28] Papazoglou M P, van den Heuvel W J. Service oriented architectures: Approaches, technologies and research issues. VLDB J, 2007, 16(3): 389-415. doi: 10.1007/s00778-007-0044-3 [29] Kivity A, KarrIay Y, Laor D, et al. KVM: The Linux Virtual Machine Monitor//Proceedings of Linux Symposium, 2007, I: 225-230. [30] Kozhirbayev Z, Sinnott R O. A performance comparison of container-based technologies for the cloud. Future Gener Comput Syst, 2017, 68: 175-182. doi: 10.1016/j.future.2016.08.025 [31] 杨鹏, 马志程, 彭博, 等. 集成Docker容器的OpenStack云平台性能研究. 计算机工程, 2017, 43(8): 26-31.Yang P, Ma Z C, Peng B, et al. Performace research of OpenStack cloud platform integrated with Docker container. Comput Eng, 2017, 43(8): 26-31. [32] 伍阳. 基于Docker的虚拟化技术研究. 信息技术, 2016, 40(1): 121-123;128.Wu Y. Research on virtualization technology based on Docker. Inf Technol, 2016, 40(1): 121-123;128. [33] Salza P, Ferrucci F. Speed up genetic algorithms in the cloud using software containers. Future Gener Comput Syst, 2019, 92: 276-289. doi: 10.1016/j.future.2018.09.066 [34] Wan X L, Guan X J, Wang T J, et al. Application deployment using Microservice and Docker containers: Framework and optimization. J Netw Comput Appl, 2018, 119: 97-109. doi: 10.1016/j.jnca.2018.07.003 [35] De Benedictis M, Lioy A. Integrity verification of Docker containers for a lightweight cloud environment. Future Gener Comput Syst, 2019, 97: 236-246. doi: 10.1016/j.future.2019.02.026 [36] Felter W, Ferreira A, Rajamony R, et al. An Updated Performance Comparison of Virtual Machines and Linux Containers//2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2015: 171-172. [37] Joy A M. Performance Comparison between Linux Containers and Virtual Machines//Proceedings of 2015 International Conference on Advances in Computer Engineering and Applications. IEEE, 2015: 342-346. [38] Merkel D. Docker: Lightweight Linux containers for consistent development and deployment. Linux J, 2014, 2014(239): No2. [39] Jaramillo D, Nguyen D V, Smart R. Leveraging Microservices Architecture by Using Docker Technology//Southeast Con. IEEE, 2016: 1-5. [40] JGraph Ltd. User Manual(version 4.2.2-28). 2020. [41] 徐拥军, 倪学磊, 郑波, 等. 基于位置的天气实况数据服务接口设计与应用. 计算机系统应用, 2023, 32(5): 77-86.Xu Y J, Ni X L, Zheng B, et al. Design and application of live weather data service interface based on user's location. Comput Syst Appl, 2023, 32(5): 77-86. -
下载:
图(9)
计量
- 摘要浏览量: 539
- HTML全文浏览量: 141
- PDF下载量: 119
- 被引次数: 0
