Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform

Huo Qing; He Wenchun; He Lin; Gao Feng; Chen Shiwang; Xu Yongjun

doi:10.11898/1001-7313.20240107

Vol.35, NO.1, 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Applied Meteorological Science > 2024 > 35(1): 80-89

Huo Qing, He Wenchun, He Lin, et al. Design and application of algorithm intensive environment for CMA big data and cloud platform. J Appl Meteor Sci, 2024, 35(1): 80-89. DOI: 10.11898/1001-7313.20240107.

Citation:

Huo Qing, He Wenchun, He Lin, et al. Design and application of algorithm intensive environment for CMA big data and cloud platform. J Appl Meteor Sci, 2024, 35(1): 80-89. DOI: 10.11898/1001-7313.20240107.

Citation:

PDF( 1910 KB)

Design and Application of Algorithm Intensive Environment for CMA Big Data and Cloud Platform

DOI: 10.11898/1001-7313.20240107

1.
National Meteorological Information Center, Beijing 100081
2.
Shaanxi Meteorological Information Center, Xi'an 710014

Received Date: 2023-11-20
Rev Recd Date: 2023-12-12
Publish Date: 2024-01-31

Abstract

Abstract

With the advancement of meteorological services, various product processing systems and supporting data management systems have been developed for different business systems. However, this has led to the problem of system non-intensification. The lack of intensification in meteorological services can result in inconsistent data standards and make the operation and maintenance more challenging, which can lead to significant waste in investment due to duplicate data storage and inconsistent data caused by untimely synchronization. Moreover, the lack of information and technology hinders the integration of upstream and downstream businesses. The intensive development of meteorological business systems and the reform of "cloud+end" technology system are important measures to achieve high-quality development of meteorological business. China Meteorological Administration proposed to build a "cloud+end" technology system in 2020. CMA Big Data and Cloud Platform serves as the cloud, while the meteorological business system is the end, clarifying the positioning of CMA Big Data and Cloud Platform as a key foundational technology platform. The data processing line (DPL) is an intensive environment for meteorological algorithms. It addresses business needs such as efficient and stable processing of data products, data sharing and collaboration among business systems, and efficient and intensive business applications. To achieve this goal, algorithm libraries and task control functions are established by utilizing technologies such as integrated digital and computing, efficient task scheduling, visual process arrangement, and containers. The algorithm library facilitates the standardization, unified management and sharing of algorithms. Task control supports multiple scheduling strategies with high reliability and fault tolerance, enabling efficient and stable scheduling operations for algorithms. All functions mentioned above are in the form of interfaces for the application frontend. At the same time, based on the meteorological business comprehensive monitoring system (referred to as Tianjing) enables automatic collection of algorithm operation status and detection of abnormal alarms. Since its operation in 2021, the processing assembly line has facilitated the real-time operation of 202 business systems nationwide, resulting in a performance improvement of 1-10 times and a significant increase in efficiency. It plays an important supporting role in improving the operational efficiency of business systems, enhancing their collaboration, accelerating the process of "cloud+end" business technology system reform, and promoting the intensive development of meteorological business.
- data processing line;
- meteorology algorithm library;
- task scheduling;
- data and calculation integration;
- intensive development;
- "cloud+end" technical system reform

FullText(HTML)

References(41)

Figures and Tables

图 1 加工流水线框架

Fig. 1 Framework of data processing line

DownLoad: Full-Size Img PowerPoint

图 2 加工流水线算法管理

Fig. 2 Algorithm management of data processing line

DownLoad: Full-Size Img PowerPoint

图 3 加工流水线任务定义

Fig. 3 Job definition of data processing line

DownLoad: Full-Size Img PowerPoint

图 4 加工流水线任务调度

Fig. 4 Task scheduling of data processing line

DownLoad: Full-Size Img PowerPoint

图 5 基于加工流水线功能开放接口构建“云+端”业务架构

Fig. 5 "cloud+end" business architecture based on open functional interface of data processing line

DownLoad: Full-Size Img PowerPoint

图 6 加工流水线容器技术

Fig. 6 Key technology of data processing line-docker and K8s

DownLoad: Full-Size Img PowerPoint

图 7 加工流水线天气雷达产品加工任务流程示意图

Fig. 7 Task flow of weather radar products on data processing line

DownLoad: Full-Size Img PowerPoint

图 8 截至2023年11月底国省融入加工流水线的算法数量(省级取前10名)

Fig. 8 Number of meteorological algorithms integrated into data processing line from national-level and provincial-level by the end of Nov 2023(only the top 10 provinces are shown)

DownLoad: Full-Size Img PowerPoint

图 9 基于加工流水线的天气雷达拼图系统V3.0优化效果

(a)原流程，(b)现流程

Fig. 9 Optimization of weather radar mosaic system V3.0 based on data processing line

(a)the original,(b)the current

DownLoad: Full-Size Img PowerPoint

References

[1]	Zheng G G. Promoting meteorological modernization by informatization. J Zhejiang Meteor, 2015, 36(2): 1-4.
[2]	Shen W H, Zhao F, Gao H Y, et al. The construction of national meteorological archival and retrieval system. J Appl Meteor Sci, 2004, 15(6): 727-736. http://qikan.camscma.cn/article/id/20040690
[3]	Li J M, Shen W H, Wang G F. Meteorological information sharing service platform and its key technologies. J Appl Meteor Sci, 2006, 17(5): 621-628. http://qikan.camscma.cn/article/id/200605105
[4]	Xiong A Y, Zhao F, Wang Y, et al. Design and implementation of China Integrated Meteorological Information Sharing System(CIMISS). J Appl Meteor Sci, 2015, 26(4): 500-512. doi: 10.11898/1001-7313.20150412
[5]	Zhao F, Xiong A Y, Zhang X Y, et al. Technical characteristics of the architecture design of China Integrated Meteorological Information Sharing System. J Appl Meteor Sci, 2017, 28(6): 750-758. doi: 10.11898/1001-7313.20170610
[6]	Sun C, Huo Q, Ren Z H, et al. Design and implementation of surface meteorological data statistical processing system. J Appl Meteor Sci, 2018, 29(5): 630-640. doi: 10.11898/1001-7313.20180511
[7]	Wu H P, Zhang Y Q, Sun J M, et al. Designing and implementation of Climate Interactive Plotting and Analysis System. J Appl Meteor Sci, 2013, 24(5): 631-640. http://qikan.camscma.cn/article/id/20130513
[8]	Wang R T, Wang J M, Huang X D, et al. The architecture design of MICAPS4 server system. J Appl Meteor Sci, 2018, 29(1): 1-12. doi: 10.11898/1001-7313.20180101
[9]	Han F, Wo W F. Design and implementation of SWAN2.0 platform. J Appl Meteor Sci, 2018, 29(1): 25-34. doi: 10.11898/1001-7313.20180103
[10]	Wu M X, Zhuang L W, Hou Y Y, et al. The design and implementation of China Agricultural Meteorological Service System(CAgMSS). J Appl Meteor Sci, 2019, 30(5): 513-527. doi: 10.11898/1001-7313.20190501
[11]	Li D Q, Li K K, Li H Y, et al. Design and implementation of mobile application for real-time monitoring of weather-modification aircraft operations. J Appl Meteor Sci, 2019, 30(6): 745-758. doi: 10.11898/1001-7313.20190610
[12]	Ma Q, Yan J H, Wei M, et al. Implementation and application of BCC CMIP6 Experimental Data Sharing Platform. J Appl Meteor Sci, 2022, 33(5): 617-627. doi: 10.11898/1001-7313.20220509
[13]	Li Y, Wang G F. Design and implementation of Meteorological Disaster Risk Management System. J Appl Meteor Sci, 2022, 33(5): 628-640. doi: 10.11898/1001-7313.20220510
[14]	Zhang J, Sun J, Shen X S, et al. Key model technologies of CMA-GFS V4.0 and application to operational forecast. J Appl Meteor Sci, 2023, 34(5): 513-526. doi: 10.11898/1001-7313.20230501
[15]	Yang H P, Zhang Q, Luo B, et al. Construction and application of Meteorological Integrated Command Platform. J Appl Meteor Sci, 2023, 34(1): 117-128. doi: 10.11898/1001-7313.20230110
[16]	Liang L, Ma S Q, Teng Y P, et al. Construction and application of Weather Radar Aerial Ecological Monitoring System. J Appl Meteor Sci, 2023, 34(5): 630-640. doi: 10.11898/1001-7313.20230511
[17]	Shen W H. Significance of cloud computing in the process of meteorological informatization. Chinese Informatization, 2015(3): 80-88.
[18]	Shen W H. Discussion on the future infrastructure of meteorological business information system-The role of "cloud computing" and "big data" in meteorological informatization. Adv Meteor Sci Tech, 2015, 5(3): 64-66.
[19]	Zhang G C. Study and thinking on the construction of meteorological cloud. Meteor Environ Sci, 2015, 38(4): 1-11. doi: 10.3969/j.issn.1673-7148.2015.04.001
[20]	Nie F Y. The research of the big data resources technical services collaboration-A case study of meteorological data. Informatization Res, 2016, 42(1): 6-11.
[21]	Shen W H. Understanding and re-understanding of meteorological informatization. Adv Meteor Sci Tech, 2013, 3(5): 56-62.
[22]	Zhou Y, Liu D J, Ma F B. Research on the framework of meteorological informatization standard system. Chinese Informatization, 2016(4): 74-80.
[23]	China Meteorological Administration. Plan of the Meteorological Big Data Action(2017-2020). 2017.
[24]	China Meteorological Administration. Plan of the Meteorological Informatization Development(2018-2022). 2017.
[25]	China Meteorological Administration. Opinions of the China Meteorological Administration on Promoting the Key Reform of Meteorological Technology System. 2020.
[26]	China Meteorological Administration. Plan for the Reform of Meteorological Technology System(2022-2025). 2022.
[27]	China Meteorological Administration. Design of the CMA Big Data and Cloud Platform. 2018.
[28]	Papazoglou M P, van den Heuvel W J. Service oriented architectures: Approaches, technologies and research issues. VLDB J, 2007, 16(3): 389-415. doi: 10.1007/s00778-007-0044-3
[29]	Kivity A, KarrIay Y, Laor D, et al. KVM: The Linux Virtual Machine Monitor//Proceedings of Linux Symposium, 2007, I: 225-230.
[30]	Kozhirbayev Z, Sinnott R O. A performance comparison of container-based technologies for the cloud. Future Gener Comput Syst, 2017, 68: 175-182. doi: 10.1016/j.future.2016.08.025
[31]	Yang P, Ma Z C, Peng B, et al. Performace research of OpenStack cloud platform integrated with Docker container. Comput Eng, 2017, 43(8): 26-31.
[32]	Wu Y. Research on virtualization technology based on Docker. Inf Technol, 2016, 40(1): 121-123;128.
[33]	Salza P, Ferrucci F. Speed up genetic algorithms in the cloud using software containers. Future Gener Comput Syst, 2019, 92: 276-289. doi: 10.1016/j.future.2018.09.066
[34]	Wan X L, Guan X J, Wang T J, et al. Application deployment using Microservice and Docker containers: Framework and optimization. J Netw Comput Appl, 2018, 119: 97-109. doi: 10.1016/j.jnca.2018.07.003
[35]	De Benedictis M, Lioy A. Integrity verification of Docker containers for a lightweight cloud environment. Future Gener Comput Syst, 2019, 97: 236-246. doi: 10.1016/j.future.2019.02.026
[36]	Felter W, Ferreira A, Rajamony R, et al. An Updated Performance Comparison of Virtual Machines and Linux Containers//2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2015: 171-172.
[37]	Joy A M. Performance Comparison between Linux Containers and Virtual Machines//Proceedings of 2015 International Conference on Advances in Computer Engineering and Applications. IEEE, 2015: 342-346.
[38]	Merkel D. Docker: Lightweight Linux containers for consistent development and deployment. Linux J, 2014, 2014(239): No2.
[39]	Jaramillo D, Nguyen D V, Smart R. Leveraging Microservices Architecture by Using Docker Technology//Southeast Con. IEEE, 2016: 1-5.
[40]	JGraph Ltd. User Manual(version 4.2.2-28). 2020.
[41]	Xu Y J, Ni X L, Zheng B, et al. Design and application of live weather data service interface based on user's location. Comput Syst Appl, 2023, 32(5): 77-86.