Vol.35, NO.6, 2024
Turn off MathJax
Article Contents
Article Navigation >  Journal of Applied Meteorological Science  > 2024  >  35(6): 641-653
Zhu Enda, Wang Yaqiang, Zhao Yan, et al. Evaluation of weather forecasts from AI big models over East Asia. J Appl Meteor Sci, 2024, 35(6): 641-653. DOI:  10.11898/1001-7313.20240601.
Citation: Zhu Enda, Wang Yaqiang, Zhao Yan, et al. Evaluation of weather forecasts from AI big models over East Asia. J Appl Meteor Sci, 2024, 35(6): 641-653. DOI:  10.11898/1001-7313.20240601.
shu

Evaluation of Weather Forecasts from AI Big Models over East Asia

DOI: 10.11898/1001-7313.20240601
  • 1.

    State Key Laboratory of Severe Weather, Chinese Academy of Meteorological Sciences, Beijing 100081

  • 2.

    Xiong'an Institute of Meteorological Artificial Intelligence, Xiong'an New Area 071000

  • Received Date: 2024-07-08
  • Rev Recd Date: 2024-10-10
  • Publish Date: 2024-11-30
    Abstract
  • Reliable medium-range weather forecasts are crucial for both science and society. Although weather predictions primarily rely on numerical weather models, the artificial intelligence (AI) weather big models have shown potential for accurate weather forecasts with reduced computational costs. However, prediction skills of big models remain uncertain, particularly in East Asia, which limits further application of weather AI models. To systematically evaluate predictive capabilities of Pangu, FuXi, and GraphCast models over East Asia, their prediction results are focusing on 500 hPa geopotential height, 2 m air temperature, 10 m wind speed, precipitation, and track of tropical cyclones.ECMWF reanalysis V5 (ERA5) datasets are utilized to provide the initial conditions for big models, and to assess their predictive skill. Additionally, precipitation observations and China Meteorological Administration tropical cyclone datasets are utilized to access big models as well. FuXi shows the highest forecasting skills among 3 big models for 500 hPa geopotential height. The forecast from FuXi is reliable for up to 9.75 days, while the forecasts from Pangu and GraphCast are reliable for 8.75 days and 8.5 days, respectively. For 2 m air temperature forecasting, FuXi presents higher skills with an averaged temporal anomaly correlation coefficient (TCC) ranging from 0.48 to 0.91, while TCCs of Pangu and GraphCast are 0.43-0.91 and 0.38-0.83, respectively. Among 3 models, only FuXi and GraphCast provide precipitation forecasts. FuXi shows higher prediction skill compared to GraphCast in forecasting precipitation, light rain, and moderate rain; however, GraphCast has advantage in heavy rain forecast. As the lead time increases, the threat scores (TSs) of FuXi for rainfall, light rainfall and moderate rainfall are 0.22-0.41, 0.15-0.24 and 0.06-0.22, respectively. The model demonstrates higher skill in the northern and southeastern regions of China. For predicting the track of cyclones, Pangu model demonstrates superior predictive skill. As the lead time increases from 6 hours to 240 hours, biases of Pangu's prediction track increase from 17.5 km to 1850 km.The study focuses on the prediction skill of various AI big models through TCC, spatial anomaly correlation coefficient, and TS. Generally, the performance of FuXi is superior for most elements. And reasonable evaluation of AI model is helpful for the development of AI models.
    • FullText(HTML)
    References(41)

  • Fig. 1  500 hPa geopotential height responsed to tropical anomalous heating of 0.1 K within region outlined by the black line integrated for 5 d, 10 d and 20 d in winter

    DownLoad: Full-Size Img PowerPoint

    Fig. 2  Spatial anomaly correlation coefficients of 500 hPa geopotential height forecast over East Asia

    DownLoad: Full-Size Img PowerPoint

    Fig. 3  Temporal anomaly correlation coefficients and normalized root mean square errors of 500 hPa geopotential height forecast over East Asia at lead times of 5 d and 10 d

    DownLoad: Full-Size Img PowerPoint

    Fig. 4  Error distributions of 2 m temperature and 10 m wind speed forecasts at lead times of 5 d and 10 d

    DownLoad: Full-Size Img PowerPoint

    Fig. 5  Temporal anomaly correlation coefficients and normalized root mean square errors of 2 m temperature forecast at lead times of 5 d and 10 d

    DownLoad: Full-Size Img PowerPoint

    Fig. 6  Temporal anomaly correlation coefficients and normalized root mean square errors of 10 m wind speed forecast at lead times of 5 d and 10 d

    DownLoad: Full-Size Img PowerPoint

    Fig. 7  TS of rainfall, light rainfall, moderate rainfall, and heavy rainfall forecast based on FuXi and GraphCast at different lead times

    DownLoad: Full-Size Img PowerPoint

    Fig. 8  Heavy rainfall (greater than 7.5 mm) forecast and observation from FuXi and GraphCast at different lead times

    DownLoad: Full-Size Img PowerPoint

    Fig. 9  Heavy rainfall observations of North China on 29 Jul 2019 and of Leshan from 16 Aug to 17 Aug in 2020

    DownLoad: Full-Size Img PowerPoint

    Fig. 10  Rainfall forecast biases initiated at different time based on FuXi and GraphCast during heavy rainfall of North China on 29 Jul 2019

    DownLoad: Full-Size Img PowerPoint

    Fig. 11  Rainfall forecast biases initiated at different time based on FuXi and GraphCast during heavy rainfall of Leshan from 16 Aug to 17 Aug in 2020

    DownLoad: Full-Size Img PowerPoint

    Fig. 12  Mean biases of tropical cyclone tracks over Northwest Pacific of 2019 (33 cyclones) and 2020 (26 cyclones) at different lead times

    DownLoad: Full-Size Img PowerPoint
  • [1]
    Shen X S, Wang J J, Li Z C, et al. China's independent and innovative development of numerical weather prediction. Acta Meteor Sinica, 2020, 78(3): 451-476.
    [2]
    Zeng Q C. Weather forecast-From empirical to physico-mathematical theory and super-computing system engineering. Physics, 2013, 42(5): 300-314.
    [3]
    Ebert-Uphoff I, Hilburn K. The outlook for AI weather prediction. Nature, 2023, 619(7970): 473-474. doi:  10.1038/d41586-023-02084-9
    [4]
    Ma L M. Development of artificial intelligence technology in weather forecast. Adv Earth Sci, 2020, 35(6): 551-560.
    [5]
    Zeng X T, Zou C X, Fan J, et al. Short-term precipitation correction based on GRU deep learning. J Appl Meteor Sci, 2024, 35(5): 513-525.
    [6]
    Li D, Lin W, Liu Q, et al. Application of machine learning to statistical evaluation of artificial rainfall enhancement. J Appl Meteor Sci, 2024, 35(1): 118-128.
    [7]
    Yin X Y, Hu Z Q, Zheng J F, et al. Filling in the dual polarization radar echo occlusion based on deep learning. J Appl Meteor Sci, 2022, 33(5): 581-593.
    [8]
    Wu P, Chang X T, Yuan W Y, et al. Fast data assimilation(FDA): Data assimilation by machine learning for faster optimize model state. J Comput Sci, 2021, 51. DOI:  10.1016/J.JOCS.2021.101323.
    [9]
    Bonavita M, Laloyaux P. Machine learning for model error inference and correction. J Adv Model Earth Syst, 2020, 12(12). DOI:  10.1029/2020MS002232.
    [10]
    Zhang H, Wang Y Q, Chen D D, et al. Temperature forecasting correction based on operational GRAPES-3 km model using machine learning methods. Atmosphere, 2022, 13(2). DOI:  10.3390/atmos13020362.
    [11]
    Xie S, Sun X G, Zhang S P, et al. Precipitation forecast correction in South China based on SVD and machine learning. J Appl Meteor Sci, 2022, 33(3): 293-304.
    [12]
    Hu Y Y, Pang L, Wang Q G. Application of deep learning bias correction method to temperature grid forecast of 7-15 days. J Appl Meteor Sci, 2023, 34(4): 426-437.
    [13]
    Nearing G, Cohen D, Dube V, et al. Global prediction of extreme floods in ungauged watersheds. Nature, 2024, 627(8004): 559-563. doi:  10.1038/s41586-024-07145-1
    [14]
    Zhuang X R, Zheng Y, Wang Y Q, et al. A deep learning-based precipitation nowcast model and its application over East China. Acta Meteor Sinica, 2023, 81(2): 286-303.
    [15]
    Zhang L, Wu L, Li F, et al. Indentification of weather radar abnormal data based on deep learning. J Appl Meteor Sci, 2023, 34(6): 694-705.
    [16]
    Yuan K, Li W J, Pang J. Hail identification technology in eastern Hubei based on decision tree algorithm. J Appl Meteor Sci, 2023, 34(2): 234-245.
    [17]
    Zhao L N, Lu S, Qi D, et al. Daily maximum air temperature forecast based on fully connected neural network. J Appl Meteor Sci, 2022, 33(3): 257-269.
    [18]
    Liu H Z, Xu H, Bao H J, et al. Application of machine learning classification algorithm to precipitation-induced landslides forecasting. J Appl Meteor Sci, 2022, 33(3): 282-292.
    [19]
    Reichstein M, Camps-Valls G, Stevens B, et al. Deep learning and process understanding for data-driven Earth system science. Nature, 2019, 566(7743): 195-204. doi:  10.1038/s41586-019-0912-1
    [20]
    Xiong G Y, Zu F, Bao Y X, et al. Expressway pavement temperature forecast based on LSTM and prior knowledge. J Appl Meteor Sci, 2024, 35(1): 68-79.
    [21]
    Zhang Y C, Long M S, Chen K Y, et al. Skillful nowcasting of extreme precipitation with NowcastNet. Nature, 2023, 619(7970): 526-532. doi:  10.1038/s41586-023-06184-4
    [22]
    Bi K F, Xie L X, Zhang H H, et al. Accurate medium-range global weather forecasting with 3D neural networks. Nature, 2023, 619(7970): 533-538. doi:  10.1038/s41586-023-06185-3
    [23]
    Chen L, Zhong X H, Zhang F, et al. FuXi: A cascade machine learning forecasting system for 15-day global weather forecast. NPJ Clim Atmos Sci, 2023, 6. DOI:  10.1038/s41612-023-00512-1.
    [24]
    Lam R, Sanchez-Gonzalez A, Willson M, et al. Learning skillful medium-range global weather forecasting. Science, 2023, 382(6677): 1416-1421. doi:  10.1126/science.adi2336
    [25]
    Weyn J A, Durran D R, Caruana R. Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere. J Adv Model Earth Syst, 2020, 12(9). DOI:  10.1029/2020MS002109.
    [26]
    Rasp S, Thuerey N. Data-driven medium-range weather prediction with a resnet pretrained on climate simulations: A new model for weather bench. J Adv Model Earth Syst, 2021, 13(2). DOI:  10.1029/2020MS002405.
    [27]
    Xiao H X, Zhang F, Wang Y Q, et al. Nowcasting of cloud images based on generative adversarial network and satellite data. J Appl Meteor Sci, 2023, 34(2): 220-233.
    [28]
    Kurth T, Subramanian S, Harrington P, et al. FourCastNet: Accelerating Global High-Resolution Weather Forecasting Using Adaptive Fourier Neural Operators//Proceedings of the Platform for Advanced Scientific Computing Conference. Davos Switzerland. ACM, 2023: 1-11.
    [29]
    Vaswani A, Shazeer N, Parmar N, et al. Attention is All You Need//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
    [30]
    Zhou J, Cui G Q, Hu S D, et al. Graph neural networks: A review of methods and applications. AI Open, 2020, 1: 57-81. doi:  10.1016/j.aiopen.2021.01.001
    [31]
    Ben Bouallègue Z, Clare M C A, Magnusson L, et al. The rise of data-driven weather forecasting: A first statistical assessment of machine learning-based weather forecasts in an operational-like context. Bull Am Meteor Soc, 2024, 105(6): E864-E883. doi:  10.1175/BAMS-D-23-0162.1
    [32]
    Hersbach H, Bell B, Berrisford P, et al. The ERA5 global reanalysis. Quart J R Meteor Soc, 2020, 146(730): 1999-2049. doi:  10.1002/qj.3803
    [33]
    Sun S, Shi C X, Pan Y, et al. Applicability assessment of the 1998-2018 CLDAS multi-source precipitation fusion dataset over China. J Meteor Res, 2020, 34(4): 879-892. doi:  10.1007/s13351-020-9101-2
    [34]
    Ying M, Zhang W, Yu H, et al. An overview of the China Meteorological Administration tropical cyclone database. J Atmos Ocean Technol, 2014, 31(2): 287-301.
    [35]
    Lu X Q, Yu H, Ying M, et al. Western North Pacific tropical cyclone database created by the China Meteorological Administration. Adv Atmos Sci, 2021, 38(4): 690-699.
    [36]
    Zeng J H, Yuan X, Ji P, et al. Effects of meteorological forcings and land surface model on soil moisture simulation over China. J Hydrol, 2021, 603. DOI:  10.1016/j.jhydrol.2021.126978.
    [37]
    Hakim G J, Masanam S. Dynamical tests of a deep learning weather prediction model. Artif Intell Earth Syst, 2024, 3(3). DOI:  10.1175/AIES-D-23-0090.1.
    [38]
    Hoskins B J, Karoly D J. The steady linear response of a spherical atmosphere to thermal and orographic forcing. J Atmos Sci, 1981, 38(6): 1179-1196.
    [39]
    Shen X S, Chen Q Y, Sun J, et al. Development of operational global medium-range forecast system in National Meteorological Centre. Meteor Mon, 2021, 47(6): 645-654.
    [40]
    China Meteorological Administration. Weather Forecast Verification-Typhoon forecast. GB/T 38308-2019. Beijing: Standards Press of China, 2019: 2-3.
    [41]
    Jiang Q, Li W Y, Fan Z D, et al. Evaluation of the ERA5 reanalysis precipitation dataset over Chinese Mainland. J Hydrol, 2021, 595. DOI: 10.1016/j.jhydrol.2020.125660.
    • Relative Articles
    • Supplements(0)
    • Cited By
  • 加载中
  • -->

Catalog

    Figures(12)

    Get Citation
    PDF
    XML
    Article views (203) PDF downloads(98) Cited by()
    • Received : 2024-07-08
    • Accepted : 2024-10-10
    • Published : 2024-11-30
    Journal Online
    Contact

    © 2020 Journal of Applied Meteorological Science   京ICP备15006967号-1

    Supported by Beijing Renhe Information Technology Co. Ltd

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return