Application of the MARS to Data Management of NWP Productions

Xiao Huadong; Sun Jing; Zhang Xi; Bian Xiaofeng

doi:10.11898/1001-7313.20150213

Vol.26, NO.2, 2015

Turn off MathJax

Article Contents

Article Navigation > Journal of Applied Meteorological Science > 2015 > 26(2): 247-256

Xiao Huadong, Sun Jing, Zhang Xi, et al. Application of the MARS to data management of NWP productions. J Appl Meteor Sci, 2015, 26(2): 247-256. DOI: 10.11898/1001-7313.20150213.

Citation:

Xiao Huadong, Sun Jing, Zhang Xi, et al. Application of the MARS to data management of NWP productions. J Appl Meteor Sci, 2015, 26(2): 247-256. DOI: 10.11898/1001-7313.20150213.

Xiao Huadong, Sun Jing, Zhang Xi, et al. Application of the MARS to data management of NWP productions. J Appl Meteor Sci, 2015, 26(2): 247-256. DOI: 10.11898/1001-7313.20150213.

Citation:

Xiao Huadong, Sun Jing, Zhang Xi, et al. Application of the MARS to data management of NWP productions. J Appl Meteor Sci, 2015, 26(2): 247-256. DOI: 10.11898/1001-7313.20150213.

PDF( 1148 KB)

Application of the MARS to Data Management of NWP Productions

DOI: 10.11898/1001-7313.20150213

National Meteorological Information Center, Beijing 100081

Received Date: 2014-04-17
Rev Recd Date: 2016-11-18
Publish Date: 2015-03-31

Abstract

Abstract

The NWP model runs several times every day, creating a large number of files of hundred megabytes each. In the current national meteorological data management system, the NWP model data are managed by the combination of file system and database system, which improves the operation of relative data compared to the traditional file system or database system alone. But data are separated from their index and description, which makes it difficult to manage the massive data of NWP productions.The MARS (Meteorological Archive and Retrieve System) based on the description of meteorology can be used to resolve the above problems. The client-server architecture of the MARS is introduced briefly, and functional components of the MARS are also explained in detail. A data management solution for the national operational NWP model is designed based on the current version of the MARS. According to the characteristic of table-driven code formats, the method of creating GRIB2 code tables and templates is introduced. The application programming interface (API) between the MARS and the Tivoli Storage Manager (TSM) is studied and implemented, considering the I/O feature of disk and tape, and API achieves direct data flushing and retrieving. Taking GRIB2 data of GRAPES global model production for example, data management is implemented by the MARS. Performance test indicates that the MARS can accelerate retrieving of certain data by over one time, demonstrating great advantages of time efficiency.The MARS integrates the meteorological content orient data management technique and the hierarchical storage management technique. Its adaptability and scalability are also verified through long time application in many meteorological centers, showing wider application prospect in the field of NWP production data management.
- the MARS;
- GRIB2 template;
- storage management software;
- data flushing and retrieving

FullText(HTML)

References(22)

Figures and Tables

Fig. 1 Architecture of the MARS (from Reference[11])

DownLoad: Full-Size Img PowerPoint

Fig. 2 Archive object cube model of meteorological data in the MARS (from Reference [11])

DownLoad: Full-Size Img PowerPoint

Fig. 3 Archive object hypercube model of meteorological data in the MARS (from Reference [11])

DownLoad: Full-Size Img PowerPoint

Fig. 4 Metadata of request manager and data manager in the MARS (from Reference [11])

DownLoad: Full-Size Img PowerPoint

Fig. 5 The architecture of NWP data management system

DownLoad: Full-Size Img PowerPoint

Fig. 6 The flow chart for reading or writing a tape file in the API between the MARS and a storage manager system

DownLoad: Full-Size Img PowerPoint

Fig. 7 The flow chart of archiving GRIB2 data of model products into the MARS

DownLoad: Full-Size Img PowerPoint

Table 1 Attributes for meteorological parameter fields

要素场属性	示例取值
类别 (Class)	Operational
版本 (Version)	1
数据流 (Stream)	Daily archive
类型 (Type)	Forecast
要素 (Parameter)	Temperature
层次类型 (Level Type)	Pressure level
层次 (Level)	1000 hPa
日期 (Date)	1993-08-10
时次 (Base Time)	1200
预报时效 (Time Step)	120 h
成员 (Member for EPS)	42

DownLoad: Download CSV

Table 2 Performance of retrieving T639L60 model production by the data management system of CIMISS and the MARS based system

系统	要素	GRIB2消息数	数据量/MB	耗时/s
CIMISS	温度	32	5.23	6
CIMISS	全部	344	61.58	60
MARS	温度	32	5.23	2.85
MARS	全部	344	61.58	28.94

DownLoad: Download CSV

References

[1]	陈德辉, 薛纪善.数值天气预报业务模式现状与展望.气象学报, 2004, 62(5):623-633. doi: 10.11676/qxxb2004.061
[2]	矫梅燕, 龚建东, 周兵, 等.天气预报的业务技术进展.应用气象学报, 2006, 17(5):594-601. doi: 10.11898/1001-7313.20060502
[3]	张玺.并行文件系统下数据迁移功能的实现.北京信息科技大学学报, 2012, 12(5):77-80. http://www.cnki.com.cn/Article/CJFDTOTAL-BJGY201205018.htm
[4]	沈文海, 赵芳, 高华云, 等.国家级气象资料存储检索系统的建立.应用气象学报, 2004, 15(6):727-736. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20040690&flag=1
[5]	李集明, 沈文海, 王国复.气象信息共享平台及其关键技术研究.应用气象学报, 2006, 17(5):621-628. doi: 10.11898/1001-7313.20060505
[6]	国家气象信息中心. 新一代天气雷达信息共享平台总体设计. 北京: 中国气象局, 2009.
[7]	杨润芝, 马强, 李德泉, 等.内存转发模型在CIMISS数据收发系统中的应用.应用气象学报, 2012, 23(3):377-384. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20120315&flag=1
[8]	Raoult B.Data Handling via MARS at ECMWF.ECMWF Newsletter, 1996, 72:15-19. http://www.ecmwf.int/
[9]	Raoult B.Implementing MARS using object-oriented technology.ECMWF Newsletter, 1997, 75:9-10. http://www.advancedtraining.com.au/training-courses/elioocjpl-implementing-objectoriented-concepts-in-the-java-programming-language-wj1103se6-sun-v20.php
[10]	Raoult B.MARS, ECMWF's Meteorological Archive:Experience in Managing a Large Archive.[2013-12-10]. http://vds.cnes.fr/manifestations/PV2002/DATA/1-5_raoult.pdf.
[11]	Raoult B.Architecture of the new MARS server.[2013-12-10]. http://www.ecmwf.int/publications/manuals/mars/server.pdf.
[12]	ECMWF Operations Department.MARS User Guide.[2013-12-20]. http://www.ecmwf.int/publications/manuals/mars/guide/MarsUserGuide.pdf.
[13]	ECMWF EMOSLIB.[2014-09-27]. https://software.ecmwf.int/wiki/display/EMOS/Emoslib.
[14]	伍湘君, 金之雁, 黄丽萍, 等.GRAPES模式软件框架与实现.应用气象学报, 2005, 16(4):539-546. doi: 10.11898/1001-7313.20050415
[15]	麻素红, 吴俞, 瞿安祥, 等.T213与T639模式热带气旋预报误差对比.应用气象学报, 2012, 23(2):167-173. doi: 10.11898/1001-7313.20120205
[16]	闫之辉, 王雨, 朱国富.国家气象中心业务数值预报发展的回顾与展望.气象, 2010, 36(7):26-32. doi: 10.7519/j.issn.1000-0526.2010.07.007
[17]	刘媛媛, 应显勋, 赵芳.GRIB2介绍及解码初探.气象科技, 2006, 34(增刊Ⅰ):61-64. http://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ2006S1014.htm
[18]	国家气象信息中心通信台.表格驱动码编码手册.北京:气象出版社, 2005.
[19]	WMO.Guide to the WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced Data In Binary Form:FM 92 GRIB Edition 2.[2013-12-10]. http://www.wmo.int/pages/prog/www/WMOCodes/Guides/GRIB/GRIB2_062006.pdf.
[20]	WMO.Introduction to GRIB Edition1 and GRIB Edition 2.[2013-12-10]. http://www.wmo.int/pages/prog/www/WMOCodes/Guides/GRIB/Introduction_GRIB1-GRIB2.pdf.
[21]	ECMWF.Grib decoder.[2013-12-10]. https://software.ecmwf.int/wiki/display/GRIB/Home.
[22]	IBM Tivoli Storage Manager for AIX Version 6.2 Using the Application Programming Interface.[2014-09-15]. http://pic.dhe.ibm.com/infocenter/tsminfo/v6r2/topic/com.ibm.itsm.client.develop.doc/b_api_using.pdf.