Yang Runzhi, Ma Qiang, Li Dequan, et al. Application of memory-forwarding model to data transmission system of CIMISS. J Appl Meteor Sci, 2012, 23(3): 377-384.
Citation: Yang Runzhi, Ma Qiang, Li Dequan, et al. Application of memory-forwarding model to data transmission system of CIMISS. J Appl Meteor Sci, 2012, 23(3): 377-384.

Application of Memory-forwarding Model to Data Transmission System of CIMISS

  • Received Date: 2011-09-20
  • Rev Recd Date: 2012-02-15
  • Publish Date: 2012-06-30
  • In China Integrated Meteorological Information Sharing System (CIMISS) Project, the transmission time of data has put forward specific requirements. As one of the 14 categories of data, satellite data and products have more types, higher capacity, and more files amounts comparing with other conventional data files. As in recent years, the number of satellite data and products categories have gradually increased, meanwhile, the scope of applications of satellite data and products is being used more frequently in both weather forecast and research. Therefore, the satellite data transmission process calls for better timeliness.Considering the features of satellite data and products from users' aspects, the current satellite data receiving and forwarding process in National Meteorological Information Center (NMIC) is analyzed, and the limitation of existing procedure is figured out. In the existing satellite data forwarding process, the data receiving process is handled by standard FTP server, and data forwarding process uses standard FTP client. So it makes data receiving and forwarding flow, two relatively independent processes. The data receiving process is only responsible for the various data source reception, which only makes sure that all the arrived data and products could be recognized and transferred to data forwarding process successfully. Also, the data forwarding process only takes the responsibility to constantly scanning the sending directory to check out whether there is data or products to send. It may be concluded that there is no communication between data receiving process and data forwarding process, thus makes the total timeliness of data transmission lower than the ideal value.In order to solve or improve the situation, a new memory-forwarding model is proposed. In this model, by the memory sharing mechanisms, the data receiving process and forwarding process are combined as a unified whole, which has reduced the number of intermediate processing links. Based on memory-forwarding model, a unified procedure is designed by restructuring the FTP server transformation module. Through the development and testing of prototype systems, memory-forwarding model and unified procedure has been fully applied and achieved stage progress, and it will continue being validated and improved in further actual business system.
  • Fig. 1  The current data transmission process

    Fig. 2  Memory-forwarding model

    Fig. 3  The unified data transmission process

    Fig. 4  The multi-thread parallel mode

    Table  1  Capacity analysis of single satellite product file received per day

    数据容量 文件个数 所占比例/%
    (0, 100 KB) 1980 11.7
    [100 KB, 500 KB) 3298 19.4
    [500 KB, 1 MB) 2550 15.0
    [1 MB, 10 MB) 4726 27.8
    [10 MB, 100 MB) 2400 13.9
    [100 MB, 1 GB) 2074 12.2
    DownLoad: Download CSV

    Table  2  Transmission testing result among six capacities of files with two kinds of schema

    文件大小 文件数量 转发用户 平均收集
    时间/ms
    平均转发
    时间/ms
    目录轮询
    /ms
    内存转发
    /ms
    节省时间
    /ms
    提高效率
    /%
    50 KB 100 3 47 145 726 198 528 72.73
    300 KB 100 3 272 826 1687 1106 581 34.44
    800KB 100 3 695 2096 3853 3227 626 16.25
    5 MB 10 3 438 1304 2168 1759 409 18.87
    20 MB 10 3 1694 5098 7235 6804 431 5.96
    500 MB 1 3 4347 13238 18324 17618 706 3.85
    DownLoad: Download CSV

    Table  3  Testing results under different combined testing factors

    测试指标说明 收集时间/s 处理时间/s 转发时间/s 并发执行收集、处理、转发总时间/s
    内存转发,数据库连接数为5个 206 207 207 211
    磁盘转发,数据库连接数为5个 313 320 325 331
    内存转发,数据库连接数为10个 204 204 205 209
    磁盘转发,数据库连接数为10个 308 305 319 325
    DownLoad: Download CSV
  • [1]
    肖文名, 酆薇, 杨根录.气象信息传输业务流程优化的初步分析.应用气象学报, 2009, 20(4):497-503. doi:  10.11898/1001-7313.200904016
    [2]
    刘聪, 顾建, 吴国平, 等.基于GPRS的远程气象观测数据实时采集传输系统及其应用.应用气象学报, 2004, 15(6):712-718. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20040688&flag=1
    [3]
    马渝勇, 徐晓莉, 宋智, 等.省级气象信息共享系统的设计与实现.应用气象学报, 2011, 22(4):505-512. doi:  10.11898/1001-7313.20110414
    [4]
    邓国, 龚建东, 邓莲堂.国家级区域集合预报系统研发和性能检验.应用气象学报, 2010, 21(5):513-523. doi:  10.11898/1001-7313.20100501
    [5]
    程宏波, 伦利, 郑宗校.基于内存文件系统的分布式文件服务器Cache系统.华东交通大学学报, 2009, 26(2):68-72. http://www.cnki.com.cn/Article/CJFDTOTAL-HDJT200902014.htm
    [6]
    张红兵, 魏波.大容量内存文件系统设计及μC/OS下的实现.单片机与嵌入式系统应用, 2004, 39(3):13-15;18. http://www.cnki.com.cn/Article/CJFDTOTAL-DPJY200403003.htm
    [7]
    李强, 杜威, 慕春棣.基于大容量闪存的嵌入式文件系统.计算机工程, 2005, 31(10):87-89. doi:  10.3969/j.issn.1007-130X.2005.10.031
    [8]
    AIX6.1 Networks and Communication Management. IBM Corporation, 2010: 262-264. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=200904016&flag=1
    [9]
    David H. Guide to Multiple Page Size Support on AIX 5L Version 5.3. IBM Corporation, 2006: 4-10.
    [10]
    钱建梅, 郑旭东.国家卫星气象中心气象卫星资料存档系统.应用气象学报, 2003, 14(6):756-762. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20030695&flag=1
    [11]
    赵立成, 关彤.网络编程实现气象卫星资料的传输.应用气象学报, 2003, 14(4):395-401. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20030449&flag=1
    [12]
    徐建平.气象卫星资料接收及其发展.应用气象学报, 1999, 10(3):361-367. http://www.cnki.com.cn/Article/CJFDTOTAL-YYQX903.014.htm
    [13]
    龚道永, 朱建涛.大规模机群中基于扩展FTP的文件管理.计算机工程, 2005, 31(7): 97-98;235. http://www.cnki.com.cn/Article/CJFDTOTAL-JSJC200507032.htm
    [14]
    陈柱成, 郝克刚, 葛玮, 等.用RPC_FTP实现分布式系统中的文件传输.计算机应用与软件, 2007, 24(3):74-76. http://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ200703025.htm
    [15]
    沈建华, 罗悦怿. Flash文件系统的研究与设计.计算机应用研究, 2004(12):246-248. doi:  10.3969/j.issn.1001-3695.2004.12.087
  • 加载中
  • -->

Catalog

    Figures(4)  / Tables(3)

    Article views (3351) PDF downloads(1613) Cited by()
    • Received : 2011-09-20
    • Accepted : 2012-02-15
    • Published : 2012-06-30

    /

    DownLoad:  Full-Size Img  PowerPoint