设为首页
加入收藏
Application of Memory-forwarding Model to Data Transmission System of CIMISS
-
摘要: 针对国家气象信息中心现有数据收发业务在时效方面存在的问题，对现有模式进行剖析，找出现有流程中存在的问题，以及这些问题的产生原因。在此基础上，以气象卫星数据为对象，提出了应用内存转发模型。在该模型中，通过利用内存共享机制将数据接收流程和数据转发流程联系起来，建立通路，减少了多个中间处理环节，提高整个数据转发时效。参照内存转发模型，设计出适用于国家气象信息中心业务流程的统一数据转发流程，并通过对FTP服务端程序模块进行改造，在服务端实现了统一数据转发流程。通过性能测试，对两种转发模式的时效进行对比和分析，验证了内存转发模型的时效优势。Abstract: In China Integrated Meteorological Information Sharing System (CIMISS) Project, the transmission time of data has put forward specific requirements. As one of the 14 categories of data, satellite data and products have more types, higher capacity, and more files amounts comparing with other conventional data files. As in recent years, the number of satellite data and products categories have gradually increased, meanwhile, the scope of applications of satellite data and products is being used more frequently in both weather forecast and research. Therefore, the satellite data transmission process calls for better timeliness.Considering the features of satellite data and products from users' aspects, the current satellite data receiving and forwarding process in National Meteorological Information Center (NMIC) is analyzed, and the limitation of existing procedure is figured out. In the existing satellite data forwarding process, the data receiving process is handled by standard FTP server, and data forwarding process uses standard FTP client. So it makes data receiving and forwarding flow, two relatively independent processes. The data receiving process is only responsible for the various data source reception, which only makes sure that all the arrived data and products could be recognized and transferred to data forwarding process successfully. Also, the data forwarding process only takes the responsibility to constantly scanning the sending directory to check out whether there is data or products to send. It may be concluded that there is no communication between data receiving process and data forwarding process, thus makes the total timeliness of data transmission lower than the ideal value.In order to solve or improve the situation, a new memory-forwarding model is proposed. In this model, by the memory sharing mechanisms, the data receiving process and forwarding process are combined as a unified whole, which has reduced the number of intermediate processing links. Based on memory-forwarding model, a unified procedure is designed by restructuring the FTP server transformation module. Through the development and testing of prototype systems, memory-forwarding model and unified procedure has been fully applied and achieved stage progress, and it will continue being validated and improved in further actual business system.
-
表 1 日接收单个卫星文件容量分析
Table 1 Capacity analysis of single satellite product file received per day
数据容量 文件个数 所占比例/% (0, 100 KB) 1980 11.7 [100 KB, 500 KB) 3298 19.4 [500 KB, 1 MB) 2550 15.0 [1 MB, 10 MB) 4726 27.8 [10 MB, 100 MB) 2400 13.9 [100 MB, 1 GB) 2074 12.2 
下载: 导出CSV
表 2 6种容量文件采用不同方案转发的测试结果
Table 2 Transmission testing result among six capacities of files with two kinds of schema
文件大小 文件数量 转发用户 平均收集
时间/ms平均转发
时间/ms目录轮询
/ms内存转发
/ms节省时间
/ms提高效率
/%50 KB 100 3 47 145 726 198 528 72.73 300 KB 100 3 272 826 1687 1106 581 34.44 800KB 100 3 695 2096 3853 3227 626 16.25 5 MB 10 3 438 1304 2168 1759 409 18.87 20 MB 10 3 1694 5098 7235 6804 431 5.96 500 MB 1 3 4347 13238 18324 17618 706 3.85
下载: 导出CSV
表 3 组合条件测试结果
Table 3 Testing results under different combined testing factors
测试指标说明 收集时间/s 处理时间/s 转发时间/s 并发执行收集、处理、转发总时间/s 内存转发，数据库连接数为5个 206 207 207 211 磁盘转发，数据库连接数为5个 313 320 325 331 内存转发，数据库连接数为10个 204 204 205 209 磁盘转发，数据库连接数为10个 308 305 319 325
下载: 导出CSV
-
[1] 肖文名, 酆薇, 杨根录.气象信息传输业务流程优化的初步分析.应用气象学报, 2009, 20(4):497-503. doi: 10.11898/1001-7313.200904016 [2] 刘聪, 顾建, 吴国平, 等.基于GPRS的远程气象观测数据实时采集传输系统及其应用.应用气象学报, 2004, 15(6):712-718. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20040688&flag=1 [3] 马渝勇, 徐晓莉, 宋智, 等.省级气象信息共享系统的设计与实现.应用气象学报, 2011, 22(4):505-512. doi: 10.11898/1001-7313.20110414 [4] 邓国, 龚建东, 邓莲堂.国家级区域集合预报系统研发和性能检验.应用气象学报, 2010, 21(5):513-523. doi: 10.11898/1001-7313.20100501 [5] 程宏波, 伦利, 郑宗校.基于内存文件系统的分布式文件服务器Cache系统.华东交通大学学报, 2009, 26(2):68-72. http://www.cnki.com.cn/Article/CJFDTOTAL-HDJT200902014.htm [6] 张红兵, 魏波.大容量内存文件系统设计及μC/OS下的实现.单片机与嵌入式系统应用, 2004, 39(3):13-15;18. http://www.cnki.com.cn/Article/CJFDTOTAL-DPJY200403003.htm [7] 李强, 杜威, 慕春棣.基于大容量闪存的嵌入式文件系统.计算机工程, 2005, 31(10):87-89. doi: 10.3969/j.issn.1007-130X.2005.10.031 [8] AIX6.1 Networks and Communication Management. IBM Corporation, 2010: 262-264. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=200904016&flag=1 [9] David H. Guide to Multiple Page Size Support on AIX 5L Version 5.3. IBM Corporation, 2006: 4-10. [10] 钱建梅, 郑旭东.国家卫星气象中心气象卫星资料存档系统.应用气象学报, 2003, 14(6):756-762. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20030695&flag=1 [11] 赵立成, 关彤.网络编程实现气象卫星资料的传输.应用气象学报, 2003, 14(4):395-401. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20030449&flag=1 [12] 徐建平.气象卫星资料接收及其发展.应用气象学报, 1999, 10(3):361-367. http://www.cnki.com.cn/Article/CJFDTOTAL-YYQX903.014.htm [13] 龚道永, 朱建涛.大规模机群中基于扩展FTP的文件管理.计算机工程, 2005, 31(7): 97-98;235. http://www.cnki.com.cn/Article/CJFDTOTAL-JSJC200507032.htm [14] 陈柱成, 郝克刚, 葛玮, 等.用RPC_FTP实现分布式系统中的文件传输.计算机应用与软件, 2007, 24(3):74-76. http://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ200703025.htm [15] 沈建华, 罗悦怿. Flash文件系统的研究与设计.计算机应用研究, 2004(12):246-248. doi: 10.3969/j.issn.1001-3695.2004.12.087 -
图(4) / 表(3)
计量
- 摘要浏览量: 3351
- HTML全文浏览量: 1028
- PDF下载量: 1613
- 被引次数: 0
