Optimization of Data Cache Function in Beijing Global Information System Center
-
摘要: 世界气象组织信息系统 (WMO Information System,WIS) 是一个支撑全球气象数据交换共享的通用信息服务平台,北京全球信息系统中心作为WIS的核心功能中心之一,必须缓存最近24 h内的WMO全球交换数据以提供高效的数据访问服务。为了检查收集到的全球交换数据的合法性,需要校验每条数据是否存在与之匹配的元数据,这些元数据信息以关系型数据库方式存储在北京全球信息系统中心中。由于每日接收到的WMO全球交换数据文件个数多且收集时间分布不均,大量频繁的数据库查询操作导致处理性能下降,特别在数据密集收集的情况下容易产生较大延迟,直接影响业务的实时性。设计并实现一种基于内存对象缓存的应用优化现有通过数据库查询的校验方式,实现在内存中一次性载入元数据信息,并在内存中完成缓存数据校验的操作,以此来减少磁盘的读写访问,提升处理效率。此外,还通过多线程方法实现与缓存数据相关的功能,使该方案具有良好的扩展性。实际应用表明:数据缓存功能优化后能满足实时业务性能要求。
-
关键词:
- 数据缓存;
- 内存对象;
- 北京全球信息系统中心
Abstract: WMO Information System (WIS) is a coordinated, distributed, global infrastructure for the collection and sharing of information for all WMO and related international programs. As the core center of WIS, each Global Information System Center (GISC) is responsible for the collection and distribution of global exchanging data, and providing data discovery and access service. As a GISC of WIS, a scalable and flexible system is designed and established to satisfy WIS/GISC functionalities in Beijing. Beijing Global Information System Center should hold at least 24-hour WMO global exchanging data files, which could be accessed by authorized users through DAR (data discovery, access and retrieval) services.GISC Beijing has to do validation check for all the global exchanging data files, and only files matching the corresponding metadata could be brought into data cache. The existing approach for the validation is based on database retrieval operation. Currently, there are more than 100000 metadata records stored in the relational database GISC Beijing, while the system receives more than 50000 global exchanging data files in an uneven distribution of collection time. The disadvantage of this way is that frequent database I/O operation would lead to a sharp decline of the system performance, especially when a large number of data entering. Therefore, the approach could not satisfy the requirement of real-time data cache service. Although establishing the table index and multi-threaded mechanism could solve the efficiency of data processing to some extent, it is inevitable that frequent database I/O operation would bring about the performance bottleneck. Therefore, the operation treatment should be optimized.It is a possible way making full use of memory technology to reduce disk I/O cache data and improve the efficiency significantly. Considering the complexity of the mature memory database, a more targeted approach is adopted which is suitable for the scenario of dynamic nature of data with the timeliness requirements. An application is designed and implemented based on the memory object caching technology. When initializing the application, the system loads metadata into memory as a hash table from the database based on stored key/value pairs, organized by the unique bulletin head information of global exchanging data. In this way, the metadata contents are encapsulated as memory objects, thereby providing fast data memory retrieval method. In addition, parallel processing is implemented to extend the functionalities, including data cache logging function and data subscription services.As a result, effects of the optimized function can satisfy the real-time business requirements, reducing the data processing time to an average of less than 5 ms. It also provides an easier way to do extensions by adding memory object using parallel processing.-
Key words:
- data cache;
- memory object;
- GISC Beijing
-
-
[1] 刘华, 周峥嵘.WIS-WMO未来信息系统.气象软科学, 2007, 83:143-150. http://www.cnki.com.cn/Article/CJFDTOTAL-JJSJ199508014.htm [2] Geoff L.The birth of WMO Information System.Bulletin of WMO, 2003, 55(4): 232-238. [3] 李湘, 王甫棣, 姜立鹏, 等.WIS的实现技术研究及应用.气象, 2011, 37(10):1301-1308. doi: 10.7519/j.issn.1000-0526.2011.10.014 [4] 祝婷, 李湘.WMO信息系统中气象元数据的设计与实现.应用气象学报, 2012, 23(2):238-244. doi: 10.11898/1001-7313.20120213 [5] 姜立鹏, 李湘.基于OAI-PMH协议的WMO信息系统元数据同步功能设计与实现.气象科技, 2012, 40(2):185-188. http://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201202009.htm [6] 曹卫.基于XML的空间元数据系统的思考.计算机技术与发展, 2010, 20(7): 32-35. http://www.cnki.com.cn/Article/CJFDTOTAL-WJFZ201007010.htm [7] 周峥嵘, 王琤, 何文春.分布式气象元数据同步系统的探索研究.应用气象学报, 2010, 21(1): 121-128. doi: 10.11898/1001-7313.20100117 [8] 高峰, 王国复, 喻雯, 等.气象数据文件快速下载服务系统的设计与实现.应用气象学报, 2010, 21(2): 243-249. doi: 10.11898/1001-7313.20100215 [9] 王国复, 徐枫, 吴增祥.气象元数据标准与信息发布技术研究.应用气象学报, 2005, 16(1): 114-121. doi: 10.11898/1001-7313.20050115 [10] Wang Fudi, Yao Yan, Li Xiang, et al.Establishment of WMO Information System in Beijing.Lecture Notes in Electrical Engineering, 2012, 100:520-527. doi: 10.1007%2F978-1-4471-2386-6_151 [11] Robert H, David T, Eliot C.WMO Information System Functional Architecture.[2012-07-30].http://www.wmo.int/pages/prog/www/TEM/ET-WISC-Ⅲ/documents/WIS-FuncArch_current.doc. [12] Thomas D, Christian E, Husband R.WMO Information System Compliance Specifications of GISC, DCPC, and NC.[2012-07-30].http://www.wmo.int/pages/prog/www/WIS/documents/TechnicalSpecification1-2.doc. [13] 刘云生, 李国徽.实时内存数据库的装入.软件学报, 2000, 26(4):829-835. http://www.cnki.com.cn/Article/CJFDTOTAL-RJXB200006018.htm [14] 杨润芝, 马强, 李德泉, 等.内存转发模型在CIMISS数据收发系统中的应用.应用气象学报, 2012, 23(3): 377-384. http://qikan.camscma.cn/jams/ch/reader/view_abstract.aspx?file_no=20120315&flag=1 [15] 杨艳, 李炜, 王纯.内存数据库在高速缓存方面的应用.现代电信科技, 2011(12):59-64. doi: 10.3969/j.issn.1002-5316.2011.12.019 [16] 赵玉伟, 赵小雨, 乔木.缓存技术在B/S架构信息系统中的应用.计算机工程, 2008, 34(1):233-235. http://www.cnki.com.cn/Article/CJFDTOTAL-JSJC200801082.htm [17] 张震波, 杨鹤标, 马振华.基于LRU算法的Web系统缓存机制.计算机工程, 2006, 32(19):68-70. doi: 10.3969/j.issn.1000-3428.2006.19.025 [18] 杨燕明, 鲁志军, 陈煜, 等.一种基于哈希索引的内存表模型.计算机应用与软件, 2012, 29(1):215-216. http://www.cnki.com.cn/Article/CJFDTOTAL-JYRJ201201063.htm [19] 王甫棣, 姚燕, 李湘.基于XML的气象数据订阅系统设计.气象科技, 2012, 40(4):591-595. http://www.cnki.com.cn/Article/CJFDTOTAL-QXKJ201204015.htm