Research and Implementation of Remote Sensing Big Data Distributed Technology

In the past ten years, the global various digital information grows explosively, and a big data era with massive data production, sharing and application is opened. In this decade, with the development of information technology, distributed storage and computing technology get great development to deal with the explosive growth of information, and the knowledge system and technical reserves are established gradually. In China, research on big data and distributed computing is being carried out widely. For satellite remote sensing data of large volume and rapid growth, the traditional archive-callback-application cannot meet demands of data analysis and data mining in the era of big data.The traditional file-based way has many limitations, especially when used for cloud computing and in-telligent services, and it is very difficult to use. The big data grid model and distributed model is the key to solve the bottleneck, enabling real-time computing and on-demand services, and therefore it has important reference significance. It overcomes the temporal and spatial fragmentation problem, making the remote sensing data possible to be stored, calculated and applied as a whole. Based on the Hilbert curve grid hash algorithm, a distributed system containing fundamental structure of grid, time slice and physical layer is established, demonstrating excellent parallel read-write performance. Hilbert hash algorithm has stable discrete degree, which is the key for the grid model to maintain spatial correlation and to map two-dimensional space to one-dimensional sequence.Using the distributed system, instead of traditional way of data file organization and management, properties flexible and intuitive data acquisition are realized. Users can truly experience a new way of what you see is what you get and what you get is what you need to get. The future system which is based on the data model, will greatly increase the work efficiency, make the focus from the data itself to data applications. Internet-based cloud computing grid cell calculation can be realized, and the extension ability of the whole system can achieve linear growth, based on the general hardware and software platform. The implementation of this system will greatly improve the work efficiency, completing high-speed parallel data reading and writing, making on-demand data application more smoothly.