CN111857534A - Data transmission method, data storage server and data storage system - Google Patents
Data transmission method, data storage server and data storage system Download PDFInfo
- Publication number
- CN111857534A CN111857534A CN201910335421.2A CN201910335421A CN111857534A CN 111857534 A CN111857534 A CN 111857534A CN 201910335421 A CN201910335421 A CN 201910335421A CN 111857534 A CN111857534 A CN 111857534A
- Authority
- CN
- China
- Prior art keywords
- data
- data storage
- storage server
- online
- database file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 521
- 238000000034 method Methods 0.000 title claims abstract description 58
- 230000005540 biological transmission Effects 0.000 title claims abstract description 37
- 230000008676 import Effects 0.000 claims abstract description 90
- 238000006243 chemical reaction Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 9
- 230000000694 effects Effects 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000013479 data entry Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Abstract
The application provides a data transmission method, an online data storage server, an offline data storage server and a data storage system, wherein a data downloading channel is established between the offline data storage server and the online data storage server, data to be imported is converted into a database file, and the data is downloaded from the offline data storage server to the online data storage server in a database file form, so that the data import time can be effectively shortened, the data import efficiency is improved, and the stability of providing query service for the data storage system is maintained.
Description
Technical Field
The application relates to the technical field of big data processing, in particular to a data transmission method, which comprises a data storage server and a data storage system, wherein the data storage server comprises an online data storage server and an offline data storage server.
Background
With the continuous development of scientific technology, the amount of network data is increased rapidly and explosively at an exponential level, at present, in order to reasonably store a large amount of data, a distributed storage system Hadoop is generally adopted to store large data, the Hadoop is mainly applied in an offline data storage scene, the timeliness of offline data in the Hadoop is poor, and the requirement of online business for low-delay data reading cannot be met, so that the offline data needs to be written into an online storage system first, and then the online storage system provides low-delay query for the online business.
Offline data in the Hadoop of the offline data storage system is written into the online data storage system, mostly, a virtual data storage engine is constructed in the Hadoop to establish connection with a data cluster in the online data storage system, after an offline data item is read, a data request for writing data is constructed and sent to the online data storage system, and the online data storage system can sequentially read the data request and the data item from each internal node. As shown in fig. 1, fig. 1 is a structural diagram of an online data storage system, where the online data storage system generally includes an application client, a VIP node, a proxy node, and a data storage node, and taking the data storage node in the online data storage system as a Redis node as an example, that is, a data cluster in the online data storage system is a Redis cluster, and in a process of writing offline data into the online data storage system from Hadoop, the Redis client needs to be constructed on the Hadoop to establish connection with the Redis cluster in the online data storage system; the Redis client side can construct a Redis request for data writing and send the Redis request to the online data storage system, after the application client side of the online data storage system receives the Redis request, the Redis request can be sequentially forwarded to the VIP node, the Proxy node and the Redis node, and each node reads and writes offline data items according to the Redis request.
Therefore, data is sent to the Redis cluster through one read-write request, each request is forwarded by a plurality of layers and finally reaches the storage node, the link is long, and meanwhile, a large number of write requests are flooded into the data storage nodes in the online data storage system, so that the query delay of the data storage nodes is increased, and the service stability is influenced.
Disclosure of Invention
In view of this, an object of the present application is to provide a data transmission method, an online data storage server, an offline data storage server, and a data storage system, which can directly transmit a database file by converting data into the database file, thereby effectively reducing data importing time, and improving data importing efficiency and stability of the data storage system for providing query services to the outside.
According to one aspect of the present application, an electronic device may include a storage medium and a processor in communication with the storage medium. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is operated, the processor is communicated with the storage medium through the bus, and when the processor executes the machine-readable instructions, one or more of the following operations can be executed:
Establishing a data downloading channel between the online data storage server and the offline data storage server;
acquiring a database file matched with a data import request sent by the online data storage server from the offline data storage server through the data download channel, wherein the database file comprises data contents of a plurality of data blocks, and the data blocks are obtained by dividing data matched with the data import request by the offline data storage server according to topology structure information of each storage node in the online data storage server;
and loading the database file to a preset storage position.
In some embodiments, the machine readable instructions, when executed by a processor, may perform one or more of the following operations:
generating a blank data storage space matched with the file format of the database file in the online data storage server;
and loading the database file into the generated data storage space.
In some embodiments, the machine readable instructions, when executed by a processor, may perform one or more of the following operations:
generating a blank data storage space in the online data storage server;
Converting the file format of the database file into a file format matched with the data storage format of the generated data storage space;
and loading the database file after converting the file format into the generated data storage space.
In some embodiments, the machine readable instructions, when executed by a processor, may perform one or more of the following operations:
and deleting the existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space.
In some embodiments, the machine readable instructions, when executed by a processor, may perform one or more of the following operations:
determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file;
and loading the database file into the determined data storage space.
In some embodiments, the machine readable instructions, when executed by a processor, may perform one or more of the following operations:
determining a generated data storage space matched with the database file;
converting the data in the database file into data in a file format matched with the determined generated data storage space;
And loading the data in the database file after format conversion into the determined generated data storage space.
According to another aspect of the present application, an electronic device may include a storage medium and a processor in communication with the storage medium. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is operated, the processor is communicated with the storage medium through the bus, and when the processor executes the machine-readable instructions, one or more of the following operations can be executed:
establishing a data downloading channel between the offline data storage server and the online data storage server;
acquiring topological structure information of each storage node in the online data storage server;
dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information, and adding the data contents of the data blocks into a database file;
and sending the database file to the online data storage server through the data downloading channel.
In some embodiments, the machine readable instructions, when executed by a processor, may perform one or more of the following operations:
Determining the data bearing attribute and the data bearing capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
and dividing the data set matched with the data import request into a plurality of sub data blocks matched with the storage nodes based on the data bearing attribute and the data bearing capacity of each storage node.
According to an aspect of the present application, there is provided a data transmission method, which is applied to an online data storage server, and includes:
establishing a data downloading channel between the online data storage server and the offline data storage server;
acquiring a database file matched with a data import request sent by the online data storage server from the offline data storage server through the data download channel, wherein the database file comprises data contents of a plurality of data blocks, and the data blocks are obtained by dividing data matched with the data import request by the offline data storage server according to topology structure information of each storage node in the online data storage server;
and loading the database file to a preset storage position.
In some embodiments, loading the database file to a preset storage location includes:
generating a blank data storage space matched with the file format of the database file in the online data storage server;
and loading the database file into the generated data storage space.
In some embodiments, loading the database file to a preset storage location includes:
generating a blank data storage space in the online data storage server;
converting the file format of the database file into a file format matched with the data storage format of the generated data storage space;
and loading the database file after converting the file format into the generated data storage space.
In some embodiments, after loading the database file to a preset storage location, the method includes:
and deleting the existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space.
In some embodiments, loading the database file to a preset storage location includes:
determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file;
And loading the database file into the determined data storage space.
In some embodiments, loading the database file to a preset storage location includes:
determining a generated data storage space matched with the database file;
converting the data in the database file into data in a file format matched with the determined generated data storage space;
and loading the data in the database file after format conversion into the determined generated data storage space.
According to another aspect of the present application, there is provided a data transmission method, which is applied to an offline data storage server, and includes:
establishing a data downloading channel between the offline data storage server and the online data storage server;
acquiring topological structure information of each storage node in the online data storage server;
dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information, and adding the data contents of the data blocks into a database file;
and sending the database file to the online data storage server through the data downloading channel.
In some embodiments, the dividing the data set matching the data import request into a plurality of sub data blocks based on the topology information includes:
determining the data bearing attribute and the data bearing capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
and dividing the data set matched with the data import request into a plurality of sub data blocks matched with the storage nodes based on the data bearing attribute and the data bearing capacity of each storage node.
According to an aspect of the present application, there is provided an online data storage server, including:
the channel establishing module is used for establishing a data downloading channel between the online data storage server and the offline data storage server;
a data downloading module, configured to acquire, from the offline data storage server through the data downloading channel established by the channel establishing module, a database file matched with the data import request sent by the online data storage server, where the database file includes data contents of a plurality of data blocks, and the plurality of data blocks are obtained by dividing, by the offline data storage server, data matched with the data import request according to topology structure information of each storage node in the online data storage server;
And the file loading module is used for loading the database file downloaded by the data downloading module to a preset storage position.
In some embodiments, the file loading module is specifically configured to:
generating a blank data storage space matched with the file format of the database file in the online data storage server;
and loading the database file into the generated data storage space.
In some embodiments, the file loading module is specifically configured to:
generating a blank data storage space in the online data storage server;
converting the file format of the database file into a file format matched with the data storage format of the generated data storage space;
and loading the database file after converting the file format into the generated data storage space.
In some embodiments, the online data storage server further comprises:
and the deleting module is used for deleting the existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space.
In some embodiments, the file loading module is further specifically configured to:
Determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file;
and loading the database file into the determined data storage space.
In some embodiments, the file loading module is further specifically configured to:
determining a generated data storage space matched with the database file;
converting the data in the database file into data in a file format matched with the determined generated data storage space;
and loading the data in the database file after format conversion into the determined generated data storage space.
According to one aspect of the present application, there is provided an offline data storage server, comprising:
the channel construction module is used for establishing a data downloading channel between the offline data storage server and the online data storage server;
the acquisition module is used for acquiring the topological structure information of each storage node in the online data storage server;
the data conversion module is used for dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information acquired by the acquisition module, and adding the data contents of the data blocks into a database file;
And the file sending module is used for sending the database file to the online data storage server through the data downloading channel established by the channel establishing module.
In some embodiments, the data conversion module is specifically configured to:
determining the data bearing attribute and the data bearing capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
and dividing the data set matched with the data import request into a plurality of sub data blocks matched with the storage nodes based on the data bearing attribute and the data bearing capacity of each storage node.
According to an aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the data transmission method of the above two aspects.
According to the data transmission method, the online data storage server, the offline data storage server and the data storage system, the offline data storage server can acquire topological structure information of each storage node in the online data storage server; dividing data matched with a data import request of the online data storage server into a plurality of data blocks based on the topological structure information, and adding the data contents of the data blocks into a database file; sending the database file to an online data storage server through a data downloading channel; therefore, the online data storage server can acquire the database file matched with the data import request sent by the online data storage server from the offline data storage server through the data download channel and load the database file to a preset storage position.
Therefore, the data can be converted into the database file, and the database file is directly transmitted, so that the times of data forwarding among all nodes of the data storage system and the times of writing in the storage nodes are effectively reduced, the data importing time is reduced, the data importing efficiency is improved, and the stability of providing query service for the data storage system is well maintained.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
FIG. 1 is a block diagram of an online data storage system;
FIG. 2 is a schematic diagram of a data storage system of the present application;
fig. 3 is a schematic flowchart of a data transmission method according to an embodiment of the present application;
fig. 4 is a schematic flowchart of a data transmission method according to an embodiment of the present application;
FIG. 5 is a schematic diagram of an online data storage server according to an embodiment of the present application;
FIG. 6 is a schematic diagram of an offline data storage server according to an embodiment of the present application;
fig. 7 shows a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it should be understood that the drawings in the present application are for illustrative and descriptive purposes only and are not used to limit the scope of protection of the present application. Additionally, it should be understood that the schematic drawings are not necessarily drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of the present application. It should be understood that the operations of the flow diagrams may be performed out of order, and steps without logical context may be performed in reverse order or simultaneously. One skilled in the art, under the guidance of this application, may add one or more other operations to, or remove one or more operations from, the flowchart.
In addition, the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
To enable those skilled in the art to use the present disclosure, the following embodiments are given in connection with the specific application scenario "importing offline data from an offline data storage server to an online data storage server". It will be apparent to those skilled in the art that the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the application. Although the present application is described primarily with respect to offline data importing from an offline data storage server to an online data storage server, it should be understood that this is merely one exemplary embodiment.
It should be noted that in the embodiments of the present application, the term "comprising" is used to indicate the presence of the features stated hereinafter, but does not exclude the addition of further features.
One aspect of the present application relates to a data storage system, which may establish a data download channel between a plurality of data storage servers by controlling, and in particular, establish a data download channel between an offline data storage server and an online data storage server, and control the offline data storage server to add offline data to a database file, where the online data storage server may directly obtain the database file, so that the offline data in the offline data storage server may be efficiently and quickly imported into the online data storage server.
It is worth noting that, at present, the storage of big data generally adopts a distributed storage system Hadoop to store, and the timeliness of offline data in Hadoop is poor, and the requirement of online business for low-delay data reading cannot be met, so that the offline data needs to be written into the online storage system first, and then the online storage system provides low-delay query for the online business.
Before the application is provided, offline data in an offline data storage system Hadoop is written into the online data storage system, mostly, after the offline data entries are read by the Hadoop, a data request for writing the data is constructed and sent to the online data storage system, the online data storage system can sequentially read the data request and the data entries at each internal node, as the data is sent to each data storage cluster in the online data storage system through one read-write request, and each request is forwarded by several layers, and finally reaches the storage node, the link is long, and meanwhile, a large number of write requests flow into the data storage nodes in the online data storage system, the query delay of the data storage nodes is increased, and the service stability is influenced.
However, the data storage system provided by the present application may establish a data downloading channel between the online data storage server and the offline data storage server, divide the offline data into a plurality of data blocks, perform format conversion on the data in the plurality of data blocks to convert the data into a database file format, and add the data content of the plurality of data blocks to the database file, so that the database file may be directly sent from the offline data storage server to the online data storage server through the established data downloading channel, and the online data storage server loads its own data in the database file to a corresponding storage location, thereby completing importing the data. The data import time and the workload of the data storage system are reduced, and the data import efficiency and the stability effect of the data storage system are improved.
Based on this, one aspect of the present application relates to a data storage system, which includes an online data storage server and an offline data storage server, the online data storage server is connected to the offline data storage server, and the offline data storage server can import stored offline data into the online data storage server, so that the online data storage server provides data query and service query of ground time delay for online business personnel. Specifically, in the data storage system, a data download channel may be established between the online data storage server and the offline data storage server, the offline data storage server performs data structure division on stored offline data to divide the stored offline data into a plurality of data blocks, and may perform format conversion on data in the plurality of data blocks to convert the data into a database file format, so as to add data contents of the plurality of data blocks after format conversion to a database file, the online data storage server may acquire the database file from the offline data storage server through the established data download channel in a downloading manner to acquire data in the plurality of data blocks, and may load the data in the data blocks to corresponding positions, i.e., may complete data import.
The online data storage server and the offline data storage server can both comprise a plurality of databases for data storage, so as to realize distributed storage of data.
In the present application, the terms "online data storage server" and "online data storage system" are used interchangeably to refer to a database or a data system, etc. that can be used to store real-time data or to call, access, etc. the terms "offline data storage server" and "offline data storage system" are also used interchangeably to refer to a database or a data system, such as a distributed storage system Hadoop, etc., that can be used to store offline data.
The data storage system may be a data platform for use in, for example, a financial industry, an internet industry, a medical industry, or an educational industry, and may be applied to, for example, a network appointment travel business in the internet industry.
Referring to fig. 2, fig. 2 is a schematic diagram of a data storage system 200 according to the present application, where the data storage system includes an online data storage server 210 and an offline data storage server 220, the online data storage server 210 may include an application client, a VIP node, a proxy node, and a data storage node, and the offline data storage server 220 may adopt a distributed storage system Hadoop, and may specifically refer to a Hadoop server cluster or a certain server in the cluster.
In some embodiments, online data storage server 210 and offline data storage server 220 may include processors. The processor may process information and/or data related to the service request to perform one or more of the functions described herein. For example, the processor may import and export data based on service requests obtained from clients. In some embodiments, a processor may include one or more processing cores (e.g., a single-core processor (S) or a multi-core processor (S)). Merely by way of example, a Processor may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction Set Processor (ASIP), a Graphics Processing Unit (GPU), a Physical Processing Unit (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), a controller, a microcontroller Unit, a Reduced Instruction Set computer (Reduced Instruction Set Computing, RISC), a microprocessor, or the like, or any combination thereof.
The following describes the data transmission method provided in the embodiment of the present application in detail with reference to the content described in the data storage system and the content described in the online data storage server and the offline data storage server shown in fig. 2.
Referring to fig. 3, fig. 3 is a schematic flow chart of a data transmission method provided in an embodiment of the present application, where the data transmission method is applied to the online data storage server in the data storage system and can be executed by one or more processors in the online data storage server, and the specific execution process is as follows:
s301: and establishing a data downloading channel between the online data storage server and the offline data storage server.
In this step, before the offline data in the offline data storage server needs to be imported into the online data storage server, the online data storage server may be configured to perform data imports on an as-needed basis, such as periodically for updates, or a user needs to perform data query or service query through a client in the online data storage server, the online data storage server may send a data import request to the offline data storage server through the client, to request the offline data in the offline data-storage server to be imported into the online data-storage server, and after the online data storage server receives a feedback instruction which is fed back by the offline data storage server and agrees to carry out data import, the establishment of a data download channel between the online data storage server and the offline data storage server may be controlled first.
The data downloading channel is established between the online data storage server and the offline data storage server, a new data channel may be completely established between the online data storage server and the offline data storage server, and the data downloading channel may also be established by means of a downloading channel for data transmission and the like inherent in the online data storage server and/or the offline data storage server.
S302: and acquiring a database file matched with a data import request sent by an online data storage server from an offline data storage server through the data download channel, wherein the database file comprises data contents of a plurality of data blocks, and the data blocks are obtained by dividing data matched with the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server.
In this step, after the online data storage server establishes the data download channel with the offline data storage server, the online data storage server may obtain, from the offline data storage server, a database file matched with the data import request sent by the online data storage server through the data download channel in a file download manner.
The database file comprises data contents of a plurality of data blocks, and the data blocks are obtained by dividing data matched with the data import request by the offline data storage server according to topology structure information of each storage node in the online data storage server.
For example, in this embodiment of the application, an example that a RocksDB storage engine is used as an online data storage server may be described, and the online data storage server may download, from the offline data storage server, a database file in a format corresponding to the RocksDB through established data download, where a format of data in a plurality of data blocks carried in the database file may be a file in an SST format corresponding to the RocksDB storage engine.
Accordingly, in view of whether the online data storage server or the offline data storage server, distributed storage may be adopted at present for storing large data, and data may be stored on a plurality of different data storage nodes, such as in a plurality of storage spaces, a plurality of storage media, or a plurality of databases, and the plurality of storage nodes are connected to each other in a network, such as by routing, so as to form a complex topology, and a connection formed between the online data storage server and the offline data storage server, such as a connection between a storage node of data storage in the online data storage server and a storage node of data storage in the offline data storage server, also forms a complex topology.
The data blocks are obtained by dividing the data matched with the data import request by the offline data storage server according to the topological structure information of each storage node in the online data storage server. The topology information of each storage node in the online data storage server may include, in addition to topology information between the interiors of the storage nodes in the online data storage server, topology information of connections between the storage nodes in the online data storage server and the storage nodes in the offline data storage server.
S303: and loading the database file to a preset storage position.
In this step, after the online data storage server obtains the database file, the database file may be loaded to a preset storage location. The preset storage location may be the same set data storage location, and all the acquired database files may be loaded to the set data storage location, or may be distributed multiple data storage locations, and the acquired database files may be stored in each data storage location, respectively.
For example, after the online data storage server obtains the database file, the database file may be cached in a storage node, and then directly downloaded to the local by the storage node, that is, the online data storage server may download a plurality of RocksDB-type spaces for data storage, such as a database.
In some embodiments of the present application, S303 comprises:
generating a blank data storage space matched with the file format of the database file in the online data storage server; and loading the database file into the generated data storage space.
In this step, after the online data storage server obtains the database file, the online data storage server may control to generate a blank data storage space in the online data storage server, where the blank data storage space matches with the file format of the database file, that is, a data storage space is newly constructed in the online data storage server, and the format of data storage in the newly constructed data storage space needs to match with the file format of the database file, that is, matches with the format of data in the database file, and then may load the obtained database file into the generated data storage space, so as to perform data storage.
Wherein the generated blank data storage space has data import and export functions.
In this way, full import of data can be achieved, and in the data import process, the database file is directly imported without sending individual application write requests, and the database file can be compressed, directly downloaded to the local, and then loaded into the RocksDB database, so the import speed is several times faster than the conventional means.
In some embodiments of the present application, S303 comprises:
generating a blank data storage space in the online data storage server; converting the file format of the database file into a file format matched with the data storage format of the generated data storage space; and loading the database file after converting the file format into the generated data storage space.
In this step, after the online data storage server obtains the database file, the online data storage server may control generation of a blank data storage space in the online data storage server, that is, a data storage space is newly constructed in the online data storage server, and a format of data stored in the newly constructed data storage space is different from a format of the database file.
In this way, the full import of data can be realized, and in the data import process, the database file is directly imported without sending individual application write requests, and the database file can be compressed, decompressed and format-converted after being cached, and directly downloaded to the local and then loaded into the database, so the import speed is several times faster than the conventional means.
In the above embodiment, after loading the database file to a preset storage location, the method includes:
and deleting the existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space.
In this step, after the online data storage server generates a new data storage space and loads the database file into the generated data storage space, the online data storage server may control to delete an existing data storage space corresponding to the database file in the online data storage server and to store data in the existing data storage space.
The existing data storage space corresponding to the database file may be a data storage space of imported data generated by the online data storage server before data import is performed this time or when data import or data update is performed last time, that is, the existing data storage space and the data storage space generated this time are the same storage space, but data in the existing data storage space is not updated in an old version.
In some embodiments of the present application, S303 comprises:
determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file; and loading the database file into the determined data storage space.
In this step, after the online data storage server obtains the database file, the online data storage server may detect a data storage space generated by itself, to determine a generated data storage space matching a file format of data in the database file from a plurality of data storage spaces, and then load the database file into the determined data storage space for storage.
The generated data storage space matched with the file format of the data in the database file is a data storage space in which partial data in the database file is stored, in addition to the data storage space matched with the stored data format, in a plurality of existing data storage spaces of the online data storage server, that is, the data in the database file needs to be completely updated.
The generated data storage space may be a data storage space generated in a data import process before the data import of this time.
For example, the online data storage server stores data storage spaces of different types of data, such as order data, user information, and the like, and the data import is to update the order data, and the existing data storage space matched with the imported database file is the data storage space for storing the order data, and the data formats of the two data storage spaces are the same.
In this way, incremental import of data can be realized, the incremental import process imports the data into an existing data storage space, namely a database, and although read-write of a hard disk may be caused in the import of a data decompression part in the import process, experiments show that compared with the existing incremental data import, the incremental data import method can effectively reduce the number and the process of application write requests, and can match with the import speed of a control file to further reduce the influence caused by the read-write requests, and the import speed of the incremental data is faster than the import speed of the incremental data by a conventional request sending mode.
In some embodiments of the present application, S303 comprises:
determining a generated data storage space matched with the database file; converting the data in the database file into data in a file format matched with the determined generated data storage space; and loading the data in the database file after format conversion into the determined generated data storage space.
In this step, after the online data storage server obtains the database file, the online data storage server may detect an existing generated data storage space thereof, so as to determine a generated data storage space matching with the database file from a plurality of data storage spaces, that is, the determined generated data storage space is a storage space that needs to be updated with data in the database file, but since a data format of data in the determined generated data storage space is different from a data format of data in the database file, format conversion may be performed on data in the database file, so as to convert the data into the data format of data in the determined generated data storage space, and then the data in the database file after format conversion may be loaded into the determined generated data storage space, and storing.
The generated data storage space may be a data storage space generated in a data import process before the data import of this time.
According to the data transmission method provided by the embodiment of the application, the data downloading channel is established between the off-line data storage server and the on-line data storage server, and the database file of the imported data is downloaded through the data downloading channel, so that the direct import of the database file is realized, the read-write request of the data corresponding to each node is effectively reduced, the data import time and the workload of the data storage system are reduced, and the data import efficiency and the stability effect of the data storage system are improved.
Referring to fig. 4, fig. 4 is a schematic flow chart of a data transmission method provided in an embodiment of the present application, and based on the same inventive concept, an embodiment of the present application further provides a data transmission method, where the data transmission method is applied to the offline data storage server in the data storage system and can be executed by one or more processors in the offline data storage server, and the specific execution process is as follows:
s401: and establishing a data downloading channel between the offline data storage server and the online data storage server.
In this step, before offline data in the offline data storage server needs to be imported into the online data storage server, the offline data storage server may send a data import request to the online data storage server, and then may control a data download channel to be established between the online data storage server and the offline data storage server after sending a feedback instruction agreeing to data import to the online data storage server.
The data downloading channel is established between the online data storage server and the offline data storage server, a new data channel may be completely established between the online data storage server and the offline data storage server, and the data downloading channel may also be established by means of a downloading channel for data transmission and the like inherent in the online data storage server and/or the offline data storage server.
S402: and acquiring the topological structure information of each storage node in the online data storage server.
In this step, after the data download channel is established, the offline data storage server may obtain topology structure information of each storage node in the online data storage server from the online data storage server.
The topology structure information of each storage node in the online data storage server may include, in addition to the topology structure information between the interiors of the storage nodes in the online data storage server, topology structure information of connections between the storage nodes in the online data storage server and the storage nodes in the offline data storage server.
S403: and dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information, and adding the data contents of the data blocks into a database file.
In this step, after the offline data storage server obtains the topology information, the offline data storage server may divide the data matching the data import request of the online data storage server into a plurality of data blocks according to each topology condition indicated in the topology information, and then add the data contents of the plurality of data blocks to a database file to form a database file that can be transmitted.
Specifically, based on the topology information, the data matching the data import request of the online data storage server is divided into a plurality of data blocks, the data carrying attribute and the data carrying capacity of each storage node may be determined according to the attribute information of each storage node indicated in the topology information, such as the connection condition between each storage node and the connected storage node, and then the data set matching the data import request may be divided into a plurality of sub data blocks matching each storage node based on the data carrying attribute and the data carrying capacity of each storage node.
For example, taking the case that the offline data in the offline data storage server is stored by Hadoop as an example, the offline data may be obtained by dividing data to be imported by combining attribute information of each storage node, such as routing information of each storage node, on the Hadoop in a MapReduce calculation manner, and then converting the divided imported data into an SST file of RocksDB and adding the SST file to a database file.
S404: and sending the database file to the online data storage server through the data downloading channel.
In this step, after the data processing is completed, the offline data storage server may send the database file to the online data storage server through the data download channel.
According to the data transmission method provided by the embodiment of the application, the data downloading channel is established between the off-line data storage server and the on-line data storage server, and the database file of the imported data is downloaded through the data downloading channel, so that the direct import of the database file is realized, the read-write request of the data corresponding to each node is effectively reduced, the data import time and the workload of the data storage system are reduced, and the data import efficiency and the stability effect of the data storage system are improved.
Based on the same inventive concept, an online data storage server and an offline data storage server corresponding to the data transmission method are also provided in the embodiments of the present application, and as the principles of solving the problems of the online data storage server and the offline data storage server in the embodiments of the present application are similar to those of the data transmission method in the embodiments of the present application, the implementation of the online data storage server and the offline data storage server can refer to the implementation of the method, and repeated parts are not described again.
Referring to fig. 5, fig. 5 is a schematic diagram of an online data storage server according to an embodiment of the present application, where the online data storage server 500 includes:
a channel establishing module 510, configured to establish a data downloading channel between the online data storage server and the offline data storage server;
a data downloading module 520, configured to obtain, through the data downloading channel established by the channel establishing module 510, a database file matched with the data import request sent by the online data storage server from the offline data storage server, where the database file includes data contents of a plurality of data blocks, and the plurality of data blocks are obtained by dividing, by the offline data storage server, data matched with the data import request according to topology structure information of each storage node in the online data storage server;
A file loading module 530, configured to load the database file downloaded by the data downloading module 520 to a preset storage location.
In some embodiments of the present application, the file loading module 530 is specifically configured to:
generating a blank data storage space matched with the file format of the database file in the online data storage server;
and loading the database file into the generated data storage space.
In some embodiments of the present application, the file loading module 530 is specifically configured to:
generating a blank data storage space in the online data storage server;
converting the file format of the database file into a file format matched with the data storage format of the generated data storage space;
and loading the database file after converting the file format into the generated data storage space.
In the above embodiment, the online data storage server 500 further includes:
a deleting module 540, configured to delete an existing data storage space in the online data storage server 500 corresponding to the database file and data in the existing data storage space.
In some embodiments of the present application, the file loading module 530 is further specifically configured to:
Determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file;
and loading the database file into the determined data storage space.
In some embodiments of the present application, the file loading module 530 is further specifically configured to:
determining a generated data storage space matched with the database file;
converting the data in the database file into data in a file format matched with the determined generated data storage space;
and loading the data in the database file after format conversion into the determined generated data storage space.
The description of the processing flow of the modules and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.
According to the online data storage server provided by the embodiment of the application, the data downloading channel is established between the offline data storage server and the online data storage server, and the database file of the imported data is downloaded through the data downloading channel, so that the database file is directly imported, the read-write request of the data corresponding to each node is effectively reduced, the data import time and the workload of a data storage system are reduced, and the data import efficiency and the stability effect of the data storage system are improved.
Referring to fig. 6, fig. 6 is a schematic diagram of an offline data storage server according to an embodiment of the present application, where the offline data storage server 600 includes:
a channel construction module 610, configured to establish a data download channel between the offline data storage server and the online data storage server;
an obtaining module 620, configured to obtain topology structure information of each storage node in the online data storage server;
a data conversion module 630, configured to divide data matched with the data import request of the online data storage server into a plurality of data blocks based on the topology information acquired by the acquisition module 620, and add data contents of the data blocks to a database file;
the file sending module 640 is configured to send the database file to the online data storage server through the data download channel established by the channel establishing module 610.
In some embodiments of the present application, the data conversion module 630 is specifically configured to:
determining the data bearing attribute and the data bearing capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
And dividing the data set matched with the data import request into a plurality of sub data blocks matched with the storage nodes based on the data bearing attribute and the data bearing capacity of each storage node.
According to the offline data storage server provided by the embodiment of the application, the data downloading channel is established between the offline data storage server and the online data storage server, and the database file of the imported data is downloaded through the data downloading channel, so that the direct import of the database file is realized, the read-write request of the data corresponding to each node is effectively reduced, the data import time and the workload of a data storage system are reduced, and the data import efficiency and the stability effect of the data storage system are improved.
The description of the processing flow of the modules and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.
An embodiment of the present application further provides a computer device 700, as shown in fig. 7, which is a schematic structural diagram of the computer device 700 provided in the embodiment of the present application, and includes: a processor 710, a memory 720, and a bus 730. The memory 720 stores machine-readable instructions executable by the processor 710, the processor 710 and the memory 720 communicating via the bus 730 when the computer device 700 is operating, the machine-readable instructions when executed by the processor 710 further operable to perform one or more of the following:
Establishing a data downloading channel between the online data storage server and the offline data storage server; acquiring a database file matched with a data import request sent by the online data storage server from the offline data storage server through the data download channel, wherein the database file comprises data contents of a plurality of data blocks, and the data blocks are obtained by dividing data matched with the data import request by the offline data storage server according to topology structure information of each storage node in the online data storage server; and loading the database file to a preset storage position.
The machine-readable instructions, when executed by the processor 710, may perform one or more of the following:
establishing a data downloading channel between the offline data storage server and the online data storage server; acquiring topological structure information of each storage node in the online data storage server; dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information, and adding the data contents of the data blocks into a database file; and sending the database file to the online data storage server through the data downloading channel.
Correspondingly, the embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps of the data transmission method are executed.
Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, and the like, and when a computer program on the storage medium is run, the data transmission method can be executed, so that the problems of high data import delay and poor server stability are solved, and the effects of reducing the data import delay, reducing the data import time, and improving the stability of the server providing the query service are achieved.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to corresponding processes in the method embodiments, and are not described in detail in this application. In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and there may be other divisions in actual implementation, and for example, a plurality of modules or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or modules through some communication interfaces, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (19)
1. A data transmission method is applied to an online data storage server, and is characterized by comprising the following steps:
establishing a data downloading channel between the online data storage server and the offline data storage server;
acquiring a database file matched with a data import request sent by the online data storage server from the offline data storage server through the data download channel, wherein the database file comprises data contents of a plurality of data blocks, and the data blocks are obtained by dividing data matched with the data import request by the offline data storage server according to topology structure information of each storage node in the online data storage server;
and loading the database file to a preset storage position.
2. The data transmission method according to claim 1, wherein loading the database file to a preset storage location comprises:
generating a blank data storage space matched with the file format of the database file in the online data storage server;
and loading the database file into the generated data storage space.
3. The data transmission method according to claim 1, wherein loading the database file to a preset storage location comprises:
generating a blank data storage space in the online data storage server;
converting the file format of the database file into a file format matched with the data storage format of the generated data storage space;
and loading the database file after converting the file format into the generated data storage space.
4. A data transmission method according to any one of claims 2 and 3, wherein after loading the database file to a preset storage location, the method comprises:
and deleting the existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space.
5. The data transmission method according to claim 1, wherein loading the database file to a preset storage location comprises:
determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file;
and loading the database file into the determined data storage space.
6. The data transmission method according to claim 1, wherein loading the database file to a preset storage location comprises:
determining a generated data storage space matched with the database file;
converting the data in the database file into data in a file format matched with the determined generated data storage space;
and loading the data in the database file after format conversion into the determined generated data storage space.
7. A data transmission method is applied to an offline data storage server, and is characterized by comprising the following steps:
establishing a data downloading channel between the offline data storage server and the online data storage server;
acquiring topological structure information of each storage node in the online data storage server;
Dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information, and adding the data contents of the data blocks into a database file;
and sending the database file to the online data storage server through the data downloading channel.
8. The data transmission method according to claim 7, wherein the dividing the data set matching the data import request into a plurality of sub data blocks based on the topology information comprises:
determining the data bearing attribute and the data bearing capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
and dividing the data set matched with the data import request into a plurality of sub data blocks matched with the storage nodes based on the data bearing attribute and the data bearing capacity of each storage node.
9. An online data storage server, the online data storage server comprising:
the channel establishing module is used for establishing a data downloading channel between the online data storage server and the offline data storage server;
A data downloading module, configured to acquire, from the offline data storage server through the data downloading channel established by the channel establishing module, a database file matched with the data import request sent by the online data storage server, where the database file includes data contents of a plurality of data blocks, and the plurality of data blocks are obtained by dividing, by the offline data storage server, data matched with the data import request according to topology structure information of each storage node in the online data storage server;
and the file loading module is used for loading the database file downloaded by the data downloading module to a preset storage position.
10. The online data storage server of claim 9, wherein the file loading module is specifically configured to:
generating a blank data storage space matched with the file format of the database file in the online data storage server;
and loading the database file into the generated data storage space.
11. The online data storage server of claim 9, wherein the file loading module is specifically configured to:
Generating a blank data storage space in the online data storage server;
converting the file format of the database file into a file format matched with the data storage format of the generated data storage space;
and loading the database file after converting the file format into the generated data storage space.
12. The online data storage server of any of claims 10 and 11, further comprising:
and the deleting module is used for deleting the existing data storage space corresponding to the database file in the online data storage server and the data in the existing data storage space.
13. The online data storage server of claim 9, wherein the file loading module is further configured to:
determining a generated data storage space in the online data storage server, which is matched with the file format of the data in the database file;
and loading the database file into the determined data storage space.
14. The online data storage server of claim 9, wherein the file loading module is further configured to:
Determining a generated data storage space matched with the database file;
converting the data in the database file into data in a file format matched with the determined generated data storage space;
and loading the data in the database file after format conversion into the determined generated data storage space.
15. An offline data storage server, the offline data storage server comprising:
the channel construction module is used for establishing a data downloading channel between the offline data storage server and the online data storage server;
the acquisition module is used for acquiring the topological structure information of each storage node in the online data storage server;
the data conversion module is used for dividing data matched with the data import request of the online data storage server into a plurality of data blocks based on the topological structure information acquired by the acquisition module, and adding the data contents of the data blocks into a database file;
and the file sending module is used for sending the database file to the online data storage server through the data downloading channel established by the channel establishing module.
16. The offline data storage server of claim 15, wherein the data conversion module is specifically configured to:
determining the data bearing attribute and the data bearing capacity of each storage node based on the attribute information of each storage node represented by the topological structure information;
and dividing the data set matched with the data import request into a plurality of sub data blocks matched with the storage nodes based on the data bearing attribute and the data bearing capacity of each storage node.
17. A data storage system comprising an online data storage server according to any of claims 9 to 14 and an offline data storage server according to any of claims 15 to 16.
18. An electronic device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating via the bus when the electronic device is operating, the processor executing the machine-readable instructions to perform the steps of the data transmission method of any one of claims 1 to 6 or to perform the steps of the data transmission method of any one of claims 7 to 8.
19. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, performs the steps of the data transmission method according to one of the claims 1 to 6 or performs the steps of the data transmission method according to one of the claims 7 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910335421.2A CN111857534A (en) | 2019-04-24 | 2019-04-24 | Data transmission method, data storage server and data storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910335421.2A CN111857534A (en) | 2019-04-24 | 2019-04-24 | Data transmission method, data storage server and data storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111857534A true CN111857534A (en) | 2020-10-30 |
Family
ID=72952266
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910335421.2A Pending CN111857534A (en) | 2019-04-24 | 2019-04-24 | Data transmission method, data storage server and data storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111857534A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116708420A (en) * | 2023-07-28 | 2023-09-05 | 联想凌拓科技有限公司 | Method, device, equipment and medium for data transmission |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020073076A1 (en) * | 2000-12-11 | 2002-06-13 | Yongjie Xu | System and method for enabling off-line database functionality |
| US20040003003A1 (en) * | 2002-06-26 | 2004-01-01 | Microsoft Corporation | Data publishing systems and methods |
| US20050223277A1 (en) * | 2004-03-23 | 2005-10-06 | Eacceleration Corporation | Online storage system |
| US20070168316A1 (en) * | 2006-01-13 | 2007-07-19 | Microsoft Corporation | Publication activation service |
| CN102355426A (en) * | 2011-06-30 | 2012-02-15 | 北京神州泰岳软件股份有限公司 | Method for transmitting off-line file and system |
| CN102917020A (en) * | 2011-09-24 | 2013-02-06 | 国网电力科学研究院 | Mobile terminal based on data packet and method for synchronizing business system data |
| CN103049224A (en) * | 2012-12-28 | 2013-04-17 | 华为技术有限公司 | Method, device and system for importing data into physical tape |
| US8732403B1 (en) * | 2012-03-14 | 2014-05-20 | Netapp, Inc. | Deduplication of data blocks on storage devices |
| US20140164487A1 (en) * | 2012-12-12 | 2014-06-12 | Hon Hai Precision Industry Co., Ltd. | File saving system and method |
| US20150261800A1 (en) * | 2014-03-12 | 2015-09-17 | Dell Products L.P. | Method for Storing and Accessing Data into an Indexed Key/Value Pair for Offline Access |
| CN104933333A (en) * | 2014-07-25 | 2015-09-23 | 湖北华中电力科技开发有限责任公司 | Special office mobile storage medium management system |
| CN106933207A (en) * | 2015-10-09 | 2017-07-07 | 费希尔-罗斯蒙特***公司 | For the data analysis service of distributed industrial performance monitoring |
| CN109067618A (en) * | 2018-09-06 | 2018-12-21 | 北京奥技异科技发展有限公司 | Distributed real-time data IOT acquisition system and method |
| CN109240624A (en) * | 2018-09-29 | 2019-01-18 | 郑州云海信息技术有限公司 | A kind of data processing method and device |
| CN109271456A (en) * | 2018-11-16 | 2019-01-25 | 中国银行股份有限公司 | Host data library file deriving method and device |
| US20190102113A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Database with nvdimm as persistent storage |
| US10254996B1 (en) * | 2018-08-10 | 2019-04-09 | Cohesity, Inc. | Fast migration of metadata |
-
2019
- 2019-04-24 CN CN201910335421.2A patent/CN111857534A/en active Pending
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020073076A1 (en) * | 2000-12-11 | 2002-06-13 | Yongjie Xu | System and method for enabling off-line database functionality |
| US20040003003A1 (en) * | 2002-06-26 | 2004-01-01 | Microsoft Corporation | Data publishing systems and methods |
| US20050223277A1 (en) * | 2004-03-23 | 2005-10-06 | Eacceleration Corporation | Online storage system |
| US20070168316A1 (en) * | 2006-01-13 | 2007-07-19 | Microsoft Corporation | Publication activation service |
| CN102355426A (en) * | 2011-06-30 | 2012-02-15 | 北京神州泰岳软件股份有限公司 | Method for transmitting off-line file and system |
| CN102917020A (en) * | 2011-09-24 | 2013-02-06 | 国网电力科学研究院 | Mobile terminal based on data packet and method for synchronizing business system data |
| US8732403B1 (en) * | 2012-03-14 | 2014-05-20 | Netapp, Inc. | Deduplication of data blocks on storage devices |
| US20140164487A1 (en) * | 2012-12-12 | 2014-06-12 | Hon Hai Precision Industry Co., Ltd. | File saving system and method |
| CN103049224A (en) * | 2012-12-28 | 2013-04-17 | 华为技术有限公司 | Method, device and system for importing data into physical tape |
| US20150261800A1 (en) * | 2014-03-12 | 2015-09-17 | Dell Products L.P. | Method for Storing and Accessing Data into an Indexed Key/Value Pair for Offline Access |
| CN104933333A (en) * | 2014-07-25 | 2015-09-23 | 湖北华中电力科技开发有限责任公司 | Special office mobile storage medium management system |
| CN106933207A (en) * | 2015-10-09 | 2017-07-07 | 费希尔-罗斯蒙特***公司 | For the data analysis service of distributed industrial performance monitoring |
| US20190102113A1 (en) * | 2017-09-29 | 2019-04-04 | Oracle International Corporation | Database with nvdimm as persistent storage |
| US10254996B1 (en) * | 2018-08-10 | 2019-04-09 | Cohesity, Inc. | Fast migration of metadata |
| CN109067618A (en) * | 2018-09-06 | 2018-12-21 | 北京奥技异科技发展有限公司 | Distributed real-time data IOT acquisition system and method |
| CN109240624A (en) * | 2018-09-29 | 2019-01-18 | 郑州云海信息技术有限公司 | A kind of data processing method and device |
| CN109271456A (en) * | 2018-11-16 | 2019-01-25 | 中国银行股份有限公司 | Host data library file deriving method and device |
Non-Patent Citations (3)
| Title |
|---|
| ROSSMANN J.等: "GML-Based Data Management and Semantic World Modelling for a 4D Forest Simulation and Information System", 《 INTERNATIONAL JOURNAL OF 3-D INFORMATION MODELING》, vol. 3, no. 3, 31 December 2014 (2014-12-31), pages 50 - 67 * |
| 徐文华等: "基于数据流集成回归的短时交通流预测", 《交通信息与安全》, vol. 32, no. 4, 20 December 2014 (2014-12-20), pages 14 - 19 * |
| 韩鲲等: "基于数据中心的数据仓库结构设计", 《信息安全与通信保密》, no. 12, 10 December 2007 (2007-12-10), pages 78 - 82 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116708420A (en) * | 2023-07-28 | 2023-09-05 | 联想凌拓科技有限公司 | Method, device, equipment and medium for data transmission |
| CN116708420B (en) * | 2023-07-28 | 2023-11-03 | 联想凌拓科技有限公司 | Method, device, equipment and medium for data transmission |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107391629B (en) | Method, system, server and computer storage medium for data migration between clusters | |
| US9485300B2 (en) | Publish-subscribe platform for cloud file distribution | |
| CN110599169B (en) | Data processing method, device, terminal and medium | |
| CN102662639A (en) | Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method | |
| CN110581890B (en) | Routing method and device for service request | |
| CN112583931B (en) | Message processing method, message middleware, electronic device, and storage medium | |
| CN110609872A (en) | Method and apparatus for synchronizing node data | |
| CN109508326B (en) | Method, device and system for processing data | |
| CN105681477B (en) | A kind of data access method and a kind of server | |
| CN113209633A (en) | Mirror image processing method and device, electronic equipment and readable storage medium | |
| CN110677461A (en) | Graph calculation method based on key value pair storage | |
| CN112199427A (en) | Data processing method and system | |
| CN113687964A (en) | Data processing method, data processing apparatus, electronic device, storage medium, and program product | |
| CN111857534A (en) | Data transmission method, data storage server and data storage system | |
| CN103294799A (en) | Method and system for parallel batch importing of data into read-only query system | |
| CN101673217A (en) | Method for realizing remote program call and system thereof | |
| CN109302449B (en) | Data writing method, data reading device and server | |
| CN111597403B (en) | Method and device for constructing graph index, electronic equipment and storage medium | |
| CN112804366B (en) | Method and device for resolving domain name | |
| CN113676525A (en) | Network collaborative manufacturing-oriented industrial internet public service platform | |
| WO2013059887A1 (en) | Data interchange system | |
| CN113326038B (en) | Method, apparatus, device, storage medium and program product for providing service | |
| CN113934767A (en) | Data processing method and device, computer equipment and storage medium | |
| CN114579311A (en) | Method, apparatus, device and storage medium for executing distributed computing task | |
| CN108809687B (en) | System and method for providing cross-network event attribution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |