CN110764700A - Data storage method and device, computer equipment and storage medium - Google Patents
Data storage method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110764700A CN110764700A CN201910969056.0A CN201910969056A CN110764700A CN 110764700 A CN110764700 A CN 110764700A CN 201910969056 A CN201910969056 A CN 201910969056A CN 110764700 A CN110764700 A CN 110764700A
- Authority
- CN
- China
- Prior art keywords
- data
- storage
- target
- route
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data storage method, a data storage device, computer equipment and a storage medium. The method comprises the following steps: intercepting an operation request for target data sent by a first terminal; determining attribute characteristics corresponding to the operation request through a preset general data operation interface; inquiring the storage route of the target data according to the attribute characteristics; calling a processing component adaptive to the storage route to send the operation request to a server pointed by the storage route, so that the server stores the target data according to the storage route; and sending the storage address of the target data returned by the server to the first terminal. By adopting the method, the reconstruction cost of the service client can be reduced and the data storage efficiency can be improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data storage method and apparatus, a computer device, and a storage medium.
Background
Distributed data storage is to use the disk space on each machine in the enterprise through the network, and form a virtual storage device by using the distributed storage resources, and the data is stored in each corner of the enterprise in a distributed manner. And the user can call different distributed storage services through different service clients to store and read data. Compared with single-point storage, distributed data storage can well improve data security and data access efficiency. However, in the conventional distributed data storage method, a single business client can only invoke a single distributed storage service. If a new distributed storage service is expected to be added, the service client needs to be reconstructed again, so that the data storage cost is improved, and the data storage efficiency is reduced.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data storage method, an apparatus, a computer device, and a storage medium, which can reduce the modification cost of a service client and improve the data storage efficiency.
A method of data storage, the method comprising: intercepting an operation request for target data sent by a first terminal; determining attribute characteristics corresponding to the operation request through a preset general data operation interface; inquiring the storage route of the target data according to the attribute characteristics; calling a processing component adaptive to the storage route to send the operation request to a server pointed by the storage route, so that the server stores the target data according to the storage route; and sending the storage address of the target data returned by the server to the first terminal.
In one embodiment, before querying the storage route of the target data according to the attribute characteristics, the method further includes: acquiring service deployment information of a plurality of servers based on a routing configuration request sent by a second terminal; sending a routing configuration page generated according to the service deployment information to the second terminal; the routing configuration page shows a plurality of server identifications and a plurality of service types; and storing the incidence relation between each service type and the server identifier returned by the second terminal based on the route configuration page as a storage route.
In one embodiment, the storage reason includes a base route and a dynamic route; the basic route records a plurality of service types and server identifications respectively associated with each service type; the dynamic route records different attribute feature combinations of each service type and server identifications associated with each attribute feature combination; the attribute feature combination comprises at least one of target data identification and data operation type.
In one embodiment, the method further comprises: when an operation request of a preset type is intercepted, inquiring a storage route corresponding to the operation request; and asynchronously copying the target data pointed by the operation request from the current server to other servers with the same storage type according to the storage route.
In one embodiment, the target data before replication is recorded as source data; the method further comprises the following steps: in the data copying process, extracting data identifications in source data and data identifications in target data copied to other servers; carrying out hash operation on the data identification to obtain a hash result; grouping the hash results according to the number of preset target threads; determining a target thread corresponding to each group; and calling a plurality of target threads to perform concurrent comparison on the source data and the target data corresponding to the hash result in the corresponding group.
In one embodiment, the grouping the hash results according to the preset number of target threads includes: determining the thread number of each target thread; carrying out division operation on the hash result and the number of target threads to obtain a remainder corresponding to the hash result; grouping source data and target data having the same remainder; the determining the target thread corresponding to each packet comprises: and distributing each group of source data and target data to be compared to a target thread with the corresponding remainder as a thread number.
A data storage device, the device comprising: the operation request module is used for intercepting an operation request for target data sent by a first terminal; the request processing module is used for determining attribute characteristics corresponding to the operation request through a preset general data operation interface; inquiring the storage route of the target data according to the attribute characteristics; the data storage module is used for calling a processing component adaptive to the storage route to send the operation request to a server pointed by the storage route, so that the server stores the target data according to the storage route; and sending the storage address of the target data returned by the server to the first terminal.
In one embodiment, the apparatus includes a data replication module, configured to query, when an operation request of a preset type is intercepted, a storage route corresponding to the operation request; and asynchronously copying the target data pointed by the operation request from the current server to other servers of the same storage type according to the storage route.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the data storage method provided in any one of the embodiments of the present application when executing the computer program.
A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the data storage method as provided in any one of the embodiments of the application.
According to the data storage method, the data storage device, the computer equipment and the storage medium, the attribute characteristics corresponding to the operation request of the first terminal for the target data can be determined through the preset general data operation interface; according to the attribute characteristics, the storage route of the target data can be inquired; the operation request can be subjected to data conversion based on the processing component adaptive to the storage route; the operation request obtained by conversion is sent to the server pointed by the storage route, so that the server can perform storage operation on the target data according to the storage route; and sending the operation result of the target data returned by the server to the first terminal, so that the data can be efficiently stored. By providing a uniform universal data operation interface and a uniform processing component, each service client accesses different distributed storage services in a uniform method and shields specific distributed storage service access details, so that the modification cost of the service client can be reduced, and the data storage efficiency is improved.
Drawings
FIG. 1 is a diagram illustrating an exemplary implementation of a data storage method;
FIG. 2 is a schematic flow chart diagram illustrating a data storage method according to one embodiment;
FIG. 3 is a block diagram of a data storage system in accordance with one embodiment;
FIG. 4 is a block diagram of a data storage device in one embodiment;
FIG. 5 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The data storage method provided by the application can be applied to the application environment shown in fig. 1. The data storage method is suitable for a data storage system. The data storage system includes a first terminal 102, a second terminal 104, a service agent 106, and a server 108. Wherein the first terminal 102 communicates with the service agent 106 via a network. The second terminal 104 communicates with the service agent 106 over a network. The service agent 106 communicates with each server 108 over a network. The first terminal 102 and the second terminal 104 may be, but are not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, respectively. The service proxy 106 may be a proxy server. The service broker 106 may be implemented as a stand-alone server or as a server cluster of multiple servers. Server 108 is a server cluster of multiple servers. The server 108 provides a distributed storage service for the service client, and organizes the data structure of the file on the disk or partition of the first terminal 102. Each server 108 may support a storage type, such as NAS, TFS (Taobao File System), GlusterFS, Ceph, HDFS, and the like. Multiple servers 104 may be deployed in the same room.
When data storage is needed, the user may send an operation request for the target data to the service agent 106 through the first terminal 102. The service agent 106 determines the attribute characteristics corresponding to the operation request through a preset general data operation interface. The developer configures attribute features corresponding to a plurality of operation requests and a storage route corresponding to each attribute feature in the service agent 106 in advance through the second terminal 104. The service agent 106 queries the storage route corresponding to the attribute feature, invokes a processing component adapted to the storage route to perform data conversion on the operation request, and sends the operation request obtained through conversion to the server 108 to which the storage route points. The server 108 performs a storage operation on the target data according to the storage route, and returns an operation result on the target data to the first terminal 102 through the service agent 106. In the data storage process, the processing components of the access logics of different distributed storage services are encapsulated by providing a uniform universal data operation interface, so that each service client can access different distributed storage services in a uniform method and shield specific distributed storage service access details, the modification cost of the service client can be reduced, and the data storage efficiency is improved.
In one embodiment, as shown in fig. 2, a data storage method is provided, which is described by taking the application of the method to the service agent in fig. 1 as an example, and includes the following steps:
Referring to FIG. 3, FIG. 3 is a block diagram illustrating an architecture of a data storage system according to an embodiment. As shown in fig. 3, at least one service client is running on the first terminal 102. Based on the service client, the service functions of various service types can be realized. The service clients running on different first terminals 102 may be the same or different. For example, the service client 1 may be a loan Web application for implementing service types such as loan application, loan audit, contract signing, and the like. The service client 2 may be an insurance APP (application) for implementing service types such as insurance application, underwriting, receipt and claim settlement.
When a certain type of service needs to be performed, a user may initiate an operation request for target data through a service client on the first terminal 102 to implement a certain service function. The target data may be described in a file. The business client integrates a component for intercepting operation requests and sending the intercepted operation requests to the service agent 106, and the component can be a Jar file package implemented based on Java language.
And step 204, determining attribute characteristics corresponding to the operation request through a preset general data operation interface.
The service agent 106 integrates a generic data manipulation interface. The generic data operation interface is used for receiving an operation request sent by the first terminal 102. The data operation interface can be a script file written based on semantic methods such as read, write, ls, rm, isExist and the like which are commonly used in file operation. The decentralized business clients may send operation requests of different operation types based on different business logic. The general file operation interface encapsulates general file operation logics corresponding to each operation type, so that the operation request can be subjected to standardized processing, and attribute characteristics corresponding to the operation types can be obtained. The attribute characteristics comprise at least one item of service type, operation request type, target data identification or data operation type and the like corresponding to the operation request.
By calling a standard general data operation interface, the business client can call various types of distributed storage services, namely remote service calling, only by integrating the Jar package, and the modification cost for upgrading and replacing the storage types of the business client is reduced.
And step 206, inquiring the storage route of the target data according to the attribute characteristics.
In one embodiment, before querying the storage route corresponding to the attribute feature, the method further includes: acquiring service deployment information of a plurality of servers based on a routing configuration request sent by a second terminal; sending a routing configuration page generated according to the service deployment information to the second terminal; the route configuration page shows a plurality of server identifications and a plurality of service types; and storing the incidence relation between each service type and the server identifier returned by the second terminal based on the route configuration page as a storage route.
And the developer performs service type isolation configuration on each server in advance through the second terminal. Specifically, attribute information of each server is predefined, such as server identification, information of a machine room to which the server belongs, running state information and the like. The server identifier may be information such as an IP address that can uniquely identify a server. Server 108 is a server cluster comprised of a plurality of servers, including location nodes and a plurality of storage nodes. The operating state information includes at least remaining storage space information. And the server identification of each server is associated with the corresponding service type in advance. One service type may be performed by multiple servers, each of which may also implement multiple service types. Different service types are distinguished, different servers are defined for each service type, storage isolation of different levels and types of services is achieved, and overall safety and expansibility can be improved.
In one embodiment, the storage reason includes a base route and a dynamic route; the basic route records a plurality of service types and server identifications respectively associated with each service type; the dynamic routing records different attribute feature combinations of each service type and server identifications associated with each attribute feature combination; the attribute feature combination includes at least one of a target data identification or a data operation type.
The developer corresponding to the service client may pre-configure the attribute features of the multiple operation requests and the storage route corresponding to each attribute feature through the second terminal 106, and store the correspondence between the attribute features and the storage routes to the service agent 106. The service agent stores the configuration information into a routing table. The routing table is used as a basis for the service agent to carry out shunt forwarding on the operation request, and records the corresponding relation between various attribute feature combinations and the stored route.
The stored routes include base routes and dynamic routes. The basic route comprises the corresponding relation between the service type and the server identification. The dynamic route records the corresponding relation between the attribute feature combination and the server identification. The attribute feature combination may include a target data identifier or a data operation type, and may further include an identifier of a machine room to which the attribute feature combination belongs, which is not limited herein. The target data identifier is identification information capable of uniquely identifying data specifically desired to be operated by the user, and specifically may be an identifier of a file in which the target data is recorded, or the like. The data operation types include read operations, write operations, save operations, and the like. From the underlying route it can be determined to which servers (denoted target storage servers) the operation request is sent. The target storage server to which the operation request is sent can be further accurately determined according to the dynamic routing.
In one embodiment, the storage route may further include a storage path composed of multiple levels of file directories, such as "root directory \ first level subdirectory \ second level subdirectory", without limitation.
The storage route is customized through a script language, a storage path, a machine room, a storage type and the like can be flexibly defined, and hot effect is supported. The data read-write separation and the fragment routing are really realized through the dynamic routing function, and the overall performance, the expansibility and the reliability are improved.
And step 208, invoking the processing component adaptive to the storage route to send the operation request to the server pointed by the storage route, so that the server stores the target data according to the storage route.
A developer defines a storage type corresponding to each server in advance, creates a processing component (Processor) of each storage type through a factory model, and integrally deploys the processing component in the service agent. Different processing components encapsulate implementation logic for different storage type servers to provide distributed storage services to corresponding business clients. In other words, different processing components provide different distributed storage services. The processing component is used for performing data conversion on the operation request. The data conversion refers to converting the request type of the operation request into a request type which can be responded by the storage route to the server. For example, a traditional service client a can only send an operation request to a server corresponding to the storage type a, and the service agent in this embodiment enables the service client a to send an operation request to servers of multiple storage types based on a generic data operation interface and a preset routing table. And when the operation request corresponding to the storage route passes through the server of the storage type B, converting the operation request sent by the service client A into the operation request corresponding to the server of the storage type B by using the processing component corresponding to the storage route.
The service agent also records the storage type corresponding to each server and the processing component identification applicable to each storage type in the routing table. The service agent deploys different processing components aiming at different storage types, so that the server can respond to operation requests from different service clients without any modification, the service clients can be transparently upgraded and replaced and migrated under the condition of realizing the specific physical realization of shielding data operation of the service clients, and the modification cost of the existing distributed storage service is reduced.
And the service agent sends the converted operation request to a server pointed by the storage route. And if the operation type of the operation request is a new operation, the server stores the target data and returns the data identifier to the first terminal through the service agent. And if the operation type of the operation request is read operation, the server returns the target number corresponding to the data identifier to the first terminal through the service agent. And if the operation type of the operation request is write operation, the server stores the written target data and returns a prompt of successful writing to the first terminal through the service agent.
And step 210, sending the storage address of the target data returned by the server to the first terminal.
And the universal data operation interface feeds the target data or the data identification back to a corresponding service client in the first terminal by adopting a communication protocol such as a universal Http protocol. It is easy to understand that, in order to improve the data transmission efficiency, the target data may also be compressed by the generic data operation interface through the gzip standard and then fed back to the service client. In another embodiment, since the target data needs to be forwarded by the service agent, a plurality of service agents may be deployed in a distributed manner in order to improve data security and transmission efficiency.
According to the data storage method, the attribute characteristics corresponding to the operation request of the first terminal for the target data can be determined through the preset general data operation interface; according to the attribute characteristics, the storage route of the target data can be inquired; the operation request can be subjected to data conversion based on the processing component adaptive to the storage route; the operation request obtained by conversion is sent to the server pointed by the storage route, so that the server can perform storage operation on the target data according to the storage route; and sending the operation result of the target data returned by the server to the first terminal, so that the data can be efficiently stored. By providing a uniform universal data operation interface and a uniform processing component, each service client accesses different distributed storage services in a uniform method and shields specific distributed storage service access details, so that the modification cost of the service client can be reduced, and the data storage efficiency is improved.
In one embodiment, the data storage method further includes: when an operation request of a preset type is intercepted, inquiring a storage route corresponding to the operation request; and asynchronously copying the target data pointed by the operation request from the current server to other servers of the same storage type according to the storage route.
The data storage system further comprises a message queue which is deployed among different servers according to the data volume and the like. The message queue monitors the operation request sent by the service agent to the server. And when the operation request is monitored to be the target operation request, the message queue generates a corresponding data replication task and sends a routing query request to a corresponding server. And the server returns the storage route corresponding to the operation request obtained by query to the message queue. The message queue asynchronously copies the target data pointed by the operation request from the current server to other servers of the same storage type according to the storage route.
In the embodiment, asynchronous data copying among different servers is realized based on the message queue, so that the method is high in timeliness, can realize second-level quasi-real-time copying operation, is high in reliability, and realizes final consistency of multiple backups through the persistence and the reliability of the message queue. Meanwhile, the scheme directly solves the problem of backup across machine rooms, and the same replication backup strategy can be adopted for remote machine room storage.
In one embodiment, the target data before replication is recorded as source data; the method further comprises the following steps: in the data copying process, extracting data identifications in source data and data identifications in target data copied to other servers; carrying out hash operation on the data identification to obtain a hash result; grouping the hash results according to the number of preset target threads; determining a target thread corresponding to each group; and calling a plurality of target threads to perform concurrent comparison on the source data and the target data corresponding to the hash result in the corresponding group.
Data storage is required in a variety of scenarios, such as a product sales scenario. The business client for product sale comprises a plurality of functional modules, such as a product module, an order module, a bill module, a user management module and the like. The different functional modules can also be regarded as an independent service client. Each functional module in the product sale client based on the microservice is provided with a corresponding distributed storage server. For example, the product module stores product data corresponding to each of a plurality of products in a server a (referred to as a product server). Order module order data corresponding to a plurality of users are stored in a server B (referred to as an order server).
Data comparison is also needed in various scenarios, for example, in the above example, when a user purchases a certain product, a corresponding data record is added to the product server. The data record may include an order number and product data and user data corresponding to the order, etc. And the product sale application synchronizes the newly added data record of the product server to the order server so as to add corresponding order data in the order server. At this time, the data in the product library and the order library need to be compared.
When data alignment is needed, the user can send a task configuration request to the service agent through the second terminal. And returning a task configuration page to the second terminal by the service agent according to the task configuration request. The task configuration page comprises configuration items corresponding to the plurality of task steps respectively. The task steps comprise sliding Window (Slide Window) generation, source data capture, target data capture, data comparison, comparison result storage and the like. The configuration items corresponding to the sliding window generating step comprise the size of the sliding window, the number of target threads and the like. And the corresponding comparison task is automatically operated by taking the size of the sliding window as the granularity, so that the timeliness and the continuity of data comparison can be ensured. The target thread data refers to how many threads are expected to be allocated to perform the matching task. The number of target threads determines the concurrency of the comparison tasks. The configuration items corresponding to the source data capturing step comprise source data capturing logic, such as a source database connection identifier, a source data table name, a data structure conversion rule and the like. Similarly, the configuration items corresponding to the target data capture step include target data capture logic. The configuration items corresponding to the data comparison step comprise fields to be compared and the like.
And in the current sliding window, the service agent sends a connection request to the source database according to the source data connection identifier recorded by the task configuration information, and pulls the source data generated in a preset time period from the source database with the established connection. And the service agent sends a connection request to the target database according to the target data connection identifier recorded by the task configuration information, and pulls target data generated in a preset time period from the target database with the established connection. The preset time interval is a time slice corresponding to the current sliding window, and the time length of the preset time interval is the size of the sliding window.
The service agent extracts the data identification in the source data and the target data. The data identification is information capable of uniquely identifying a piece of source data or target data. For example, in the above example, the data identifier may be an order number, a transaction serial number, or the like, or may be identifying information obtained by splicing contents of other fields based on the order number or the transaction serial number, or the like.
And the server performs hash calculation on the data identification through a hash algorithm to obtain a hash result corresponding to each data identification. The data identifier such as the transaction serial number may contain information such as letters, and has no numbering function. The hash result obtained by hashing the data identifier may be regarded as a data number, such as source data 1, target data 2, and the like. The hash result may be understood as a hash value that can uniquely distinguish each piece of source data from the target data.
And the server sends the hash result and the associated source data and target data to the server in a Key Value pair Key-Value form, wherein the Key Value is the hash result corresponding to the data identifier, and the Value is the source data and the target data corresponding to the corresponding data identifier. Value values may be stored in a linked List (List) form in the current database, including the source data linked List and the destination data linked List.
And the service agent groups the pulled source data and the pulled target data according to the target thread number recorded by the task configuration information, namely, the source data and the target data are divided into groups with the target thread number. For example, if the number of target threads is 5, the source data and the target data are divided into 5 packets. The server assigns a target thread to each packet.
In one embodiment, grouping the hash results according to a preset number of target threads comprises: determining the thread number of each target thread; carrying out division operation on the hash result and the number of the target threads to obtain a remainder corresponding to the hash result; grouping source data and target data having the same remainder; determining the target thread corresponding to each packet comprises: and distributing each group of source data and target data to be compared to a target thread with the corresponding remainder as a thread number.
And the service agent calls a corresponding number of target threads according to the target thread data recorded by the task configuration information, and numbers each target thread according to a preset rule to obtain a thread number corresponding to each target thread. The thread number may be a preset initial value and a result value of performing an adding count operation on the preset initial value. For example, if the preset initial value is 1, the thread numbers of the above-mentioned exemplary 5 target threads may be 1, 2, 3, 4, and 5 in sequence.
And the service agent performs division operation on each hash result and the number of the target threads to obtain a remainder corresponding to each hash result. The server divides the source data and the target data corresponding to the hash results with the same remainder into a group to obtain a plurality of groups. The remainder may be the packet number of the corresponding packet. It is easy to understand that this grouping scheme can ensure that at least one source data and at least one target data corresponding to the same data identifier are necessarily grouped into the same group. For example, the data IDs 1-ID 10000, the target thread data is 10, and a plurality of groups can be obtained by dividing, wherein the group 1 includes data ID1, data ID11, data ID21 and other data whose remainder is 1; packet 2 includes data having a data identification of 2 with a remainder of 10, such as data ID2, data ID12, data ID22, and so on, thereby dividing data IDs 1 to ID10000 into 10 packets.
The service agent distributes the source data and the target data to be compared in each group to a target thread taking the corresponding remainder as a thread number, and stores the thread number of the distributed target thread and the corresponding hash result together as a Key value in a redis database. For example, in the above example, the target thread with the thread number "1" may be called to compare the packet endogenous data with the packet target data with the thread number "1". Different target threads are isolated during data comparison, grouping is carried out according to the data identification, all source data and target data corresponding to the same service can be guaranteed to be distributed to the same target thread for execution, and then data comparison accuracy can be guaranteed.
And the service agent calls a plurality of target threads to perform concurrent comparison on the source data and the target data corresponding to the hash result in the corresponding packet. After dividing a large amount of source data and target data into a plurality of groups and distributing a corresponding target thread for each group, the server calls the plurality of target threads to perform concurrent comparison on the source data and the target data in the corresponding groups.
In this embodiment, grouping according to the data identifier can ensure that all source data and target data corresponding to the same service are allocated to the same target thread for execution, thereby avoiding a phenomenon that the source data and the target data corresponding to the same data identifier are misjudged to be inconsistent because the source data and the target data are allocated to different target threads for isolation comparison, and ensuring data comparison accuracy. The tasks are grouped according to the data identification, and the concurrent comparison is realized, so that the data comparison efficiency can be improved under the condition of ensuring the accuracy of the data comparison result.
It should be understood that, although the steps in the flowchart of fig. 2 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 2 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 4, there is provided a data storage device comprising: an operation request module 402, a request processing module 404, and a data storage module 406, wherein:
an operation request module 402, configured to intercept an operation request for target data sent by a first terminal;
a request processing module 404, configured to determine, through a preset general data operation interface, an attribute feature corresponding to the operation request; inquiring the storage route of the target data according to the attribute characteristics;
the data storage module 406 is configured to invoke a processing component adapted to the storage route to send the operation request to the server to which the storage route points, so that the server stores the target data according to the storage route; and sending the storage address of the target data returned by the server to the first terminal.
In an embodiment, the data storage device further includes a route configuration module 408, configured to obtain service deployment information of the plurality of servers based on a route configuration request sent by the second terminal; sending a routing configuration page generated according to the service deployment information to the second terminal; the route configuration page shows a plurality of server identifications and a plurality of service types; and storing the incidence relation between each service type and the server identifier returned by the second terminal based on the route configuration page as a storage route.
In one embodiment, the storage reason includes a base route and a dynamic route; the basic route records a plurality of service types and server identifications respectively associated with each service type; the dynamic routing records different attribute feature combinations of each service type and server identifications associated with each attribute feature combination; the attribute feature combination includes at least one of a target data identification or a data operation type.
In one embodiment, the data storage apparatus includes a data replication module 410, configured to query a storage route corresponding to an operation request when the operation request of a preset type is intercepted; and asynchronously copying the target data pointed by the operation request from the current server to other servers of the same storage type according to the storage route.
In one embodiment, the target data before replication is recorded as source data; the data storage device further includes a data comparison module 412, configured to extract data identifiers in the source data and data identifiers in the target data copied to other servers in the data copying process; carrying out hash operation on the data identification to obtain a hash result; grouping the hash results according to the number of preset target threads; determining a target thread corresponding to each group; and calling a plurality of target threads to perform concurrent comparison on the source data and the target data corresponding to the hash result in the corresponding group.
In one embodiment, the data comparison module 412 is further configured to determine a thread number for each target thread; carrying out division operation on the hash result and the number of the target threads to obtain a remainder corresponding to the hash result; grouping source data and target data having the same remainder; determining the target thread corresponding to each packet comprises: and distributing each group of source data and target data to be compared to a target thread with the corresponding remainder as a thread number.
For specific limitations of the data storage device, reference may be made to the above limitations of the data storage method, which are not described herein again. The various modules in the data storage device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store a routing table. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data storage method.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
A computer-readable storage medium, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of the data storage method as provided in any one of the embodiments of the application.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of data storage, the method comprising:
intercepting an operation request for target data sent by a first terminal;
determining attribute characteristics corresponding to the operation request through a preset general data operation interface;
inquiring the storage route of the target data according to the attribute characteristics;
calling a processing component adaptive to the storage route to send the operation request to a server pointed by the storage route, so that the server stores target data according to the storage route;
and sending the storage address of the target data returned by the server to the first terminal.
2. The method of claim 1, wherein before querying the storage route of the target data according to the attribute characteristics, the method further comprises:
acquiring service deployment information of a plurality of servers based on a routing configuration request sent by a second terminal;
sending a routing configuration page generated according to the service deployment information to the second terminal; the routing configuration page shows a plurality of server identifications and a plurality of service types;
and storing the incidence relation between each service type and the server identifier returned by the second terminal based on the route configuration page as a storage route.
3. The method according to claim 1 or 2, wherein the stored reason comprises a base route and a dynamic route; the basic route records a plurality of service types and server identifications respectively associated with each service type; the dynamic route records different attribute feature combinations of each service type and server identifications associated with each attribute feature combination; the attribute feature combination comprises at least one of target data identification and data operation type.
4. The method of claim 1, further comprising:
when an operation request of a preset type is intercepted, inquiring a storage route corresponding to the operation request;
and asynchronously copying the target data pointed by the operation request from the current server to other servers with the same storage type according to the storage route.
5. The method according to claim 4, wherein target data before copying is regarded as source data; the method further comprises the following steps:
in the data copying process, extracting data identifications in source data and data identifications in target data copied to other servers;
carrying out hash operation on the data identification to obtain a hash result;
grouping the hash results according to the number of preset target threads;
determining a target thread corresponding to each group;
and calling a plurality of target threads to perform concurrent comparison on the source data and the target data corresponding to the hash result in the corresponding group.
6. The method of claim 5, wherein grouping the hash results according to a preset number of target threads comprises:
determining the thread number of each target thread;
carrying out division operation on the hash result and the number of target threads to obtain a remainder corresponding to the hash result;
grouping source data and target data having the same remainder;
the determining the target thread corresponding to each packet comprises: and distributing each group of source data and target data to be compared to a target thread with the corresponding remainder as a thread number.
7. A data storage device, characterized in that the device comprises:
the operation request module is used for intercepting an operation request for target data sent by a first terminal;
the request processing module is used for determining attribute characteristics corresponding to the operation request through a preset general data operation interface; inquiring the storage route of the target data according to the attribute characteristics;
the data storage module is used for calling a processing component adaptive to the storage route to send an operation request to a server pointed by the storage route, so that the server performs storage operation on the target data according to the storage route; and sending the storage address of the target data returned by the server to the first terminal.
8. The device according to claim 7, wherein the device comprises a data replication module, configured to, when an operation request of a preset type is intercepted, query a storage route corresponding to the operation request; and asynchronously copying the target data pointed by the operation request from the current server to other servers with the same storage type according to the storage route.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910969056.0A CN110764700B (en) | 2019-10-12 | 2019-10-12 | Data storage method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910969056.0A CN110764700B (en) | 2019-10-12 | 2019-10-12 | Data storage method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110764700A true CN110764700A (en) | 2020-02-07 |
| CN110764700B CN110764700B (en) | 2023-02-07 |
Family
ID=69331930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910969056.0A Active CN110764700B (en) | 2019-10-12 | 2019-10-12 | Data storage method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110764700B (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111585897A (en) * | 2020-04-02 | 2020-08-25 | 深圳壹账通智能科技有限公司 | Request route management method, system, computer system and readable storage medium |
| CN111756748A (en) * | 2020-06-24 | 2020-10-09 | 中国建设银行股份有限公司 | Data interaction method and device, electronic equipment and storage medium |
| CN112308726A (en) * | 2020-11-26 | 2021-02-02 | 泰康保险集团股份有限公司 | Insurance service rights pushing method, insurance service rights pushing device, insurance service rights pushing medium and electronic equipment |
| CN112486988A (en) * | 2020-11-27 | 2021-03-12 | 京东方科技集团股份有限公司 | Data processing method, device, equipment and storage medium |
| CN112597222A (en) * | 2020-12-16 | 2021-04-02 | 中国建设银行股份有限公司 | Financial database connection processing method, device, equipment and storage medium |
| CN112685182A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN113190576A (en) * | 2021-06-01 | 2021-07-30 | 京东科技控股股份有限公司 | Data processing method and device, computer equipment and readable storage medium |
| CN113220759A (en) * | 2021-04-27 | 2021-08-06 | 深圳市云网万店科技有限公司 | Big data storage service sharing method and device, computer equipment and storage medium |
| CN113821162A (en) * | 2021-02-24 | 2021-12-21 | 北京沃东天骏信息技术有限公司 | Storage object operation method and device |
| CN114063931A (en) * | 2021-11-26 | 2022-02-18 | 重庆科创职业学院 | Data storage method based on big data |
| CN114760326A (en) * | 2022-03-02 | 2022-07-15 | 杭州华橙软件技术有限公司 | Data storage method, data query method, data storage system and electronic device |
| CN117194298A (en) * | 2023-09-18 | 2023-12-08 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741850A (en) * | 2009-12-25 | 2010-06-16 | 北京邮电大学 | Multitask concurrent executive system and method for hybrid network service |
| CN104050249A (en) * | 2011-12-31 | 2014-09-17 | 北京奇虎科技有限公司 | Distributed query engine system and method and metadata server |
| CN104504122A (en) * | 2014-12-29 | 2015-04-08 | 浪潮(北京)电子信息产业有限公司 | Database migration data verification method and system |
| CN105956029A (en) * | 2016-04-25 | 2016-09-21 | 吉林大学 | Data access method under hybrid storage architecture |
| CN106708917A (en) * | 2016-06-30 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data processing method and device and OLAP system |
| CN106777225A (en) * | 2016-12-26 | 2017-05-31 | 腾讯科技（深圳）有限公司 | The moving method and system of a kind of data |
| CN107645508A (en) * | 2017-10-16 | 2018-01-30 | 深圳市买买提乐购金融服务有限公司 | A kind of data handling system, method, client and server |
| CN109445754A (en) * | 2018-09-14 | 2019-03-08 | 中国建设银行股份有限公司 | Data integration access mechanism, method and storage medium |
| CN109634912A (en) * | 2018-12-10 | 2019-04-16 | 苏州思必驰信息科技有限公司 | Data migration method and system |
-
2019
- 2019-10-12 CN CN201910969056.0A patent/CN110764700B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101741850A (en) * | 2009-12-25 | 2010-06-16 | 北京邮电大学 | Multitask concurrent executive system and method for hybrid network service |
| CN104050249A (en) * | 2011-12-31 | 2014-09-17 | 北京奇虎科技有限公司 | Distributed query engine system and method and metadata server |
| CN104504122A (en) * | 2014-12-29 | 2015-04-08 | 浪潮(北京)电子信息产业有限公司 | Database migration data verification method and system |
| CN105956029A (en) * | 2016-04-25 | 2016-09-21 | 吉林大学 | Data access method under hybrid storage architecture |
| CN106708917A (en) * | 2016-06-30 | 2017-05-24 | 腾讯科技（深圳）有限公司 | Data processing method and device and OLAP system |
| CN106777225A (en) * | 2016-12-26 | 2017-05-31 | 腾讯科技（深圳）有限公司 | The moving method and system of a kind of data |
| CN107645508A (en) * | 2017-10-16 | 2018-01-30 | 深圳市买买提乐购金融服务有限公司 | A kind of data handling system, method, client and server |
| CN109445754A (en) * | 2018-09-14 | 2019-03-08 | 中国建设银行股份有限公司 | Data integration access mechanism, method and storage medium |
| CN109634912A (en) * | 2018-12-10 | 2019-04-16 | 苏州思必驰信息科技有限公司 | Data migration method and system |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111585897B (en) * | 2020-04-02 | 2022-04-15 | 深圳壹账通智能科技有限公司 | Request route management method, system, computer system and readable storage medium |
| CN111585897A (en) * | 2020-04-02 | 2020-08-25 | 深圳壹账通智能科技有限公司 | Request route management method, system, computer system and readable storage medium |
| CN111756748A (en) * | 2020-06-24 | 2020-10-09 | 中国建设银行股份有限公司 | Data interaction method and device, electronic equipment and storage medium |
| CN111756748B (en) * | 2020-06-24 | 2022-11-15 | 中国建设银行股份有限公司 | Data interaction method and device, electronic equipment and storage medium |
| CN112308726A (en) * | 2020-11-26 | 2021-02-02 | 泰康保险集团股份有限公司 | Insurance service rights pushing method, insurance service rights pushing device, insurance service rights pushing medium and electronic equipment |
| CN112308726B (en) * | 2020-11-26 | 2024-03-12 | 泰康保险集团股份有限公司 | Insurance service equity pushing method, apparatus, medium and electronic device |
| CN112486988A (en) * | 2020-11-27 | 2021-03-12 | 京东方科技集团股份有限公司 | Data processing method, device, equipment and storage medium |
| CN112597222A (en) * | 2020-12-16 | 2021-04-02 | 中国建设银行股份有限公司 | Financial database connection processing method, device, equipment and storage medium |
| CN112685182A (en) * | 2020-12-29 | 2021-04-20 | 平安普惠企业管理有限公司 | Service data processing method and device, computer equipment and storage medium |
| CN113821162A (en) * | 2021-02-24 | 2021-12-21 | 北京沃东天骏信息技术有限公司 | Storage object operation method and device |
| CN113220759A (en) * | 2021-04-27 | 2021-08-06 | 深圳市云网万店科技有限公司 | Big data storage service sharing method and device, computer equipment and storage medium |
| CN113190576A (en) * | 2021-06-01 | 2021-07-30 | 京东科技控股股份有限公司 | Data processing method and device, computer equipment and readable storage medium |
| CN114063931A (en) * | 2021-11-26 | 2022-02-18 | 重庆科创职业学院 | Data storage method based on big data |
| CN114760326A (en) * | 2022-03-02 | 2022-07-15 | 杭州华橙软件技术有限公司 | Data storage method, data query method, data storage system and electronic device |
| CN114760326B (en) * | 2022-03-02 | 2024-07-23 | 杭州华橙软件技术有限公司 | Data storage method, data query method, data storage system and electronic device |
| CN117194298A (en) * | 2023-09-18 | 2023-12-08 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
| CN117194298B (en) * | 2023-09-18 | 2024-05-31 | 上海鸿翼软件技术股份有限公司 | Control method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110764700B (en) | 2023-02-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110764700B (en) | Data storage method and device, computer equipment and storage medium | |
| US11928029B2 (en) | Backup of partitioned database tables | |
| CN108829459B (en) | Nginx server-based configuration method and device, computer equipment and storage medium | |
| EP4047487A1 (en) | File storage method, terminal, and storage medium | |
| US10776174B2 (en) | Managing hosted resources across different virtualization platforms | |
| US11036591B2 (en) | Restoring partitioned database tables from backup | |
| CN110290212B (en) | Service call recording method, device, computer equipment and storage medium | |
| CN108197200B (en) | Log tracking method and device, computer equipment and storage medium | |
| CN108804618B (en) | Database configuration method, device, computer equipment and storage medium | |
| US9563426B1 (en) | Partitioned key-value store with atomic memory operations | |
| US7490265B2 (en) | Recovery segment identification in a computing infrastructure | |
| US9632878B1 (en) | Verification of database table partitions during backup | |
| US11048591B1 (en) | Efficient name space organization in a global name space cluster | |
| CN108959385B (en) | Database deployment method, device, computer equipment and storage medium | |
| CN110908778B (en) | Task deployment method, system and storage medium | |
| US8660996B2 (en) | Monitoring files in cloud-based networks | |
| CN109460252B (en) | Configuration file processing method and device based on git and computer equipment | |
| US11442927B1 (en) | Storage performance-based distribution of deduplicated data to nodes within a clustered storage environment | |
| CN110765162A (en) | Data comparison method and device, computer equipment and storage medium | |
| CN110781214A (en) | Database reading and writing method and device, computer equipment and storage medium | |
| CN109933338B (en) | Block chain deployment method, device, computer equipment and storage medium | |
| CN114281263B (en) | Storage resource processing method, system and equipment of container cluster management system | |
| WO2024016624A1 (en) | Multi-cluster access method and system | |
| CN117642724A (en) | Stream analysis using server-less computing system | |
| CN107276998B (en) | OpenSSL-based performance optimization method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200201 Address after: 200120 floor 15, 1333 Lujiazui Ring Road, free trade Experimental Zone, Pudong New Area, Shanghai Applicant after: Weikun (Shanghai) Technology Service Co.,Ltd. Address before: 13th Floor, 1333 Lujiazui Ring Road, Shanghai Free Trade Pilot Area, 200120 Applicant before: SHANGHAI LUJIAZUI INTERNATIONAL FINANCIAL ASSETS TRANSACTION MARKET CO.,LTD. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230717 Address after: No. 2-1, Building B3, Group B, Phase I, Nanning ASEAN Enterprise Headquarters Base, No. 10, Xinji Road, High tech Zone, Guangxi Zhuang Autonomous Region, 530000 Patentee after: Guangxi sanfangda Supply Chain Technology Service Co.,Ltd. Address before: 15 / F, 1333 Lujiazui Ring Road, Pudong New Area pilot Free Trade Zone, Shanghai, 200120 Patentee before: Weikun (Shanghai) Technology Service Co.,Ltd. |