CN106484311A - A kind of data processing method and device - Google Patents
A kind of data processing method and device Download PDFInfo
- Publication number
- CN106484311A CN106484311A CN201510547258.8A CN201510547258A CN106484311A CN 106484311 A CN106484311 A CN 106484311A CN 201510547258 A CN201510547258 A CN 201510547258A CN 106484311 A CN106484311 A CN 106484311A
- Authority
- CN
- China
- Prior art keywords
- node
- data
- write
- asked
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of data processing method and device, for solving the problems, such as data writing mode in prior art it cannot be guaranteed that being successfully written data on the desired node of user.The data processing method includes：Calculate node receives the write request that client sends, and the write request is comprising the data for being asked to write；The calculate node is determined for storing the primary storage node for being asked the data for writing and M secondary node；The calculate node writes the data for being asked to write in the primary storage node；The calculate node receives the write success message that the primary storage node is returned；The calculate node is respectively written into the data for being asked to write to the M secondary node；The calculate node determination receives the write success message that each node is returned respectively in N number of preassigned secondary node；The calculate node is returned to the client and writes success message.
Description
Technical field
The present invention relates to field of data storage, more particularly to a kind of data processing method and device.
Background technology
Distributed system refers to computing resource and storage resource are distributed on different nodes, will by network
The system that each node contacts get up.Distributed system by managing and can control multiple nodes completing one
Task, but the system for externally showing as an independent completion, user are perceived less than including distributed system task
The execution details in portion.
Fig. 1 is the schematic diagram of distributed system, and wherein, client (Client) is used for receive user demand
(as write request and read request), calculate node (English：Distributed Object Process；Referred to as：
DOP) it is responsible for distribution route, and memory node is (English：Key-Value Store；Referred to as：KVS) bear
The management of duty single-deck and actual storage, in addition, also include management node (English in system：Distributed Object
Management；Referred to as：DOM), it is responsible for being managed multiple DOP and KVS.
Distributed system preserves data using many replication policies, will same data be stored on multiple KVS,
With the loss of data for avoiding KVS fault from causing.Meanwhile, system will also ensure multiple copies of same data
Uniformity, i.e., with multiple KVS store same data when, it is ensured that on multiple KVS store data one
Cause.
NRW mechanism is a kind of strategy for taking into account many copy consistencies and data read-write efficiency, wherein, N table
According to the number of copies having, R represents the minimum number of copies for completing to read required for read operation to registration, and W represents
Complete the minimum node number participated in required for time write operation of minimum number of copies, i.e., of write required for write operation
Mesh.For example, in NRW322 mechanism, in system, data have 3 different copies, grasp when row write is entered
When making, need at least 2 copies of wait to complete the write operation system and the state for running succeeded can be just returned,
And when read operation is carried out, then need to read at least two copies, determine when two copy datas are consistent
Read operation success.
In addition, distributed system also solves concurrency conflict during write operation using primary copy locking, i.e. from
A memory node is selected in multiple memory nodes of data storage as primary storage node, remaining memory node
Referred to as slack storage node.When data are write, calculate node needs the lock of writing for first competing primary storage node to weigh
Limit, writes data to primary storage node after lock authority is write in acquisition, is write to secondary node after write success again
Data, when there is the secondary node for setting quantity return to be write as work(, calculate node is returned to client and is write as work(
Message, remaining secondary node proceed data writing operation using asynchronous mechanism.
With reference to Fig. 2, by taking NRW322 mechanism as an example, primary storage node writes the process of data in memory node
Comprise the steps：Step 101, client send write request to calculate node；Step 102, calculates section
That puts acquisition primary storage node writes lock authority, and writes data to primary storage node；Step 103, primary storage
Node returns write success message to calculate node；Step 104, calculate node write data to secondary node；
Step 105, arbitrary secondary node (might as well be set to secondary node 2) are returned to write to calculate node and are successfully disappeared
Breath；Step 106, calculate node return write success message to client.
In actual conditions, user may expect successfully to preserve data on the secondary node of part, for example, number
According to data are preserved on the high node of reading rate, read so as to quick, or, the high section of storage stability
Data are preserved on point, to avoid loss of data, or, preserve on the node of user office
There are data.But, in technique scheme, calculate node is returned in the secondary node for receiving setting quantity
Return to client when writing success message and success message is write, but regardless of specifically returned by which secondary node writing
Success message, accordingly, it is possible to when the desired secondary node of user not yet writes data, calculate node is
Return to client and write success message, and the desired secondary node of user is failed due to fault afterwards
Write data, cause data to be not stored on the desired node of user.
Content of the invention
The embodiment of the present invention provides a kind of data processing method and device, writes for solving data in prior art
Enter mode it cannot be guaranteed that the problem of data is successfully written on the desired node of user.
In a first aspect, the embodiment of the present invention provides a kind of data processing method, including：
Calculate node receives the write request that client sends, and the write request is comprising the data for being asked to write；
The calculate node determines standby for storing the primary storage node for being asked the data for writing and M
With node, wherein, the M secondary node includes N number of preassigned secondary node, M be more than
1 positive integer, N are the positive integer less than M；
The calculate node writes the data for being asked to write in the primary storage node；
The calculate node receives the write success message that the primary storage node is returned；
The calculate node is respectively written into the data for being asked to write to the M secondary node；
The calculate node determines that receiving each node in N number of preassigned secondary node divides
The write success message not returned；
The calculate node is returned to the client and writes success message.
In conjunction with a first aspect, in the first possible implementation of first aspect, N number of advance finger
Fixed secondary node is the node that storage performance parameter meets preset requirement in the M secondary node.
In conjunction with the first possible implementation of first aspect or first aspect, at second of first aspect
In possible implementation, after the calculate node is returned to the client and writes success message, also wrap
Include：
The calculate node receives the read request that the client sends, and the read request packet contains asked reading
According to mark；
The calculate node determines the corresponding described N number of advance finger for being asked to read data of the storage mark
Fixed secondary node；And
Read in N number of preassigned secondary node the corresponding with the mark of storage in any node
Asked read data, and by read the data return to the client.
In conjunction with second possible implementation of first aspect, in the third possible realization of first aspect
In mode, before the data for reading are returned to the client by the calculate node, also include：
The calculate node determines in N number of preassigned secondary node the right with the mark of storage
That answered is asked reading data all same.
The third possible realization of second possible implementation or first aspect in conjunction with first aspect
Mode, in the 4th kind of possible implementation of first aspect, the calculate node is stored with mapping table,
The mapping table preserves the corresponding relation of Data Identification and data storage location, the data storage location bag
Include described N number of preassigned secondary node of data storage；
The calculate node determines the corresponding described N number of advance finger for being asked to read data of the storage mark
Fixed secondary node, including：
The calculate node determines the corresponding institute of the storage mark according to the mark and the mapping table
Described N number of preassigned secondary node of data is read in request.
Second aspect, the embodiment of the present invention provide a kind of data processing equipment, including：
Receiver module, for receiving the write request of client transmission, the write request includes is asked write
Data；
Determining module, for determining for storing the primary storage node for being asked the data for writing and M
Secondary node, wherein, the M secondary node includes N number of preassigned secondary node, and M is big
In 1 positive integer, N is the positive integer less than M；
Writing module, is asked, for the write in the primary storage node, the data for writing；
The receiver module is additionally operable to：Receive the write success message of primary storage node return；
Said write module is additionally operable to：The write of primary storage node return is received in the receiver module
After success message, the data for being asked to write are respectively written into the M secondary node；
The determining module is additionally operable to：Determine and receive each in N number of preassigned secondary node
The write success message that node is returned respectively；
Described device also includes：Sending module, described N number of pre- for receiving in determining module determination
After the write success message that each node is returned respectively in the secondary node that first specifies, return to the client
Write-back success message.
In conjunction with second aspect, in the first possible implementation of second aspect, the determining module is used
In：Determine that N number of node that storage performance parameter meets preset requirement in the M secondary node is described
Preassigned secondary node.
In conjunction with the first possible implementation of second aspect or second aspect, at second of second aspect
In possible implementation, the receiver module is additionally operable to：Receive the read request of client transmission, institute
Read request packet is stated containing the mark for being asked to read data；
The determining module is additionally operable to：Determine that the storage mark is corresponding to be asked to read the described N number of of data
Preassigned secondary node；
Described device also includes：Read module, appoints for reading in N number of preassigned secondary node
In one node, the corresponding with the mark of storage is asked to read data；
The sending module is additionally operable to：The data for reading are returned to the client.
In conjunction with second possible implementation of second aspect, in the third possible realization of second aspect
In mode, the determining module is additionally operable to：Determine in N number of preassigned secondary node storage with
The mark is corresponding to be asked to read data all same；
The sending module is used for：Determine in N number of preassigned secondary node in the determining module
Storage corresponding with the mark ask reading data all same after, by reading the data return to
The client.
The third possible realization of second possible implementation or second aspect in conjunction with second aspect
Mode, in the 4th kind of possible implementation of second aspect, described device is stored with mapping table, described
Mapping table preserves the corresponding relation of Data Identification and data storage location, and the data storage location includes to deposit
Described N number of preassigned secondary node of storage data；The determining module is used for：According to described mark with
And the mapping table, determine that the storage mark is corresponding and asked to read the described N number of preassigned of data
Secondary node.
The one or more technical schemes provided in the embodiment of the present invention, at least have the following technical effect that or excellent
Point：
In the embodiment of the present invention, calculate node when data are write according to write request in memory node, to
In primary storage node after write data success, number is all write only in N number of preassigned secondary node
Success message is write according to just returning to client after success, wherein, N number of preassigned secondary node can be
The desired secondary node of user (the preferable node of such as storage performance).Therefore, technique scheme energy
Enough ensure data to be successfully written on the desired secondary node of user, meet the demand of user, improve data
The stability of reading.
Description of the drawings
For the technical scheme being illustrated more clearly that in the embodiment of the present invention, below will be to institute in embodiment description
The accompanying drawing for using is needed to briefly introduce, it should be apparent that, drawings in the following description are only the present invention's
Some embodiments, for one of ordinary skill in the art, in the premise for not paying creative labor
Under, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is the schematic diagram of distributed system；
Fig. 2 is the schematic flow sheet for writing data in background technology；
Fig. 3 is the schematic flow sheet of data processing method in the embodiment of the present invention；
Fig. 4 is another schematic flow sheet of data processing method in the embodiment of the present invention；
Fig. 5 is the another schematic flow sheet of data processing method in the embodiment of the present invention；
Fig. 6 is the schematic diagram for writing data in the embodiment of the present invention；
Fig. 7 is the schematic diagram for reading data in the embodiment of the present invention；
Fig. 8 is the structural schematic block diagram of data processing equipment in the embodiment of the present invention；
Fig. 9 is the structural schematic block diagram of data processing equipment in the embodiment of the present invention.
Specific embodiment
Technical solution of the present invention is described in detail below by accompanying drawing and specific embodiment, it should be understood that
Specific features in the embodiment of the present invention and embodiment are the detailed description to technical solution of the present invention, and
It is not the restriction to technical solution of the present invention, in the case of not conflicting, the embodiment of the present invention and embodiment
In technical characteristic can be mutually combined.
Fig. 3 is the schematic flow sheet of data processing method provided in an embodiment of the present invention, and the flow process includes as follows
Step：
Step 101：Calculate node receives the write request that client sends, and write request includes is asked write
Data；
Step 102：Calculate node determines primary storage node and the M for being asked the data for writing for storage
Individual secondary node, wherein, M secondary node includes N number of preassigned secondary node, M be more than 1
Positive integer, N is the positive integer less than M；
Step 103：Calculate node writes the data for being asked to write in primary storage node；
Step 104：Calculate node receives the write success message that primary storage node is returned；
Step 105：Calculate node is respectively written into the data for being asked to write to M secondary node；
Step 106：Calculate node determines that receiving each node in N number of preassigned secondary node divides
The write success message not returned；
Step 107：Calculate node is returned to client and writes success message.
Specifically, in step 101, the request request that client writes data according to user generates write request,
And write request is sent to calculate node, and calculate node receives the write request, and can obtain from write request
Asked the data for writing.In actual conditions, the write request can be sent to arbitrary calculate node by client,
The target computing nodes for sending write request can also be determined according to preset rules, for example, write request is sent to
The maximum calculate node of the computing capability of current free time, or, it is sent to the calculating most short with oneself time delay
Node, etc., will not be described in detail herein.
In step 102, calculate node determine write data target storage node, target storage node that is,
For storing the primary storage node for being asked the data for writing and M secondary node.Concrete determination mode
Including：
Mode 1, during the new data not stored before the data for being asked to write are data-storage system, meter
Operator node is the data distribution primary storage node and M secondary node from memory node；Or, meter
Operator node sends memory node distribution request to management node, is carried out the distribution of memory node by management node,
Determine above-mentioned primary storage node and M secondary node.Wherein, the method for salary distribution of memory node refer to
Related art method in prior art, will not be described in detail herein.
Mode 2, the data storage being stored in before the data for being asked to write are in data-storage system
When, also comprising having asked to write the mark of data in the write request that client sends, safeguard in calculate node
There are data mapping tables, be stored with the table Data Identification and the corresponding relation of data storage location, calculate node
The mark for being asked to write data included in inquiry write request in the mapping table, you can determine that storage is asked
Seek the primary storage node and M secondary node for writing data.
In the embodiment of the present invention, no matter adopt and all wrap in aforesaid way 1 or mode 2, a M secondary node
Containing N number of preassigned secondary node, wherein N is less than M, and N number of preassigned node can be
Specified asking to write when data carry out secondary node distribution first, N number of preassigned section
Point can be according to user's characteristic of interest (e.g., the reading rate of memory node, the stablizing of memory node
Property, the life-span of memory node, the speed of response of memory node, the burden of memory node, etc.) carry out really
Fixed, i.e. select from M secondary node to meet the requirements in terms of user's characteristic of interest (or ranking
Forward) N number of node as preassigned secondary node.
Then, execution step 103, calculate node is write in the primary storage node that determines is asked write
Data, write success after, primary storage node will to calculate node return write success message, calculate node
Execution step 104, receives the write success message that primary storage node is returned, determines successfully in primary storage section
Data are write in point.
Then, calculate node execution step 105, is respectively written into the number for being asked to write to M secondary node
According to.In M secondary node, any node will be returned to write to calculate node and successfully be disappeared in write data success
Breath.
Then, calculate node will be received after the write success message of secondary node return, and judges this
Whether write success message is returned by preassigned secondary node.Only receive in determination N number of pre-
After the write success message that each node is returned respectively in the secondary node that first specifies, calculate node just determines
Data success is write, is returned to client and writes success message.
In other words, as long as receiving the write success message that N number of preassigned secondary node is returned, and not
Manage whether remaining node has been write as work(, calculate node will be returned to client and write success message.Or,
Even if the write success message for having had the individual secondary node of setting quantity (N) to return in M secondary node,
As long as there is a preassigned secondary node not yet to return write success message, calculate node also will not be to visitor
Family end returns and writes success message, and is to wait for preassigned secondary node and all returns write success message
Afterwards, just return to client and write success message.
In technique scheme, calculate node when data are write according to write request in memory node, to
In primary storage node after write data success, number is all write only in N number of preassigned secondary node
Success message is write according to just returning to client after success, wherein, N number of preassigned secondary node can be
The desired secondary node of user.Therefore, technique scheme is ensure that on the desired secondary node of user
Data are successfully written, meet the demand of user.
Optionally, in the embodiment of the present invention, N number of preassigned secondary node is in M secondary node
Storage performance parameter meets the node of preset requirement.Wherein, storage performance parameter can include：Read write
The parameters such as speed, job stability, service life, operational capability, memory space.Preset requirement can root
Set according to the applied environment of data-storage system and the real needs of user.
By determining that from M secondary node N number of storage performance parameter is full previously according to storage performance parameter
The node of sufficient preset requirement is used as preassigned node, so that N number of preassigned node when data are write
Return is write as work(and is write success message to client return, in the case of data write efficiency is ensured, it is ensured that
The stability of data storage, and the speed subsequently data being read out,
Optionally, in the embodiment of the present invention, after step 105, before step 106, can also include as
Lower step：If calculate node is writing the setting duration after data to M secondary node write request
Write success message that all preassigned nodes return inside is not received by, then calculate node is not to returning
It is written back into the preassigned node data that write request writes again of success message.
In technique scheme, can fail to return write success message in time in preassigned secondary node
When, actively write data again to which, it is to avoid write data take long.
Optionally, in the embodiment of the present invention, before step 103, also include step：Calculate node is obtained
Primary storage node write lock authority.Lock control write by primary storage node, it is to avoid many calculate nodes for
The concurrent operating conflict of one data.
Optionally, in the embodiment of the present invention, the write request that client sends also includes to be asked to write data
Mark and version number, before step 103, also include step：Calculate node determines what write request included
Version number is asked the corresponding version number of write data more than currently stored.By Version Control, it is to avoid have
Effect data are capped.
Further, present inventor has found, in prior art when data are read, in order to ensure the one of data
Cause property, needs the data read on primary storage node and sets the data of the secondary node of quantity, in checking
After data consistency, could return, to client, the data that request is read.
For continuing to use the example of NRW322, calculate node is needed to primary storage node and a standby section
Point sends read data request, then receives the two data for returning, and judges whether two data are consistent,
When consistent, the data for reading are returned to client.
In above-mentioned prior art, both need primary storage node is read, need again to read setting quantity (for example,
When data are write, the write success message of N number of preassigned secondary node return is received i.e. to client
Success message is write in return, then read to need to read being asked for N number of preassigned secondary node storage during data
Read data) secondary node storage asked reading data, digital independent amount is larger, and bandwidth occupancy is larger,
Time-consuming also longer.
For the problems referred to above, with reference to Fig. 4, data processing method provided in an embodiment of the present invention also includes as follows
Step：
Step 108：Calculate node receives the read request that client sends, and read request packet contains is asked to read data
Mark；
Step 109：Calculate node determines that mark that storage read request includes is corresponding ask reading data
N number of preassigned secondary node；
Step 110：Calculate node read in N number of preassigned secondary node storage in any node with
The mark is corresponding to be asked to read data；
Step 111：The data of reading are returned to client by calculate node.
Specifically, in step 108, client generates read request according to user's request, and read request is sent
To calculate node, comprising the mark for being asked reading data in the read request.Wherein, before being asked to read data
Ablation process can be responsible for by different calculate nodes from current reading process, or same calculating
Node is responsible for, and the embodiment of the present invention is not limited.
In step 109, calculate node determines that the mark that storage read request includes is corresponding preassigned standby
Included with the mode of node：
First, calculate node sends request to management node, request includes the mark, asks management node
The preassigned secondary node of the corresponding data of the mark is determined according to its data management information that safeguards, meter
Operator node receives the information that management node is returned, and therefrom determines and is asked to read the N number of preassigned standby of data
Use node.
The mapping table second, calculate node itself is stored with, the mapping table preserve Data Identification and data storage
The corresponding relation of position, data storage location include the N number of preassigned secondary node of data storage.Meter
Operator node is according to mark and mapping table, you can determines that storage mark is corresponding and is asked to read the N number of pre- of data
The secondary node that first specifies.
In step 110, calculate node is read and the mark from any node in N number of preassigned secondary node
Know corresponding data, the data are returned to client.
Compared with prior art, in technique scheme, calculate node is without the need for both reading on primary storage node
Data read the data set on secondary node again, but directly read on arbitrary preassigned secondary node
Storage asked read data.The reason for so processing feasible is, in writing in data procedures before,
Just returned to client when N number of preassigned secondary node writes as work(and success message is write, so, in advance
The requested data stored on the secondary node that specifies is the data being newly written, and returns the data to client
Can ensure to read the correctness of data.
By technique scheme, it is possible to reduce volume of transmitted data during data is read, reduce data and take band
Width, and minimizing reading data are time-consuming, improve data reading performance using redundancy.
Optionally, as another embodiment, before step 111, with reference to Fig. 5, also comprise the steps：
Step 112：Calculate node determines in N number of preassigned secondary node the corresponding with mark of storage
Asked to read data all same.
Specifically, calculate node sends to N number of preassigned secondary node and reads the mark corresponding data
Request, the data for receiving N number of node return are returned, and judge whether the data that N number of node is returned are consistent,
Execution step 111 when consistent, return the data for reading to client.If the data that N number of node is returned
Inconsistent, then report an error to client, processed using default error mechanism, embodiment of the present invention here
Not describe in detail.
By the uniformity of the data of the N number of preassigned secondary node of comparison, data can be preferably ensured
Uniformity, meanwhile, compared with prior art, calculate node still can read the number of primary storage node less
According to, reducing the read volume of data, the efficiency of data is read in raising.
It should be noted that primary storage node in the embodiment of the present invention and preassigned secondary node,
Secondary node etc. be for concrete data, for different data, its corresponding primary storage node with
And the secondary node for pre-establishing all can be different.
Below as a example by using improvement project of the embodiment of the present invention technical scheme to NRW322 technology, to sending out
Bright described in detail.
Specifically, storage system passes through primary storage node, preassigned secondary node 1, secondary node 2
Data storage A.Fig. 6 is the schematic diagram of the process for writing data A, comprises the steps：
Step 201, client send the request of write data A to calculate node；
Step 202, calculate node write data A to primary storage node；
Step 203, primary storage node return write success message to calculate node；
Step 204, calculate node write data A to secondary node；
Step 205, the write success message that secondary node 2 is returned to calculate node；
Step 206, calculate node determine that the write success message for receiving is not from the standby section for pre-establishing
Point 1；
Step 207, the write success message that secondary node 1 is returned to calculate node；
Step 208, calculate node determine the write success message for receiving from the secondary node 1 for pre-establishing,
Execution step 209；
Step 209, calculate node return write success message to client.
Corresponding, Fig. 7 is the schematic diagram of the process for reading data A, comprises the steps：
Step 210, client send the request for reading data A to calculate node；
Step 211, calculate node send the request for reading data A to preassigned secondary node 1；
Step 212, secondary node 1 is to calculate node returned data A；
Step 213, calculate node is to client returned data A.
When entering row write data by the way, ensure that write data become in the memory node that user specifies
Work(, and when reading data by the way, it is possible to increase read data efficiency, reduce bandwidth occupancy.
Based on identical technology design, the embodiment of the present invention additionally provides a kind of data processing equipment 300, ginseng
See Fig. 8, be the structural schematic block diagram of data processing equipment 300, including：
Receiver module 301, for receiving the write request of client transmission, write request includes is asked write
Data；
Determining module 302, for determining the primary storage node and the M that are asked the data for writing for storage
Individual secondary node, wherein, M secondary node includes N number of preassigned secondary node, M be more than 1
Positive integer, N is the positive integer less than M；
Writing module 303, is asked, for the write in primary storage node, the data for writing；
Receiver module 301 is additionally operable to：Receive the write success message of primary storage node return；
Writing module 303 is additionally operable to：The write success message of primary storage node return is received in receiver module
Afterwards, the data for being asked to write are respectively written into M secondary node；
Determining module 302 is additionally operable to：Determine and receive each node in N number of preassigned secondary node
The write success message for returning respectively；
Device 300 also includes：Sending module 304, receives N number of advance finger for determining in determining module
After the write success message that each node is returned respectively in fixed secondary node, return to client and write as work(
Message.
Optionally, in the embodiment of the present invention, determining module 302 is used for：Determine and deposit in M secondary node
It is preassigned secondary node that storage performance parameter meets N number of node of preset requirement.
Optionally, in the embodiment of the present invention, receiver module 301 is additionally operable to：The reading for receiving client transmission please
Ask, read request packet is containing the mark for being asked reading data；
Determining module 302 is additionally operable to：Determine that storage mark is corresponding to be asked to read the N number of predesignated of data
Secondary node；
Device 300 also includes：Read module 305, appoints for reading in N number of preassigned secondary node
In one node, the corresponding with mark of storage is asked to read data；
Sending module 304 is additionally operable to：The data of reading are returned to client.
Optionally, determining module 302 is additionally operable to：Determine in N number of preassigned secondary node storage with
Mark is corresponding to be asked to read data all same；
Sending module 304 is used for：Determining module determine in N number of preassigned secondary node storage with
Mark corresponding ask reading data all same after, the data of reading are returned to client.
Optionally, in the embodiment of the present invention, device 300 is stored with mapping table, and mapping table preserves data mark
Know the corresponding relation with data storage location, data storage location includes the N number of preassigned of data storage
Secondary node；Determining module 302 is used for：According to mark and mapping table, the corresponding institute of storage mark is determined
The N number of preassigned secondary node of data is read in request.
Data processing equipment 300 in the present embodiment and aforementioned data processing method be based on same inventive concept
Under two aspects, be described in detail in the implementation process above to method, so this area skill
Art personnel can be according to the structure of the data processing equipment 300 being well understood in the present embodiment described above and reality
Process is applied, succinct for specification, here is just repeated no more.
Based on identical inventive concept, the embodiment of the present invention also provides data processing equipment 400, Fig. 9 for number
According to the structural schematic block diagram of processing equipment 400, including：Bus 401, and it is connected respectively to bus 401
Processing unit 402, memory cell 403, transceiver 404.
Wherein, transceiver 404 is used for carrying out information transfer with client, and for carrying out with memory node
Information transfer.
Memory cell 403 is used for store instruction.
Processing unit 402 is used for executing the instruction of the storage of memory cell 403, so that when the instruction is executed,
Execute following operation：After the write request that transceiver 404 receives that client sends, determine and asked for storage
The primary storage node of the data of write and M secondary node is asked, wherein, write request is comprising being asked to write
The data for entering, M secondary node include N number of preassigned secondary node, and M is just whole more than 1
Number, N is the positive integer less than M；Write in primary storage node by transceiver 404 and asked write
Data；The write success message that primary storage node is returned is received by transceiver 404；By transceiver 404
The data for being asked to write are respectively written into M secondary node；Determination receives N number of preassigned standby
The write success message for being returned with each node in node respectively；Returned to client by transceiver 404
Write success message.
Optionally, in the present embodiment, N number of preassigned secondary node is to store in M secondary node
Performance parameter meets the node of preset requirement.
Optionally, in the present embodiment, processing unit 402 is additionally operable to：Client is being received in transceiver 404
After the read request of transmission, the corresponding N number of preassigned standby section for being asked to read data of storage mark is determined
Point, wherein, the read request packet is containing the mark for being asked reading data；Read by transceiver 404 N number of advance
In the secondary node that specifies, in any node, the corresponding with mark of storage is asked to read data, and by transmitting-receiving
The data of reading are returned to client by machine 404.
Optionally, in the present embodiment, processing unit 402 is being used for：The number that will be read by transceiver 404
According to before returning to client, it is additionally operable to：Determine in N number of preassigned secondary node storage with mark
Corresponding asked read data all same.
Optionally, in the present embodiment, data processing equipment 400 is stored with mapping table, and mapping table preserves number
According to mark and the corresponding relation of data storage location, data storage location includes the N number of advance finger of data storage
Fixed secondary node.Processing unit 402 is used for：Determine that storage mark is corresponding to be asked to read the N number of of data
Preassigned secondary node, including：According to mark and mapping table, determine that storage mark is corresponding and asked
Seek the N number of preassigned secondary node for reading data.
Data processing equipment 400 in the present embodiment and aforementioned data processing method be based on same inventive concept
Under two aspects, be described in detail in the implementation process above to method, so this area skill
Art personnel can be according to the structure of the data processing equipment 400 being well understood in the present embodiment described above and reality
Process is applied, succinct for specification, here is just repeated no more.
It should be noted that bus 401 can be industry standard architecture (English：Industry Standard
Architecture；Referred to as：ISA) bus, external equipment interconnection (English：Peripheral Component；
Referred to as：PCI) bus or extended industry-standard architecture (English：Extended Industry Standard
Architecture；Referred to as：EISA) bus etc..It is total that bus 401 can specifically be divided into address bus, data
Line, controlling bus etc..For ease of representing, only represented with a line in Fig. 9, it is not intended that only one
Bus or a type of bus.
Processing unit 402 can be the general designation of a processor, or multiple treatment elements.For example,
Processing unit 402 can be central processing unit (English：Central Processing Unit；Referred to as：CPU),
Can also be specific integrated circuit (English：Application Specific Intergrated Circuit；Referred to as：
ASIC), or be arranged to implement the embodiment of the present invention one or more integrated circuits, for example：One
Individual or multi-microprocessor (English：digital singnal processor；Referred to as：DSP), or, one or
The multiple field programmable gate arrays of person (English：Field Programmable Gate Array；Referred to as：FPGA).
Memory cell 403 can be the general designation of a storage device, or multiple memory elements, and use
In storage executable program code, access network equipment or parameter, data etc. required for terminal operating.And store
Device can include random access memory (English：Random Access Memory；Referred to as：RAM), also may be used
To include nonvolatile memory (non-volatile memory), such as magnetic disc store, flash memory (Flash)
Deng.
The one or more technical schemes provided in the embodiment of the present invention, at least have the following technical effect that or excellent
Point：
In the embodiment of the present invention, calculate node when data are write according to write request in memory node, to
In primary storage node after write data success, number is all write only in N number of preassigned secondary node
Success message is write according to just returning to client after success, wherein, N number of preassigned secondary node can be
The desired secondary node of user.Therefore, technique scheme is ensure that on the desired secondary node of user
Data are successfully written, meet the demand of user.
Those skilled in the art are it should be appreciated that embodiments of the invention can be provided as method, system or meter
Calculation machine program product.Therefore, the present invention can be using complete hardware embodiment, complete software embodiment or knot
The form of the embodiment in terms of conjunction software and hardware.And, the present invention can be adopted and wherein be wrapped one or more
Computer-usable storage medium containing computer usable program code (including but not limited to magnetic disc store,
CD-ROM, optical memory etc.) the upper computer program that implements form.
The present invention is produced with reference to method according to embodiments of the present invention, equipment (system) and computer program
The flow chart of product and/or block diagram are describing.It should be understood that can by computer program instructions flowchart and
/ or block diagram in each flow process and/or square frame and flow chart and/or the flow process in block diagram and/
Or the combination of square frame.These computer program instructions can be provided to all-purpose computer, special-purpose computer, embedded
The processor of formula processor or other programmable data processing device is to produce a machine so that by calculating
The instruction of the computing device of machine or other programmable data processing device is produced for realizing in flow chart one
The device of the function of specifying in individual flow process or one square frame of multiple flow processs and/or block diagram or multiple square frames.
These computer program instructions may be alternatively stored in and computer or the process of other programmable datas can be guided to set
In the standby computer-readable memory for working in a specific way so that be stored in the computer-readable memory
Instruction produce and include the manufacture of command device, the command device is realized in one flow process or multiple of flow chart
The function of specifying in one square frame of flow process and/or block diagram or multiple square frames.
Although preferred embodiments of the present invention have been described, but those skilled in the art once know base
This creative concept, then can make other change and modification to these embodiments.So, appended right will
Ask and be intended to be construed to include preferred embodiment and fall into being had altered and changing for the scope of the invention.
Obviously, those skilled in the art can carry out various changes and modification without deviating from this to the present invention
Bright spirit and scope.So, if the present invention these modification and modification belong to the claims in the present invention and
Within the scope of its equivalent technologies, then the present invention is also intended to comprising these changes and modification.
Claims (10)
1. a kind of data processing method, it is characterised in that include：
Calculate node receives the write request that client sends, and the write request is comprising the data for being asked to write；
The calculate node determines standby for storing the primary storage node for being asked the data for writing and M
With node, wherein, the M secondary node includes N number of preassigned secondary node, M be more than
1 positive integer, N are the positive integer less than M；
The calculate node writes the data for being asked to write in the primary storage node；
The calculate node receives the write success message that the primary storage node is returned；
The calculate node is respectively written into the data for being asked to write to the M secondary node；
The calculate node determines that receiving each node in N number of preassigned secondary node divides
The write success message not returned；
The calculate node is returned to the client and writes success message.
2. the method for claim 1, it is characterised in that N number of preassigned standby section
Point meets the node of preset requirement for storage performance parameter in the M secondary node.
3. method as claimed in claim 1 or 2, it is characterised in that in the calculate node to described
Client is returned after writing success message, is also included：
The calculate node receives the read request that the client sends, and the read request packet contains asked reading
According to mark；
The calculate node determines the corresponding described N number of advance finger for being asked to read data of the storage mark
Fixed secondary node；And
Read in N number of preassigned secondary node the corresponding with the mark of storage in any node
Asked read data, and by read the data return to the client.
4. method as claimed in claim 3, it is characterised in that in the institute that the calculate node will read
State before data return to the client, also include：
The calculate node determines in N number of preassigned secondary node the right with the mark of storage
That answered is asked reading data all same.
5. the method as described in claim 3 or 4, it is characterised in that the calculate node is stored with and reflects
Firing table, the mapping table preserve the corresponding relation of Data Identification and data storage location, the data storage
Position includes described N number of preassigned secondary node of data storage；
The calculate node determines the corresponding described N number of advance finger for being asked to read data of the storage mark
Fixed secondary node, including：
The calculate node determines the corresponding institute of the storage mark according to the mark and the mapping table
Described N number of preassigned secondary node of data is read in request.
6. a kind of data processing equipment, it is characterised in that include：
Receiver module, for receiving the write request of client transmission, the write request includes is asked write
Data；
Determining module, for determining for storing the primary storage node for being asked the data for writing and M
Secondary node, wherein, the M secondary node includes N number of preassigned secondary node, and M is big
In 1 positive integer, N is the positive integer less than M；
Writing module, is asked, for the write in the primary storage node, the data for writing；
The receiver module is additionally operable to：Receive the write success message of primary storage node return；
Said write module is additionally operable to：The write of primary storage node return is received in the receiver module
After success message, the data for being asked to write are respectively written into the M secondary node；
The determining module is additionally operable to：Determine and receive each in N number of preassigned secondary node
The write success message that node is returned respectively；
Described device also includes：Sending module, described N number of pre- for receiving in determining module determination
After the write success message that each node is returned respectively in the secondary node that first specifies, return to the client
Write-back success message.
7. device as claimed in claim 6, it is characterised in that the determining module is used for：Determine institute
State storage performance parameter in M secondary node N number of node of preset requirement is met for described preassigned
Secondary node.
8. device as claimed in claims 6 or 7, it is characterised in that：
The receiver module is additionally operable to：The read request of client transmission is received, the read request packet contains institute
The mark of data is read in request；
The determining module is additionally operable to：Determine that the storage mark is corresponding to be asked to read the described N number of of data
Preassigned secondary node；
Described device also includes：Read module, appoints for reading in N number of preassigned secondary node
In one node, the corresponding with the mark of storage is asked to read data；
The sending module is additionally operable to：The data for reading are returned to the client.
9. device as claimed in claim 8, it is characterised in that the determining module is additionally operable to：Determine
In N number of preassigned secondary node, the corresponding with the mark of storage is asked reading data homogeneous
With；
The sending module is used for：Determine in N number of preassigned secondary node in the determining module
Storage corresponding with the mark ask reading data all same after, by reading the data return to
The client.
10. device as claimed in claim 8 or 9, it is characterised in that described device is stored with mapping table,
The mapping table preserves the corresponding relation of Data Identification and data storage location, the data storage location bag
Include described N number of preassigned secondary node of data storage；The determining module is used for：According to the mark
Know and the mapping table, determine the corresponding described N number of advance finger for being asked to read data of the storage mark
Fixed secondary node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510547258.8A CN106484311B (en) | 2015-08-31 | 2015-08-31 | A kind of data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510547258.8A CN106484311B (en) | 2015-08-31 | 2015-08-31 | A kind of data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106484311A true CN106484311A (en) | 2017-03-08 |
| CN106484311B CN106484311B (en) | 2019-07-19 |
Family
ID=58236081
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510547258.8A Active CN106484311B (en) | 2015-08-31 | 2015-08-31 | A kind of data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106484311B (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107037987A (en) * | 2017-04-01 | 2017-08-11 | 北京华云网际科技有限公司 | To the treating method and apparatus rolled up in distributed memory system |
| CN107797945A (en) * | 2017-10-31 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of storage system and its date storage method, device, system and equipment |
| CN107908370A (en) * | 2017-11-30 | 2018-04-13 | 新华三技术有限公司 | Date storage method and device |
| CN108462756A (en) * | 2018-03-29 | 2018-08-28 | 新华三技术有限公司 | A kind of method for writing data and device |
| CN110196680A (en) * | 2018-03-27 | 2019-09-03 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN110955382A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Method and device for writing data in distributed system |
| CN111309805A (en) * | 2019-12-13 | 2020-06-19 | 华为技术有限公司 | Data reading and writing method and device for database |
| CN111857603A (en) * | 2020-07-31 | 2020-10-30 | 重庆紫光华山智安科技有限公司 | Data processing method and related device |
| CN112347335A (en) * | 2020-10-30 | 2021-02-09 | 银盛支付服务股份有限公司 | Method, system, computer equipment and storage medium for efficient retrieval of Elasticissearch |
| WO2021052230A1 (en) * | 2019-09-18 | 2021-03-25 | 华为技术有限公司 | Storage system, storage node, and data storage method |
| CN113301084A (en) * | 2020-06-30 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN113360287A (en) * | 2021-06-21 | 2021-09-07 | 上海哔哩哔哩科技有限公司 | Data processing method and device |
| CN114979180A (en) * | 2022-05-24 | 2022-08-30 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1336068A (en) * | 1998-10-20 | 2002-02-13 | 安德鲁·杜根 | An intelligent network |
| CN101895472A (en) * | 2010-07-16 | 2010-11-24 | 华为技术有限公司 | Traffic flow load sharing method and processing method and corresponding device and system |
| CN102546782A (en) * | 2011-12-28 | 2012-07-04 | 北京奇虎科技有限公司 | Distribution system and data operation method thereof |
| CN102594849A (en) * | 2011-01-06 | 2012-07-18 | 阿里巴巴集团控股有限公司 | Data backup and recovery method and device, virtual machine snapshot deleting and rollback method and device |
| CN103856511A (en) * | 2012-11-30 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Data packet upload method, client terminal, nodes, information server and system |
-
2015
- 2015-08-31 CN CN201510547258.8A patent/CN106484311B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1336068A (en) * | 1998-10-20 | 2002-02-13 | 安德鲁·杜根 | An intelligent network |
| CN101895472A (en) * | 2010-07-16 | 2010-11-24 | 华为技术有限公司 | Traffic flow load sharing method and processing method and corresponding device and system |
| CN102594849A (en) * | 2011-01-06 | 2012-07-18 | 阿里巴巴集团控股有限公司 | Data backup and recovery method and device, virtual machine snapshot deleting and rollback method and device |
| CN102546782A (en) * | 2011-12-28 | 2012-07-04 | 北京奇虎科技有限公司 | Distribution system and data operation method thereof |
| CN103856511A (en) * | 2012-11-30 | 2014-06-11 | 腾讯科技（深圳）有限公司 | Data packet upload method, client terminal, nodes, information server and system |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107037987A (en) * | 2017-04-01 | 2017-08-11 | 北京华云网际科技有限公司 | To the treating method and apparatus rolled up in distributed memory system |
| CN107797945A (en) * | 2017-10-31 | 2018-03-13 | 郑州云海信息技术有限公司 | A kind of storage system and its date storage method, device, system and equipment |
| CN107908370A (en) * | 2017-11-30 | 2018-04-13 | 新华三技术有限公司 | Date storage method and device |
| CN107908370B (en) * | 2017-11-30 | 2021-07-06 | 新华三技术有限公司 | Data storage method and device |
| CN110196680A (en) * | 2018-03-27 | 2019-09-03 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN110196680B (en) * | 2018-03-27 | 2021-10-26 | 腾讯科技（深圳）有限公司 | Data processing method, device and storage medium |
| CN108462756B (en) * | 2018-03-29 | 2020-11-06 | 新华三技术有限公司 | Data writing method and device |
| CN108462756A (en) * | 2018-03-29 | 2018-08-28 | 新华三技术有限公司 | A kind of method for writing data and device |
| CN110955382A (en) * | 2018-09-26 | 2020-04-03 | 华为技术有限公司 | Method and device for writing data in distributed system |
| JP2022522123A (en) * | 2019-09-18 | 2022-04-14 | 華為技術有限公司 | Storage system, storage node and data storage method |
| WO2021052230A1 (en) * | 2019-09-18 | 2021-03-25 | 华为技术有限公司 | Storage system, storage node, and data storage method |
| JP7313458B2 (en) | 2019-09-18 | 2023-07-24 | 華為技術有限公司 | Storage system, storage node and data storage method |
| CN111309805A (en) * | 2019-12-13 | 2020-06-19 | 华为技术有限公司 | Data reading and writing method and device for database |
| WO2021114848A1 (en) * | 2019-12-13 | 2021-06-17 | 华为技术有限公司 | Data reading and writing method and device for database |
| CN111309805B (en) * | 2019-12-13 | 2023-10-20 | 华为技术有限公司 | Data reading and writing method and device for database |
| US11868333B2 (en) | 2019-12-13 | 2024-01-09 | Huawei Technologies Co., Ltd. | Data read/write method and apparatus for database |
| CN113301084A (en) * | 2020-06-30 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Data processing method and device |
| CN111857603A (en) * | 2020-07-31 | 2020-10-30 | 重庆紫光华山智安科技有限公司 | Data processing method and related device |
| CN112347335A (en) * | 2020-10-30 | 2021-02-09 | 银盛支付服务股份有限公司 | Method, system, computer equipment and storage medium for efficient retrieval of Elasticissearch |
| CN112347335B (en) * | 2020-10-30 | 2024-06-04 | 银盛支付服务股份有限公司 | Method, system, computer equipment and storage medium for efficient retrieval of elastic search |
| CN113360287A (en) * | 2021-06-21 | 2021-09-07 | 上海哔哩哔哩科技有限公司 | Data processing method and device |
| CN114979180A (en) * | 2022-05-24 | 2022-08-30 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
| CN114979180B (en) * | 2022-05-24 | 2024-05-17 | 超聚变数字技术有限公司 | Data synchronization method, system and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106484311B (en) | 2019-07-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106484311A (en) | A kind of data processing method and device | |
| EP3761170A1 (en) | Virtual machine creation method and apparatus | |
| CN113641457B (en) | Container creation method, device, apparatus, medium, and program product | |
| US8307170B2 (en) | Information processing method and system | |
| CN105468302B (en) | A kind of method, apparatus and system of processing data | |
| CN105516230B (en) | A kind of data processing method and device | |
| CN105224255B (en) | A kind of storage file management method and device | |
| CN110865769B (en) | Method for processing read/write request, network storage system and electronic equipment | |
| CN109491788A (en) | A kind of virtual platform implementation of load balancing and device | |
| US20230266999A1 (en) | Resource scheduling method, resource scheduling system, and device | |
| CN109213695A (en) | Buffer memory management method, storage system and computer program product | |
| CN108845877A (en) | The methods, devices and systems of managing internal memory | |
| US10642817B2 (en) | Index table update method, and device | |
| CN105391755B (en) | Data processing method, apparatus and system in a kind of distributed system | |
| CN105094742B (en) | A kind of method and apparatus for writing data | |
| JP6331944B2 (en) | Information processing apparatus, memory control apparatus, and information processing apparatus control method | |
| CN109062514A (en) | A kind of band width control method based on NameSpace, device and storage medium | |
| JP2022045666A (en) | Resource allocation control device, computing system and resource allocation control method | |
| CN105354153A (en) | Implement method for data exchange and cache of tightly-coupled heterogeneous multi-processor | |
| CN102833295B (en) | Data manipulation method and device in distributed cache system | |
| CN108196954A (en) | A kind of file read/write method, system, equipment and computer storage media | |
| CN107967150A (en) | A kind of thread execution order determines method, apparatus, equipment and storage medium | |
| US8769104B2 (en) | System and method for loading web page using multiple paths in multiple interface circumstances | |
| CN111930315A (en) | Data access method, data access device and storage medium | |
| CN104281587A (en) | Connection establishing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220907 Address after: No. 1899 Xiyuan Avenue, high tech Zone (West District), Chengdu, Sichuan 610041 Patentee after: Chengdu Huawei Technologies Co.,Ltd. Address before: 611731 Qingshui River District, Chengdu hi tech Zone, Sichuan, China Patentee before: HUAWEI DIGITAL TECHNOLOGIES (CHENG DU) Co.,Ltd. |