CN107491975B - Data slot data processing method and device for server and consumer - Google Patents

Data slot data processing method and device for server and consumer Download PDF

Info

Publication number
CN107491975B
CN107491975B CN201610411103.6A CN201610411103A CN107491975B CN 107491975 B CN107491975 B CN 107491975B CN 201610411103 A CN201610411103 A CN 201610411103A CN 107491975 B CN107491975 B CN 107491975B
Authority
CN
China
Prior art keywords
data
data slot
information
consumer
slot
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610411103.6A
Other languages
Chinese (zh)
Other versions
CN107491975A (en
Inventor
周洲
孙廷韬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201610411103.6A priority Critical patent/CN107491975B/en
Publication of CN107491975A publication Critical patent/CN107491975A/en
Application granted granted Critical
Publication of CN107491975B publication Critical patent/CN107491975B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/02—Marketing; Price estimation or determination; Fundraising
    • G06Q30/0201—Market modelling; Market analysis; Collecting market data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types

Abstract

The application provides a data slot data processing method for a server, which comprises the following steps: receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer. The application also provides a data slot data processing method for consumers. The method and the device can solve the problem of automatic processing of data sequential consumption on the data slot with ancestral relation.

Description

Data slot data processing method and device for server and consumer
Technical Field
The application relates to the technical field of data processing, in particular to a data slot data processing method and device for a server and a data slot data processing method and device for a consumer.
Background
A typical application scenario for data processing of data slots is to have a data warehouse, in which a plurality of data slots are provided for data producers and consumers. Each data slot has the following characteristics:
1) the data slot has two states of read-only and read-write. Read-only means that the data slot does not receive write data, but data can be read from the data slot; the reading and writing indicates that the data slot is readable and writable;
2) each data slot is associated with a set of hash values. Initially, all data slots are in a read-write state, and hash value sets of any data slot are not intersected;
3) the data slots may have merge actions. For example, merging (data slot 1, data slot 2) means that a data slot in read-write state is newly created, and the hash value set associated with the newly created slot is the union of the data slot 1 and the data slot 2; after the merge is complete, data slot 1 and data slot 2 will become read-only;
4) the data slots may be subject to a splitting action. For example, division (data slot 1) means dividing the data slot 1 into two data slots in read/write states; the first newly partitioned data slot will be partially cut from the associated hash value set of data slot 1 as its own hash value set, the remaining hash value set will be assigned to another newly partitioned data slot, and data slot 1 will become read-only.
In actual practice, upstream of the data slot is the data producer. A batch of data producers will produce data continuously and plug the data into a particular data slot according to their hash values. Specifically, writing a particular data slot must satisfy the following two factors: this data slot is read-write; the hash value set associated with this data slot contains the hash value of the data.
Due to the above-mentioned characteristics of data slots and data writing, each piece of data can only be written into a unique data slot; furthermore, each piece of data has its position information in a specific data slot, and the piece of data can be read from the specific data slot according to the position information.
In actual practice, downstream of the data slot are several consumers. Each consumer holds a portion of the data slots and consumes the data in those data slots. These consumers must satisfy the following consumption rules:
1) data slots held by different consumers are disjoint;
2) sharing the data slots by all consumers, and ensuring that the absolute value of the difference value of the number of the data slots held by any two consumers is less than or equal to 1;
3) the data of the same hash value must be sequentially consumed.
In practical applications, in the process of cooperatively consuming data of data slots in a data warehouse by multiple consumers, the following situations may occur:
1) merging or dividing the data slots to generate new slots in read-write state, and changing the original data slots into read-only state;
2) the number of consumers changes dynamically, e.g., new consumers join, existing consumers withdraw;
3) the data in the read-only data slot is consumed, and the data slot is no longer held by the consumer.
When the above situation occurs, how to ensure that the consumer still satisfies the above consumption rules is a problem to be solved urgently. The general solutions are basically based on data slots to provide consumer cooperative consumption, and these solutions usually only provide the most basic cooperative consumption scheme, and cannot meet the demand of data sequential consumption. Particularly, when the data slots are merged or divided, the ancestor data slots are changed into read-only attributes due to the ancestor relationship among the data slots, the newly generated data slots are read-write attributes, and the data which is written into the original data slots according to the hash value can be written into the newly generated data slots; if in-order consumption is to be satisfied, it is required that the new data slot should not be consumed until the original data slot is consumed. The existing technical scheme cannot process the data slot with ancestral relation and cannot solve the problem of data sequential consumption.
Disclosure of Invention
The application provides a data slot data processing method and device for a server and a data slot data processing method and device for consumers, and aims to solve the problem that a plurality of consumers consume data of data slots in sequence.
The application provides a data slot data processing method for a server, which comprises the following steps:
receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer.
Optionally, the reporting information further includes: the consumption breakpoint updating information and the consumption breakpoint acquiring information; the consumption breakpoint update information includes: identification of a data warehouse, identification of a data slot and consumption breakpoint position information; the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
Optionally, the heartbeat information is received at fixed time intervals.
Optionally, the heartbeat information further includes an identifier of a data warehouse to which the data slot held by the consumer belongs.
Optionally, the consumer and data slot status database includes: the data slot management method comprises the following steps of information of a consumer, physical information of the data slot, ancestor relation information of the data slot, data state information of the data slot, consumption breakpoint position information of the data slot and correlation information of the consumer, the data slot and the data state of the data slot.
Optionally, the data slot data processing logic of the server further includes:
if the report information comprises the consumption breakpoint updating information, updating the consumption breakpoint position information of the data slot in the consumer and data slot state database according to the consumption breakpoint updating information, and returning the information whether the consumption breakpoint position information is successfully updated;
and if the report information comprises the consumption breakpoint acquisition information, returning the consumption breakpoint position information of the data slot read from the consumer and the data slot state database.
Optionally, the data state information of the data slot includes: unallocated, allocated, waiting, transferred, and done states; wherein the unallocated state indicates that the data slot is not held by the consumer and is waiting to be allocated; the allocated status, indicating that the data slot has been held and consumed by the consumer; the waiting state represents that the data slot cannot be consumed currently, and waits for the ancestor data slot of the data slot to finish consumption; the transition state represents that the data slot is in an excessive state for transition to another consumer for consumption; the done state, indicating that the data of the data slot has been consumed and is not held by the consumer.
Optionally, the data slot data processing logic of the server further includes:
if a newly added data slot appears, judging whether the data slot has ancestor relation with each data slot in the same data warehouse; if not, setting the data state information of the data slot to be in an unallocated state; if yes, judging whether the data of the data slot is consumed; if so, setting the data state information of the data slot to be in an unallocated state; and if not, setting the data state information of the data slot to be in a waiting state.
Optionally, the data slot data processing logic of the server further includes:
if the data of the ancestor data slot of the data slot in the wait state has been consumed, the data state information of the data slot is changed to the unallocated state.
Optionally, the data slot data processing logic of the server further includes:
if a data slot in an unallocated state is allocated to a consumer, the data state information of the data slot is changed to an allocated state, and the consumer is associated with the data slot.
Optionally, the data slot data processing logic of the server further includes:
if the difference value between the time when the consumer in the distributed state receives the heartbeat information last time and the current time is larger than a preset threshold value, recovering the data slot held by the consumer, changing the data state information of the data slot into the non-distributed state, and canceling the association between the consumer and the data slot.
Optionally, the data slot data processing logic of the server further includes:
if the difference between the number of data slots held by the consumer holding the most data slots and the number of data slots held by the consumer holding the least data slots is greater than 1, changing the data state information of one data slot held by the consumer holding the most data slots from the allocated state to the transition state, removing the data slot changed to the transition state from the identification of the data slot returned to the consumer holding the most data slots, and associating the data slot changed to the transition state with the consumer holding the least data slots.
Optionally, the data slot data processing logic of the server further includes:
and if the received heartbeat information indicates that the consumer abandons the held data slot with the transferred state, adding the data slot into the identification of the data slot returned to the associated consumer, and changing the data state information of the data slot from the transferred state to the distributed state.
Optionally, the data slot data processing logic of the server further includes:
if the difference value between the last heartbeat information receiving time of the consumer associated with the data slot in the transition state and the current time is larger than the preset threshold value, the consumer holding the data slot continues to hold the data slot, the data state of the data slot is changed into the distributed state, and the data slot is associated with the consumer holding the data slot.
Optionally, the data slot data processing logic of the server further includes:
and if the consumption breakpoint updating information of the data slot in the finished state indicates that the updated consumption breakpoint position is more ahead than the last consumption end position, changing the data state information of the data slot into the unallocated state.
Optionally, the data slot data processing logic of the server further includes:
and if the data of the data slot with the read-only attribute is consumed, changing the data state information of the data slot into a finished state, and canceling the association between the data slot and the consumer holding the data slot.
Optionally, the data slot data processing logic of the server further includes:
and if the difference value between the time when the consumer with the data slot in the transition state receives the heartbeat information last time and the current time is larger than a preset threshold value, canceling the association between the data slot and the consumer, and changing the data state information of the data slot into an unallocated state.
The present application also provides a data slot data processing apparatus for a server, comprising:
the report receiving information unit is used for receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
the execution data processing unit is used for triggering a data slot data processing logic unit of the server according to the received report information and the maintained related information of the consumer and the data slot state database so as to update the consumer and the data slot state database and return response information corresponding to the report information to the consumer; the data slot data processing logic unit of the server at least comprises: a return heartbeat unit to return an identification of a data slot intended to be held by the consumer.
Optionally, the data slot data processing logic unit of the server further includes:
a consumption breakpoint updating information returning unit, configured to update, in the consumer and data slot state database, consumption breakpoint position information of the data slot according to the consumption breakpoint updating information if the report information includes the consumption breakpoint updating information, and return information indicating whether the consumption breakpoint position information is successfully updated;
and the consumption breakpoint return information obtaining unit is used for returning the consumption breakpoint position information of the data slot read from the consumer and the data slot state database if the report information comprises the consumption breakpoint obtaining information.
Optionally, the data slot data processing logic unit of the server further includes:
introducing a new data slot unit for judging whether the data slot has ancestor relation with each data slot in the same data warehouse if a newly added data slot appears; if not, setting the data state information of the data slot to be in an unallocated state; if yes, judging whether the data of the data slot is consumed; if so, setting the data state information of the data slot to be in an unallocated state; and if not, setting the data state information of the data slot to be in a waiting state.
Optionally, the data slot data processing logic unit of the server further includes:
and the waiting change unallocated unit is used for changing the data state information of the data slot into an unallocated state if the data of the ancestor data slot of the data slot in the waiting state is consumed up.
Optionally, the data slot data processing logic unit of the server further includes:
and an unallocated change allocated unit configured to change the data state information of the data slot to an allocated state and associate the consumer with the data slot, if the data slot in an unallocated state is allocated to a consumer.
Optionally, the data slot data processing logic unit of the server further includes:
and the distributed change unallocated unit is used for recovering the data slot held by the consumer if the difference value between the time when the consumer in the distributed state receives the heartbeat information last time and the current time is greater than a preset threshold value, changing the data state information of the data slot into the unallocated state, and canceling the association between the consumer and the data slot.
Optionally, the data slot data processing logic unit of the server further includes:
and the data slot load balancing unit is used for changing the data state information of one data slot held by the consumer holding the most data slots into a transition state from the distributed state if the difference value between the number of the data slots held by the consumer holding the most data slots and the number of the data slots held by the consumer holding the least data slots is larger than 1, removing the data slot changed into the transition state from the identification of the data slot returned to the consumer holding the most data slots, and associating the data slot changed into the transition state with the consumer waiting for consumption.
Optionally, the data slot data processing logic unit of the server further includes:
and the transfer change distributed unit is used for adding the data slot into the identifier of the data slot returned to the associated consumer if the received heartbeat information indicates that the consumer abandons the held data slot in the transfer state, and changing the data state information of the data slot from the transfer state to the distributed state.
Optionally, the data slot data processing logic unit of the server further includes:
and the transition timeout changing unit is used for continuing holding the data slot by the consumer holding the data slot if the difference value between the last heartbeat information receiving time of the consumer associated with the data slot in the transition state and the current heartbeat information receiving time is greater than a preset threshold value, changing the data state of the data slot into the distributed state, and associating the data slot with the consumer holding the data slot.
Optionally, the data slot data processing logic unit of the server further includes:
and the change-over-completion unallocated unit is used for changing the data state information of the data slot into an unallocated state if the consumption breakpoint update information of the data slot in the received completion state indicates that the updated consumption breakpoint position is more ahead than the last consumption end position.
Optionally, the data slot data processing logic unit of the server further includes:
and the change finishing unit is used for changing the data state information of the data slot into a finishing state and canceling the association between the data slot and the consumer holding the data slot if the data of the data slot with the read-only attribute is consumed.
Optionally, the data slot data processing logic unit of the server further includes:
and the transfer change non-distribution unit is used for canceling the association between the data slot and the consumer and changing the data state information of the data slot into a non-distribution state if the difference value between the time when the consumer with the data slot in the transfer state receives the heartbeat information last time and the current time is greater than a preset threshold value.
The present application further provides an electronic device for processing data of a data slot of a server, comprising:
a display;
a processor;
a memory for storing a program of data slot data for a server, the program when read executed by the processor performing the following: receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer.
The application provides a data slot data processing method for a consumer, which comprises the following steps:
sending report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
receiving response information returned by the server;
executing data slot data processing logic of a consumer according to the data slot consumption requirement, the held consumption information of the data slot and the response information received from the server so as to consume the held data slot data and generate corresponding report information;
and returning to execute the step of sending the report information to the server.
Optionally, the data slot data processing logic of the consumer further includes:
if the report information includes consumption breakpoint update information, the consumption breakpoint update information includes: the data warehouse identification, the data slot identification and the consumption breakpoint position information of the data slot;
if the report information includes the consumption breakpoint acquisition information, the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
Optionally, the data slot data processing logic of the consumer further includes:
and if the received response information shows that the identification set of the data slot which the server wishes to hold is smaller than the identification set of the data slot which the server actually holds, abandoning the consumption of the data slot which is actually held and the data slot which is supposed to hold, and generating heartbeat information of the identification of the data slot which is actually held and consumption breakpoint updating information of the data slot which is actually held and hopefully held.
Optionally, the data slot data processing logic of the consumer further includes:
and if the received response information shows that the identification set of the data slot which the server wishes to hold is larger than the identification set of the data slot which is actually held, writing consumption breakpoint request information of the data slot which is hoped to hold and is actually held into the report information.
Optionally, the data slot data processing logic of the consumer further includes:
if the received response message indicates that the server wishes to have the same set of identifications for the data slots as the set of identifications for the data slots actually held, then the held data slots continue to be consumed.
Optionally, the data slot data processing logic of the consumer further includes:
and if the received response information comprises that the consumption breakpoint of the data slot is not successful, rewriting the consumption breakpoint updating information of the data slot into the report information.
The present application also provides a data slot data processing apparatus for a consumer, comprising:
a report information sending unit, configured to send report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
a response information receiving unit, configured to receive response information returned by the server;
the execution data processing unit is used for triggering the data slot data processing logic unit of the consumer according to the data slot consumption requirement, the held data slot consumption information and the response information received from the server so as to consume the held data slot data and generate corresponding report information;
and the return triggering report sending information unit is used for returning and triggering the report sending information unit.
Optionally, the data slot data processing logic unit of the consumer further includes:
a consumption breakpoint updating unit, configured to, if the report information includes consumption breakpoint updating information, the consumption breakpoint updating information includes: the data warehouse identification, the data slot identification and the consumption breakpoint position information of the data slot;
a consumption breakpoint acquisition unit, configured to, if the report information includes the consumption breakpoint acquisition information, the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
Optionally, the data slot data processing logic unit of the consumer further includes:
and the excess load balancing unit is used for giving up consumption of the difference set data slot which is actually held and hoped to hold if the received response information shows that the identification set of the data slot which the server hopes to hold is smaller than the identification set of the data slot which is actually held, and generating heartbeat information of removing the identification of the difference set data slot and consumption breakpoint updating information of the difference set data slot.
Optionally, the data slot data processing logic unit of the consumer further includes:
and the load balancing unit is used for writing consumption breakpoint request information of a data slot which is expected to hold and a difference set data slot which is actually held into the report information if the received response information shows that the identification set of the data slot which is expected to hold by the server is larger than the identification set of the data slot which is actually held.
Optionally, the data slot data processing logic unit of the consumer further includes:
and the same load balancing unit is used for continuing consuming the held data slots if the received response information indicates that the identification set of the data slots which the server wishes to hold is the same as the identification set of the data slots which are actually held.
Optionally, the data slot data processing logic unit of the consumer further includes:
and the re-breakpoint updating unit is used for rewriting the consumption breakpoint updating information of the data slot into the report information if the received response information includes that the consumption breakpoint of the data slot is not successful.
The present application further provides a data slot data processing electronic device for a consumer, comprising:
a display;
a processor;
a memory for storing a program of data slot data for a consumer, the program when read executed by the processor performing the following: sending report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; receiving response information returned by the server; executing data slot data processing logic of a consumer according to the data slot consumption requirement, the held consumption information of the data slot and the response information received from the server so as to consume the held data slot data and generate corresponding report information; and returning to execute the step of sending the report information to the server.
Compared with the prior art, the method has the following advantages:
according to the method and the system, after the data slots are merged or divided and ancestral relations are generated among the data slots, in the process that a plurality of consumers consume data of the data slots in the data warehouse in a coordinated mode, corresponding data slot data processing logics are executed through information interaction between the server and the consumers, and the problem of data sequential consumption is automatically processed.
The method and the system can also implement a strategy considering load balancing at the server end, and perform targeted consumption load management on consumers.
Drawings
Fig. 1 is a flowchart of a data slot data processing method for a server according to a first embodiment of the present application;
FIG. 2 is a schematic diagram of data slot ancestry relationships and data write hash value order as provided in the first embodiment of the present application;
FIG. 3 is a schematic diagram illustrating a data state change of a data slot according to a first embodiment of the present application;
FIG. 4 is a diagram of a data slot data processing apparatus for a server according to a second embodiment of the present application;
FIG. 5 is a schematic diagram of data slot data processing electronics for a server according to a third embodiment of the present application;
FIG. 6 is a flow chart of a data slot data processing method for a consumer according to a fourth embodiment of the present application;
FIG. 7 is a schematic diagram of a data sink data processing apparatus for a consumer according to a fifth embodiment of the present application;
FIG. 8 is a schematic diagram of a data slot data processing electronic device for a consumer according to a sixth embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
The application provides a data slot data processing method for a server, a data slot data processing device for a server, a data slot data processing electronic device for a server, a data slot data processing method for a consumer, a data slot data processing device for a consumer and a data slot data processing electronic device for a consumer. The following describes embodiments of the present application in detail with reference to the drawings in turn.
Please refer to fig. 1, which is a flowchart illustrating a data slot data processing method for a server according to a first embodiment of the present application. The data slot data processing method for the server comprises the following steps:
step S101: receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer.
Heartbeat information sent by a consumer including at least an identification of a data slot currently held by the consumer is received. The step triggers the server to carry out follow-up actions and provides information basis for the follow-up actions of the server.
The server receives the report information sent from each consumer. Because the identification of the data slots in the same data warehouse cannot be the same, but the identification of the data slots in different data warehouses can be the same, pointing to a specific data slot necessarily requires to know the data warehouse to which the data slot belongs; moreover, in the case that the consumer consumes the data slot, the end position of the last consumption data slot, that is, the consumption breakpoint position, needs to be known, so that the consumer can continue to consume data backwards from the consumption breakpoint position.
The server receives report information sent by the consumer, including heartbeat information, consumption breakpoint updating information and consumption breakpoint acquisition information.
The heartbeat information is sent by the consumer at regular intervals. The heartbeat information includes an identification of the data slot currently held by the consumer, and in the case of multiple data warehouses, an identification of the data warehouse to which the data slot held by the consumer belongs. The server acquires the data slot currently held by the consumer by receiving the heartbeat information, and judges whether the consumer is normal or not by receiving the time interval of the heartbeat information.
Consuming breakpoint update information, comprising: the data warehouse comprises an identifier of a data warehouse to which the data slot belongs, an identifier of the data slot held and consumed by a consumer and specific consumption breakpoint position information of the specific data slot. The server acquires the accurate condition of the specific consumption breakpoint position of the consumer data consumption by receiving the consumption breakpoint updating information.
Consuming breakpoint acquisition information, comprising: an identification of the data warehouse to which the data slot belongs and an identification of the data slot. The server acquires the information by receiving the consumption breakpoint, and acquires the specific position of the consumption breakpoint which is consumed last time by the consumer when the consumer requests to acquire the data slot.
For example, in the present embodiment, suppose there are three consumers A, B and C, consuming seven data slots numbered 0 to 6 in one data warehouse X, respectively, and the hash value relationship between the data slots is as shown in fig. 2; where A holds data slots No. 0, 1, and 5, B holds data slots No. 2 and 3, and C holds data slots No. 4 and 6. The server receives heartbeat information sent by a consumer A; in this heartbeat information, the identification of the data slot currently held by consumer A, i.e., numbers 0, 1, and 5, is included.
Step S103: executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer.
Data slot data processing logic of the server that updates the relevant information of the consumer and data slot status databases and returns response information corresponding to the report information to the consumer is performed based on the received report information sent from a particular consumer and the relevant information of the consumer and data slot status databases maintained by the server itself. The processing logic executed includes at least: an identification of a data slot intended to be held by a particular consumer is returned for the particular heartbeat information of the consumer.
The consumer and data slot state database, maintained by the server itself, includes: the data slot management method comprises the following steps of information of a consumer, physical information of the data slot, ancestor relation information of the data slot, data state information of the data slot, consumption breakpoint position information of the data slot and correlation information of the consumer, the data slot and the data state of the data slot.
The information of the consumers, namely the information of all the consumers managed by the server, performing information interaction with the server and performing data consumption of the data slot according to the distribution arrangement of the server, comprises the identification of each consumer, the time of receiving the heartbeat information sent by the consumer last time, the time interval threshold value preset by the server for sending the heartbeat information by the consumer and the like.
The physical information of the data slot includes physical condition information of all data slots managed by the server, including data warehouse information to which the data slot belongs, specific data size and capacity of the data slot, whether the data slot is working normally, and the like.
The data slot ancestor relationship information comprises mutual ancestor relationship among all data slots in the data warehouse and information that the data slots are data read-only or data read-write attributes. For example, in FIG. 2, data slot number 0 is the ancestor of data slots 3, 4, and 6, data slots 1 and 2 are the ancestor of data slots 5 and 6, and data slots 4 and 5 are the ancestor of data slot 6; data slots 0, 1, 2, 4 and 5 are data read-only attributes, and data slots 3 and 6 are data read-write attributes.
Data state information of the data slot, including unallocated, allocated, waiting, transferring and finishing states; wherein the unallocated state indicates that the data slot is not held by the consumer and is waiting to be allocated; the allocated status, indicating that the data slot has been held and consumed by the consumer; the waiting state represents that the data slot cannot be consumed currently, and waits for the ancestor data slot of the data slot to finish consumption; the transition state represents that the data slot is in a transition state for transition to another consumer for consumption; the done state indicates that the data of the data slot has been consumed.
Here, the waiting state of the data slot and the problem that the consumer consumes the data in order need to be described in detail. Assuming that there are only 3 data slots 0, 1 and 2 in the data slot evolution relationship shown in fig. 2 at the beginning, and the interval below each data slot in fig. 2 represents the set of hash values associated with the data slot, where for the sake of simplicity, the value of the hash value is represented by a numerical type, data with the hash value of 7 will be written into data slot 0, and then data slot 0 is divided into data slots 3 and 4, data slot 0 becomes a read-only state, and data slots 3 and 4 become a read-write state, at which time the data with the hash value of 7 will not be written into data slot 0 but will be written into data slot 4; after data slots 4 and 5 are merged into data slot 6, data having a hash value of 7 is written to data slot 6 only.
If a consumer were to sequentially consume data having a hash value of 7, it would be necessary to ensure that data slot 4 should not be consumed by any consumer until data slot 0 is consumed; similarly, data slot 6 should not be consumed before the data in data slot 4 is consumed, data slots 0 and 4 are considered ancestors of data slot 6, and data slot 6 is referred to as descendants of data slots 0 and 4. A particular data slot may be consumed if and only if the data in its ancestor data slot is consumed. For this reason, data slots in a wait state may not be consumed.
And the consumption breakpoint position information of the data slot comprises the position information of the stop of the data consumed by the consumer on the data slot after the last time the data slot is consumed by the consumer. For example, in this example, in data slot number 1, there is a total of 1024KB of data, and Consumer A has consumed the last time data to the location of 800 KB.
The association information of the consumer and the data slot and the data state of the data slot includes the association correspondence relationship between the consumer and the data slot, i.e. the relationship between the consumer and the held data slot, and the relationship between the data slot and the data state of the data slot. For example, in this example, the association of data slot number 1 and Consumer A represents a holding relationship between them, and the data state of data slot number 1 is the allocated state.
The server executes the data processing logic of the data slot of the server by taking the maintained information of the consumer and the data slot state database and the interaction information between the consumer and the server as input parameters, and essentially maintains a finite state machine with five states of unallocated, allocated, waiting, transferring and finishing for each data slot. The server makes explicit the holding relationship of the consumer and the data slot, the data state of the data slot and sends the corresponding instruction of holding and consuming the data slot to the consumer through the predetermined data slot data processing logic.
The data slot data processing logic of the server executed in this step at least includes: returning the identification of the data slot that the particular consumer wishes to have for the particular heartbeat information of the particular consumer, i.e., for the heartbeat information, the server needs to return to the consumer information that the server explicitly wishes the data slot that the consumer possesses. For example, in the present embodiment, the identifications of the data slots currently held by Consumer A, i.e., data slots Nos. 0, 1, and 5, are received, and assuming that the server, after executing the processing logic, recognizes that the consumer continues to hold these three data slots, consumer A is returned with data slot identifications 0, 1, and 5, indicating that it wishes to have data slot Nos. 0, 1, and 5 held by Consumer A.
The data slot data processing logic of the server covers all the specific rules required by the server to perform data slot data processing, which are respectively described in the following preferred steps of the data slot data processing logic of this embodiment. These preferred steps include:
step S103-1: if the report information comprises the consumption breakpoint updating information, updating the consumption breakpoint position information of the data slot in the consumer and data slot state database according to the consumption breakpoint updating information, and returning the information whether the consumption breakpoint position information is successfully updated; and if the report information comprises the consumption breakpoint acquisition information, returning the consumption breakpoint position information of the data slot read from the consumer and the data slot state database.
The information interaction between the server and the consumer comprises receiving the heartbeat information, returning the mark of the data slot which the server hopes the consumer to hold, and receiving the consumption breakpoint updating information and the consumption breakpoint acquisition information.
If the server receives the consumption breakpoint updating information, the server indicates that the consumer performs data consumption on a specific data slot and reports the specific breakpoint position where the last consumption is stopped to the server, the server updates the consumption breakpoint position information of the corresponding data slot in the consumer and data slot state database according to the consumption breakpoint updating information, and returns information whether the consumption breakpoint position information is successfully updated to the consumer after the consumption breakpoint position information of the data slot is updated.
If the server receives the consumption breakpoint acquisition information, the consumption breakpoint position of the data slot is indicated to be ready to consume a certain data slot but needs to be known before consumption, namely the consumption breakpoint position of the data slot is requested to be acquired by the consumer, and the server sends the consumption breakpoint position information of the corresponding data slot read from the consumer and the data slot state database to the consumer.
The optimal selection step updates the breakpoint position of the data slot consumed by the consumer and returns the specific breakpoint position of the data slot to the consumer aiming at the consumption breakpoint updating information and the consumption breakpoint acquisition information; and in addition, the data slot information which is expected to be held by the consumer and is returned to the consumer aiming at the data slot holding information fixedly reported by the consumer in the S103, and the three interaction modes form a complete information interaction channel between the server and the consumer.
Please refer to fig. 3, which is a schematic diagram illustrating a data state change of a data slot according to a first embodiment of the present application. In the present application, the server maintains its data state information for each data slot. In FIG. 3, N indicates unallocated, A indicates allocated, W indicates waiting, T indicates branching, and O indicates completed; the sequence numbers 1 to 9 respectively represent the specific data state transition relations caused by the server executing the corresponding actions according to the data slot data processing logic. Preferred steps S103-3 to S103-XX described below are described with reference to FIG. 3.
Step S103-3: if a newly added data slot appears, judging whether the data slot has ancestor relation with each data slot in the same data warehouse; if not, setting the data state information of the data slot to be in an unallocated state; if yes, judging whether the data of the data slot is consumed; if so, setting the data state information of the data slot to be in an unallocated state; and if not, setting the data state information of the data slot to be in a waiting state.
This preferred step is the situation shown by the number 1 in fig. 3. If the server learns that the newly added data slot exists, checking whether the newly added data slot has an ancestor relationship with the existing data slot in the same data warehouse; if the ancestor relationship does not exist, the newly added data slot is not related to the existing data slot, and the data state information is set to be in an unallocated state for the data slot with the consumption requirement to use; if the ancestor relationship exists, the newly added data slot is related to the existing data slot, and whether the data of the ancestor data slot of the newly added data slot is consumed or not needs to be further checked; if the consumption is finished, setting the data state information into an unallocated state for the data slot with the consumption requirement to use; and if the consumption is not finished, setting the data state information of the data slot into a waiting state.
For example, in this embodiment, it is assumed that No. 7 data slot is newly added to the data warehouse for consumption by the consumer, it is determined that No. 7 data slot newly added has no ancestor relationship with the existing data slot, and if no, the No. 7 data slot newly added is directly set to an unallocated state; if so, judging whether the data of the ancestor data slot of the newly added No. 7 data slot is consumed or not, for example, the ancestor of the No. 7 data slot is the No. 2 data slot, judging whether the data of the No. 2 data slot is consumed or not, and if not, setting the newly added No. 7 data slot in a waiting state.
Step S103-5: if the data of the ancestor data slot of the data slot in the wait state has been consumed, the data state information of the data slot is changed to the unallocated state.
This preferred step is the situation shown as number 2 in fig. 3. If the data of the ancestor data slot of the data slot in the waiting state is consumed, for example, the data of the ancestor data slot is consumed by updating the breakpoint position updating information of the ancestor data slot to the end of the data slot by the consumer, the data state of the ancestor data slot is changed into the unallocated state for the consumer with the demand to consume.
For example, in this embodiment, if the data status is the waiting data slot No. 7 and the data of the ancestor data slot No. 2 has been consumed by the consumer B, the data slot No. 7 is set to the unallocated status.
Step S103-7: if a data slot in an unallocated state is allocated to a consumer, the data state information of the data slot is changed to an allocated state, and the consumer is associated with the data slot.
This preferred step is the situation shown by the number 3 in fig. 3. If the data slot in the unallocated state is allocated to a consumer, the data state information of the data slot is changed into the allocated state, and the allocated consumer and the allocated data slot are associated, namely, the relationship that the allocated consumer holds the allocated data slot is recorded.
For example, in the present embodiment, data slot No. 7 is assigned to be held by consumer C for consumption, the data status information of data slot No. 7 is changed to the assigned status, and data slot No. 7 is associated with consumer C.
Step S103-9: if the difference value between the last heartbeat information of the consumer consuming the distributed data slot and the current heartbeat information is larger than a preset threshold value, the data slot held by the consumer is recovered, the data state information of the data slot is changed into a non-distributed state, and the association between the consumer and the data slot is cancelled.
This preferred step is the situation shown by the number 4 in fig. 3. The server sets the standard of the heartbeat timeout of the consumer as a preset time interval threshold, the size of the threshold needs to be set according to the actual situation, and the smaller the setting, the faster the reaction time of the interaction between the server and the consumer. If the difference value between the last heartbeat information received by the consumer and the current heartbeat information is larger than the preset threshold value, namely the heartbeat information of the consumer is overtime, the data slots held by the consumer are recycled, the distributed state of the data slots is changed into the non-distributed state, and the association between the consumer and the data slots is cancelled.
For example, in the present embodiment, if the heartbeat information of consumer B times out, data slots No. 2 and 3 held by consumer B are collected, the data state of data slot No. 2 and 3 is changed from allocated to unallocated, and the association between consumer B and data slot No. 2 and 3 is cancelled.
Step S103-11: if the difference between the number of data slots held by the consumer holding the most data slots and the number of data slots held by the consumer holding the least data slots is greater than 1, changing the data state information of one data slot held by the consumer holding the most data slots from the allocated state to the transition state, removing the data slot changed to the transition state from the identification of the data slot returned to the consumer holding the most data slots, and associating the data slot changed to the transition state with the consumer holding the least data slots.
This preferred step triggers the load balancing logic of the data slot, which is the situation shown by the sequence number 5 in fig. 3. If the difference between the number of data slots held by the consumer holding the most data slots and the number of data slots held by the consumer holding the least data slots is greater than 1, the preferred method selects one of the data slots held by the consumer holding the most data slots, changes the data state information of the selected data slot from the assigned data slot to a transition state, removes the data slot changed to the transition state from the identification of the data slot returned to the consumer holding the most data slots, and associates the data slot changed to the transition state with the consumer holding the least data slots.
For example, assuming that data slot No. 3 held by consumer B fails and cannot be serviced, consumer B holds only data slot No. 1 and has a difference of 2 from 3 data slots (0, 1, and 5) held by consumer a greater than 1, selects one data slot held by consumer a, assumes that data slot No. 1 is selected, changes the status information of data slot No. 1 from assigned to transferred status, removes data slot No. 1 from the identification of the data slot returned to consumer a, and associates data slot No. 1 with consumer B holding the fewest data slots.
When the triggering condition of the preferred step is met, the preferred step can be executed each time to remove one data slot from the data slots held by the consumers holding the most data slots and associate the removed data slot with the consumer holding the least data slot; the preferred steps may also be performed once per each time to remove a plurality of data slots for a plurality of consumers from among the data slots held by the consumers having the most holding data slots and associate the removed plurality of data slots with the consumers having the least holding data slots. In this way, the load balancing of the data slots is performed by transferring one data slot from the consumer with the most data slots to the consumer with the least data slots, so that the probability of repeated data slot transfer can be reduced, and the efficiency and stability after load balancing can be maintained as much as possible.
For example, in this embodiment, assuming that consumers a and B respectively hold 4 data slots, i.e., 0 to 3 and 4 to 7, and newly join consumers C and D at this time, the difference between the number of held data slots of the consumer a and B that has the most data slots and the number of held data slots of the consumers C and D that have the least data slots is 4 or more than 1, then the preferred step is executed once, and then 2 data slots are transferred to consumers C and D in batch for consumers a and B; then the number of the most data slots held by the consumer (3 data slots held by consumers A and B) is still greater than 1 than the number of the least data slots held (1 data slot held by consumers C and D), and the preferred step is performed again, i.e., the balance requirement that the difference between the number of the most and least data slots to reach the consumer is not greater than 1. The final possible equalization results may not be unique, e.g., one final result is consumer A holding data slots numbers 0 and 1, consumer B holding data slots numbers 4 and 5, consumer C holding data slots numbers 2 and 6, and consumer D holding data slots numbers 3 and 7.
Step S103-13: and if the received heartbeat information indicates that the consumer abandons the held data slot with the transferred state, adding the data slot into the identification of the data slot returned to the associated consumer, and changing the data state information of the data slot from the transferred state to the distributed state.
The preferred procedure is the case shown by reference number 6 in figure 3. If receiving heartbeat information of a consumer, indicating that the consumer abandons the data slot with the transition state held by the consumer, adding the data slot with the transition state into the identification of the data slot returned to the associated consumer, and changing the data state information of the data slot with the transition state from the transition state to the distributed state.
For example, in this embodiment, if no 1 in the identifier of the data slot in the heartbeat message sent by consumer a is received, indicating that consumer a abandons the data slot No. 1 holding the transition state, consumer B transfers the data slot No. 1 to the associated consumer B for holding, adds the data slot No. 1 to the identifier of the data slot returned to associated consumer B, and changes the data state information of the data slot No. 1 from the transition state to the allocated state.
Step S103-15: if the difference value between the last heartbeat information receiving time of the consumer associated with the data slot in the transition state and the current time is larger than the preset threshold value, the consumer holding the data slot continues to hold the data slot, the data state of the data slot is changed into the distributed state, the data slot is associated with the consumer holding the data slot again, and the data slot is added into the identifier of the data slot returned by the consumer holding the data slot.
The preferred procedure is another case where the data slot in the transition state is changed as shown by the reference numeral 6 in fig. 3. If the difference value between the last heartbeat information receiving time of the consumer associated with the data slot in the transition state and the current time is larger than the preset threshold value, namely the heartbeat information of the consumer associated with the data slot in the transition state is overtime, the consumer holding the data slot in the transition state continues to hold the data slot in the transition state, the data state of the data slot in the transition state is changed into the allocated state, the association between the data slot and the consumer with the overtime heartbeat information is cancelled, and the data slot is associated with the consumer which originally holds the data slot again.
For example, in this embodiment, if the difference between the last time when the consumer associated with the data slot No. 1 in the transition state receives the heartbeat information and the current time is greater than the preset threshold, that is, the heartbeat information of the consumer B is overtime, the data state of the data slot No. 1 is changed to the allocated state, the association between the data slot No. 1 and the consumer B with the overtime heartbeat information is cancelled, the data slot No. 1 is associated with the consumer a that originally holds the data slot No. 1 again, and the data slot No. 1 is added to the identifier of the data slot returned to the consumer a.
Step S103-17: and if the consumption breakpoint updating information of the data slot in the finished state indicates that the updated consumption breakpoint position is more ahead than the last consumption end position, changing the data state information of the data slot into the unallocated state.
The preferred procedure is the situation shown by reference number 7 in fig. 3. If the breakpoint updating information of the consumer is received to indicate that the consumption breakpoint position of the updated data slot in the finish state is more front than the last consumption end position of the data slot, but not more back according to the normal consumption sequence, the data slot in the finish state is forcibly updated to be in the unallocated state. Particular care is required to perform this preferred step because the data slot's descendant may have been consumed, potentially resulting in sequential consumption of data that fails to fit the hash value.
For example, if consumer C received by the server sends consumption breakpoint update information for data slot No. 4 in the completed state, and the updated consumption breakpoint position of data slot No. 4 is earlier than the last consumption end position of data slot No. 4, assuming that the size of data slot No. 4 itself is 2000KB, the last consumption end position is 1500KB and the updated consumption breakpoint position is 1200KB earlier, the state information of data slot No. 4 is changed to the unallocated state.
Step S103-19: and if the data of the data slot with the read-only attribute is consumed, changing the data state information of the data slot into a finished state, and canceling the association between the data slot and the consumer holding the data slot.
The preferred procedure is the case shown by reference number 8 in fig. 3. If the data of the data slot of the read-only attribute is consumed, namely the received consumption breakpoint update information indicates that the updated consumption breakpoint position is at the tail end of the data slot of the read-only attribute, the data state information of the data slot is changed into a finished state, and the association between the data slot and a consumer holding the data slot is cancelled.
For example, if the consumer C consumes the data slot No. 4, for example, the server receives the consumption breakpoint update information sent by the consumer C, which indicates that the updated consumption breakpoint position is at the end of the data slot No. 4, the data slot No. 4 is changed to the complete state, and the association between the data slot No. 4 and the consumer C is cancelled.
Step S103-21: and if the difference value between the time when the consumer with the data slot in the transition state receives the heartbeat information last time and the current time is larger than a preset threshold value, changing the data state information of the data slot into an unallocated state.
The preferred procedure is the situation shown by the reference number 9 in fig. 3. If the heartbeat information of the consumer is overtime, namely the difference value between the last time of receiving the heartbeat information and the current time is larger than a preset threshold value, the data slot which is held by the consumer and is in the transition state is changed into the unallocated state.
For example, assuming that consumer A holds data slot number 1 for transition state, if consumer A's heartbeat information times out, data slot number 1 is changed to unallocated state.
Please refer to fig. 4, which is a diagram illustrating a data slot data processing apparatus for a server according to a second embodiment of the present application. Since the apparatus embodiments are substantially similar to the method embodiments, they are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for relevant points. The device embodiments described below are merely illustrative. The data slot data processing device for the server comprises:
a report information receiving unit 201, configured to receive report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
the execution data processing unit 203 is used for triggering the data slot data processing logic unit 204 of the server according to the received report information and the maintained related information of the consumer and data slot state databases, so as to update the consumer and data slot state databases and return response information corresponding to the report information to the consumer; the data slot data processing logic unit 204 of the server at least includes: a return heartbeat unit to return an identification of a data slot intended to be held by the consumer.
The data slot data processing logic unit 204 of the server further includes:
optionally, the return consumption breakpoint update information unit 204-1 is configured to, if the report information includes the consumption breakpoint update information, update, in the consumer and data slot state database, consumption breakpoint position information of the data slot according to the consumption breakpoint update information, and return information on whether the update of the consumption breakpoint position information is successful;
optionally, the consumption breakpoint return information unit 204-3 is configured to return the consumption breakpoint position information of the data slot read from the consumer and data slot state database if the reporting information includes the consumption breakpoint acquisition information.
Optionally, a new data slot unit 204-5 is introduced, configured to determine, if a newly added data slot occurs, whether an ancestor relationship exists between the data slot and each data slot in the same data warehouse; if not, setting the data state information of the data slot to be in an unallocated state; if yes, judging whether the data of the data slot is consumed; if so, setting the data state information of the data slot to be in an unallocated state; and if not, setting the data state information of the data slot to be in a waiting state.
Optionally, a wait for change unallocated unit 204-7 is configured to change the data state information of the data slot to the unallocated state if the data of the ancestor data slot of the data slot in the wait state has been consumed.
Optionally, the unallocated-changed allocated unit 204-9 is configured to, if a data slot in an unallocated state is allocated to a consumer, change the data state information of the data slot to an allocated state, and associate the consumer with the data slot.
Optionally, the distributed change unallocated unit 204-11 is configured to, if a difference between a last time when a consumer in a distributed state receives heartbeat information and a current time is greater than a preset threshold, recycle a data slot held by the consumer, change data state information of the data slot into an unallocated state, and cancel association between the consumer and the data slot.
Optionally, the data slot load balancing unit 204-13 is configured to, if a difference between the number of data slots held by the consumer holding the most data slots and the number of data slots held by the consumer holding the least data slots is greater than 1, change the data state information of one data slot held by the consumer holding the most data slots from the allocated state to the transition state, remove the data slot changed to the transition state from the identifier of the data slot returned to the consumer holding the most data slots, and associate the data slot changed to the transition state with the consumer waiting for consumption.
Optionally, the transition change allocated unit 204-15 is configured to, if the received heartbeat information indicates that the consumer abandons the held data slot in the transition state, add the data slot to the identifier of the data slot returned to the associated consumer, and change the data state information of the data slot from the transition state to the allocated state.
Optionally, the transition timeout changing unit 204-17 is configured to, if a difference between a last time when the consumer associated with the data slot in the transition state receives the heartbeat information and a current time is greater than a preset threshold, continue to hold the data slot by the consumer holding the data slot, change the data state of the data slot to the allocated state, and associate the data slot with the consumer holding the data slot.
Optionally, the change done unallocated unit 204-19 is configured to, if the consumption breakpoint update information of the data slot in the done state indicates that the updated consumption breakpoint position is earlier than the last consumption end position, change the data state information of the data slot to the unallocated state.
Optionally, the changing completion unit 204-21 is configured to, if the data of the data slot with the read-only property has been consumed, change the data state information of the data slot to a completion state, and cancel the association between the data slot and the consumer holding the data slot.
Optionally, the transition change non-allocation unit 204-23 is configured to cancel association between the data slot and the consumer and change the data state information of the data slot to be in a non-allocation state if a difference between a last time when the consumer holding the data slot in the transition state receives the heartbeat information and a current time is greater than a preset threshold.
Please refer to fig. 5, which is a schematic diagram of an electronic device for processing data of a data slot of a server according to a third embodiment of the present application. Since the electronic device embodiments are substantially similar to the method embodiments, they are described relatively simply, and reference may be made to some descriptions of the method embodiments for relevant points. The electronic device embodiments described below are merely illustrative. The data slot data processing electronic equipment for the server comprises:
a display 301;
a processor 303;
a memory 305 for storing a program of data slot data for a server, which when read executed by the processor, performs the following operations: receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer.
Please refer to fig. 6, which is a flowchart illustrating a data slot data processing method for a consumer according to a fourth embodiment of the present application. The data slot data processing method for the consumer is used with the data slot data processing method for the server according to the first embodiment, and comprises the following steps:
step S401: sending report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer.
The application is suitable for the situation that a plurality of consumers collaboratively share one server, namely, one server manages a plurality of consumers. Each consumer sends report information to the server, wherein the report information at least comprises heartbeat information which is sent at fixed time intervals and reports the identification set of all data slots currently held by the consumers to the server. The consumer indicates to the server that the consumer is in a normal working state by sending heartbeat information at fixed time intervals; otherwise, the consumer is in an abnormal state.
The report information sent by the consumer is the report information received by the server, and includes consumption breakpoint update information and consumption breakpoint acquisition information in addition to the heartbeat information, and the contents of these information are completely the same as the contents of the information received by the server, and are not described herein again.
For example, consumer A sends heartbeat information to the server at intervals of 20 seconds with data slots 0, 1, and 5 indicating that consumer A is working properly and holding three data slots 0, 1, and 5.
Step S403: and receiving response information returned by the server.
And each consumer receives the report information sent by the server and returns response information sent to the original consumer. The response information is the response information sent by the server, and the content thereof is not described in detail.
And the consumer receives the response information to acquire the data processing instruction sent by the server.
For example, consumer a receives response information fed back by the server for the heartbeat information sent last time, and the content of this response information is data slot numbers 0 and 5.
Step S405: and executing data slot data processing logic of the consumer according to the data slot consumption requirement, the held consumption information of the data slot and the response information received from the server so as to consume the held data slot data and generate corresponding report information.
And executing the data slot data processing logic of the consumer according to the specific requirement change of the consumer to the consumption data slot, the consumption condition of the held data slot and the data processing instruction obtained from the response information received from the server, so as to consume the data of the held data slot and generate report information which is prepared to be sent to the server and corresponds to the action of the consumer.
The consumer itself needs to maintain consumption information for the data slots it holds, and also needs to know the change in demand for consuming data slots. The data slot data processing logic of the consumer is the core logic of automatically processing how to consume data and how to make corresponding reports to the server according to the interaction of the report information and the response information with the server and the consumption condition of the consumer. Each consumer consumes data according to the data processing logic of the data slot of the consumer, so that the requirement of sequential consumption is met.
For example, consumer a receives response information fed back by the server for the heartbeat information sent last time, and the content of this response information is data slot numbers 0 and 5. And the consumer A executes the data processing logic of the data slot of the consumer, knows that the server wants to abandon the consumption of the data slot No. 1, and abandons the consumption of the data slot No. 1 according to the consumption requirement of the consumer, and then abandons the consumption of the data slot No. 1 and removes the identifier of the data slot No. 1 from the heartbeat information returned to the server, namely only the identifiers of the data slots No. 0 and 5 are generated in the heartbeat information.
The data slot data processing logic of the consumer covers all the specific rules required by the consumer to perform data slot data processing, which are respectively described in the following preferred steps of the data slot data processing logic of this embodiment. These preferred steps include:
step S405-1: if the report information includes consumption breakpoint update information, the consumption breakpoint update information includes: the data warehouse identification, the data slot identification and the consumption breakpoint position information of the data slot; if the report information includes the consumption breakpoint acquisition information, the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
In addition to sending heartbeat information, the consumer sends report information for two other types of content to the server. After consuming the data of the data slot, the consumer needs to send consumption breakpoint update information updated on the consumed data slot to the server; the content of this consumption breakpoint update information includes: the identifier of the data warehouse where the consumed data slot is located, the identifier of the consumed data slot and the specific consumption breakpoint position information of the data slot. After acquiring a new data slot, the consumer also needs to request the server for acquiring the position information of the specific consumption breakpoint of the last consumption on the data slot, so as to continue to consume data from the breakpoint position and continue to consume data backwards; this consumption breakpoint acquisition information includes: an identification of the data warehouse in which the data slot is located and an identification of this data slot.
For example, after holding data slot No. 7, consumer C generates report information including consumption breakpoint update information, and the content of this consumption breakpoint update information includes the data slot No. 7 and the identifier of the data warehouse in which the data slot No. 7 is located.
Step S405-3: and if the received response information shows that the identification set of the data slot which the server wishes to hold is smaller than the identification set of the data slot which the server actually holds, abandoning consumption of the data slot which is actually held and the data slot which is supposed to hold, and generating heartbeat information of removing the identification of the data slot which is actually held and consumption breakpoint updating information of the data slot which is actually held and hopefully held.
If the response information received from the server indicates that the server desires the set of data slots that the consumer holds to be smaller than the set of data slots that the consumer actually holds, then consumption of the data slots of the difference portion between actually holding and intended to hold is aborted, and heartbeat information that removes the identification of those difference data slots and consumption breakpoint update information for those difference data slots are generated.
For example, the content of the response message received by consumer a is data slots 0 and 5, while consumer a currently holds data slots 0, 1 and 5, actually holds the difference set data slot with the expected hold data slot 1, consumer a abandons consumption of data slot 1, removes the identifier of data slot 1 from the heartbeat message returned to the server, i.e. only generates the identifiers of data slots 0 and 5 in the heartbeat message, and sends consumption breakpoint update message of data slot 1 to the server.
Step S405-5: and if the received response information shows that the identification set of the data slot which the server wishes to hold is larger than the identification set of the data slot which is actually held, writing consumption breakpoint request information of the data slot which is hoped to hold and is actually held into the report information.
If the response information received from the server indicates that the server wishes the consumer to hold a larger set of data slots than the consumer actually holds, then a request is made to the server for data slots holding a portion of the difference between the intended hold and the actual hold, generating heartbeat information that adds the identification of those difference data slots.
For example, consumer B receives the response information in data slots 1, 2, and 3, while consumer B currently holds data slots 2 and 3 and wishes to hold the difference set data slot 1 from the actual data slot, and then consumer B requests the server for data slot 1 holding the difference set portion and generates heartbeat information with the identification of data slot 1 added, i.e., heartbeat information including the identifications of data slots 1, 2, and 3.
Step S405-5: if the received response message indicates that the server wishes to have the same set of identifications for the data slots as the set of identifications for the data slots actually held, then the held data slots continue to be consumed.
If the response information received from the server indicates that the server wishes the set of data slots held by the consumer to be the same as the set of data slots actually held by the consumer, then the held data slots continue to be consumed, generating heartbeat information including the identified set of held data slots.
For example, if consumer B receives the response information in data slots 2 and 3, and consumer B currently holds data slots 2 and 3 and wishes to hold the same identification set as the actually held data slot, then consumer B continues to consume the held data slot and generates heartbeat information including the identification set of data slots 2 and 3.
Step S405-7: and if the received response information comprises that the consumption breakpoint of the data slot is not successful, rewriting the consumption breakpoint updating information of the data slot into the report information.
And if the received response information returned by the server includes the consumption breakpoint unsuccessful for updating the data slot, rewriting the consumption breakpoint updating information of the data slot into the report information so that the server updates the breakpoint position information of the data slot again.
For example, in this embodiment, the consumer a sends the consumption breakpoint update information of the data slot No. 1 to the server, and the server subsequently returns that the consumption breakpoint of the updated data slot is unsuccessful, and then writes the consumption breakpoint update information of the data slot No. 1 into the report information again.
Step S407: and returning to execute the step of sending the report information to the server.
Returning to the step of sending the report information to the server in the step S401, and sending the report information generated after the data slot data processing logic of the consumer is executed to the server.
For example, consumer A sends heartbeat information for data slots 0 and 5 to the server at the next fixed time interval.
Please refer to fig. 7, which is a diagram illustrating a data sink data processing apparatus for a consumer according to a fifth embodiment of the present application. Since the apparatus embodiments are substantially similar to the method embodiments, they are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for relevant points. The device embodiments described below are merely illustrative. The data slot data processing device for consumers comprises:
a report information sending unit 501, configured to send report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
a response information receiving unit 503, configured to receive response information returned by the server;
the execution data processing unit 505 is configured to trigger the data slot data processing logic unit 506 of the consumer according to the data slot consumption requirement, the consumption information of the held data slot, and the response information received from the server, so as to consume the data of the held data slot and generate the corresponding report information;
a return trigger report sending information unit 507, configured to return trigger the report sending information unit 501.
The data slot data processing logic 506 of the consumer further comprises:
optionally, the consumption breakpoint updating unit 506-1 is configured to, if the report information includes consumption breakpoint updating information, the consumption breakpoint updating information includes: the data warehouse identification, the data slot identification and the consumption breakpoint position information of the data slot;
optionally, the consumption breakpoint acquiring unit 506-3 is configured to, if the report information includes the consumption breakpoint acquiring information, the consumption breakpoint acquiring information includes: an identification of a data warehouse and an identification of a data slot.
Optionally, the excess load balancing unit 506-5 is configured to abandon consumption of a difference set data slot that is actually held and desired to hold if the received response information indicates that the identification set of the data slot that the server wishes to hold is smaller than the identification set of the data slot that is actually held, and generate heartbeat information with the identification of the difference set data slot removed and consumption breakpoint update information of the difference set data slot.
Optionally, the load balancing unit 506-7 is configured to, if the received response information indicates that the identification set of the data slot that the server wishes to hold is larger than the identification set of the data slot that is actually held, write consumption breakpoint request information for the data slot that is expected to hold and the data slot that is actually held into the report information.
Optionally, the same load balancing unit 506-9 is configured to continue consuming the held data slot if the received response information indicates that the identification set of the data slot that the server wishes to hold is the same as the identification set of the data slot that is actually held.
Optionally, the re-breakpoint updating unit 506-11 is configured to, if the received response information includes that the consumption breakpoint of the data slot is not successful, re-write the consumption breakpoint updating information of the data slot into the report information.
Please refer to fig. 8, which is a diagram illustrating a data slot data processing electronic device for a consumer according to a sixth embodiment of the present application. Since the electronic device embodiments are substantially similar to the method embodiments, they are described relatively simply, and reference may be made to some descriptions of the method embodiments for relevant points. The electronic device embodiments described below are merely illustrative. The data slot data processing electronic device for a consumer, comprising:
a display 601;
a processor 603;
a memory 605 for storing a program of data slot data for a consumer, which when read and executed by the processor, performs the following operations: sending report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; receiving response information returned by the server; executing data slot data processing logic of a consumer according to the data slot consumption requirement, the held consumption information of the data slot and the response information received from the server so as to consume the held data slot data and generate corresponding report information; and returning to execute the step of sending the report information to the server.
Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application, and those skilled in the art can make variations and modifications without departing from the spirit and scope of the present application, therefore, the scope of the present application should be determined by the claims that follow.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
1. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
2. As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

Claims (43)

1. A data slot data processing method for a server, comprising:
receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer.
2. The data slot data processing method for the server according to claim 1, wherein the reporting information further comprises: the consumption breakpoint updating information and the consumption breakpoint acquiring information; the consumption breakpoint update information includes: identification of a data warehouse, identification of a data slot and consumption breakpoint position information; the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
3. The data slot data processing method for the server according to claim 1, wherein the heartbeat information is received at regular time intervals.
4. The data slot data processing method for the server according to claim 1, wherein the heartbeat information further includes an identification of a data warehouse to which the data slot held by the consumer belongs.
5. The data slot data processing method for the server of claim 1, wherein the consumer and data slot status database comprises: the data slot management method comprises the following steps of information of a consumer, physical information of the data slot, ancestor relation information of the data slot, data state information of the data slot, consumption breakpoint position information of the data slot and correlation information of the consumer, the data slot and the data state of the data slot.
6. The data slot data processing method for the server of claim 2, wherein the data slot data processing logic of the server further comprises:
if the report information comprises the consumption breakpoint updating information, updating the consumption breakpoint position information of the data slot in the consumer and data slot state database according to the consumption breakpoint updating information, and returning the information whether the consumption breakpoint position information is successfully updated;
and if the report information comprises the consumption breakpoint acquisition information, returning the consumption breakpoint position information of the data slot read from the consumer and the data slot state database.
7. The data slot data processing method for the server according to claim 5 or 6, wherein the data state information of the data slot includes: unallocated, allocated, waiting, transferred, and done states; wherein the unallocated state indicates that the data slot is not held by the consumer and is waiting to be allocated; the allocated status, indicating that the data slot has been held and consumed by the consumer; the waiting state represents that the data slot cannot be consumed currently, and waits for the ancestor data slot of the data slot to finish consumption; the transition state represents that the data slot is in a transition state for transition to another consumer for consumption; the done state, indicating that the data of the data slot has been consumed and is not held by the consumer.
8. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
if a newly added data slot appears, judging whether the data slot has ancestor relation with each data slot in the same data warehouse; if not, setting the data state information of the data slot to be in an unallocated state; if yes, judging whether the data of the data slot is consumed; if so, setting the data state information of the data slot to be in an unallocated state; and if not, setting the data state information of the data slot to be in a waiting state.
9. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
if the data of the ancestor data slot of the data slot in the wait state has been consumed, the data state information of the data slot is changed to the unallocated state.
10. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
if a data slot in an unallocated state is allocated to a consumer, the data state information of the data slot is changed to an allocated state, and the consumer is associated with the data slot.
11. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
if the difference value between the time when the consumer in the distributed state receives the heartbeat information last time and the current time is larger than a preset threshold value, recovering the data slot held by the consumer, changing the data state information of the data slot into the non-distributed state, and canceling the association between the consumer and the data slot.
12. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
if the difference between the number of data slots held by the consumer holding the most data slots and the number of data slots held by the consumer holding the least data slots is greater than 1, changing the data state information of one data slot held by the consumer holding the most data slots from the allocated state to the transition state, removing the data slot changed to the transition state from the identification of the data slot returned to the consumer holding the most data slots, and associating the data slot changed to the transition state with the consumer holding the least data slots.
13. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
and if the received heartbeat information indicates that the consumer abandons the held data slot with the transferred state, adding the data slot into the identification of the data slot returned to the associated consumer, and changing the data state information of the data slot from the transferred state to the distributed state.
14. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
if the difference value between the last heartbeat information receiving time of the consumer associated with the data slot in the transition state and the current time is larger than the preset threshold value, the consumer holding the data slot continues to hold the data slot, the data state of the data slot is changed into the distributed state, and the data slot is associated with the consumer holding the data slot.
15. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
and if the consumption breakpoint updating information of the data slot in the finished state indicates that the updated consumption breakpoint position is more ahead than the last consumption end position, changing the data state information of the data slot into the unallocated state.
16. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
and if the data of the data slot with the read-only attribute is consumed, changing the data state information of the data slot into a finished state, and canceling the association between the data slot and the consumer holding the data slot.
17. The data slot data processing method for the server of claim 7, wherein the data slot data processing logic of the server further comprises:
and if the difference value between the time when the consumer with the data slot in the transition state receives the heartbeat information last time and the current time is larger than a preset threshold value, canceling the association between the data slot and the consumer, and changing the data state information of the data slot into an unallocated state.
18. A data slot data processing apparatus for a server, comprising:
the report receiving information unit is used for receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
the execution data processing unit is used for triggering a data slot data processing logic unit of the server according to the received report information and the maintained related information of the consumer and the data slot state database so as to update the consumer and the data slot state database and return response information corresponding to the report information to the consumer; the data slot data processing logic unit of the server at least comprises: a return heartbeat unit to return an identification of a data slot intended to be held by the consumer.
19. The data slot data processing apparatus for the server of claim 18, wherein the data slot data processing logic of the server further comprises:
a consumption breakpoint updating information returning unit, configured to update, in the consumer and data slot state database, consumption breakpoint position information of the data slot according to the consumption breakpoint updating information if the report information includes the consumption breakpoint updating information, and return information indicating whether the consumption breakpoint position information is successfully updated;
and the consumption breakpoint return information obtaining unit is used for returning the consumption breakpoint position information of the data slot read from the consumer and the data slot state database if the report information comprises the consumption breakpoint obtaining information.
20. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
introducing a new data slot unit for judging whether the data slot has ancestor relation with each data slot in the same data warehouse if a newly added data slot appears; if not, setting the data state information of the data slot to be in an unallocated state; if yes, judging whether the data of the data slot is consumed; if so, setting the data state information of the data slot to be in an unallocated state; and if not, setting the data state information of the data slot to be in a waiting state.
21. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the waiting change unallocated unit is used for changing the data state information of the data slot into an unallocated state if the data of the ancestor data slot of the data slot in the waiting state is consumed up.
22. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and an unallocated change allocated unit configured to change the data state information of the data slot to an allocated state and associate the consumer with the data slot, if the data slot in an unallocated state is allocated to a consumer.
23. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the distributed change unallocated unit is used for recovering the data slot held by the consumer if the difference value between the time when the consumer in the distributed state receives the heartbeat information last time and the current time is greater than a preset threshold value, changing the data state information of the data slot into the unallocated state, and canceling the association between the consumer and the data slot.
24. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the data slot load balancing unit is used for changing the data state information of one data slot held by the consumer holding the most data slots into a transition state from the distributed state if the difference value between the number of the data slots held by the consumer holding the most data slots and the number of the data slots held by the consumer holding the least data slots is larger than 1, removing the data slot changed into the transition state from the identification of the data slot returned to the consumer holding the most data slots, and associating the data slot changed into the transition state with the consumer waiting for consumption.
25. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the transfer change distributed unit is used for adding the data slot into the identifier of the data slot returned to the associated consumer if the received heartbeat information indicates that the consumer abandons the held data slot in the transfer state, and changing the data state information of the data slot from the transfer state to the distributed state.
26. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the transition timeout changing unit is used for continuing holding the data slot by the consumer holding the data slot if the difference value between the last heartbeat information receiving time of the consumer associated with the data slot in the transition state and the current heartbeat information receiving time is greater than a preset threshold value, changing the data state of the data slot into the distributed state, and associating the data slot with the consumer holding the data slot.
27. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the change-over-completion unallocated unit is used for changing the data state information of the data slot into an unallocated state if the consumption breakpoint update information of the data slot in the received completion state indicates that the updated consumption breakpoint position is more ahead than the last consumption end position.
28. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the change finishing unit is used for changing the data state information of the data slot into a finishing state and canceling the association between the data slot and the consumer holding the data slot if the data of the data slot with the read-only attribute is consumed.
29. The data slot data processing apparatus for the server of claim 19, wherein the data slot data processing logic of the server further comprises:
and the transfer change non-distribution unit is used for canceling the association between the data slot and the consumer and changing the data state information of the data slot into a non-distribution state if the difference value between the time when the consumer with the data slot in the transfer state receives the heartbeat information last time and the current time is greater than a preset threshold value.
30. A data slot data processing electronic device for a server, the electronic device comprising:
a display;
a processor;
a memory for storing a program of data slot data for a server, the program when read executed by the processor performing the following: receiving report information sent by a consumer; the report information at least comprises heartbeat information; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; executing data slot data processing logic of a server according to the received report information and the maintained related information of the consumer and data slot state database so as to update the consumer and data slot state database and return response information corresponding to the report information to the consumer; wherein the data slot data processing logic comprises at least: returning an identification of a data slot intended to be held by the consumer.
31. A data slot data processing method for a consumer for use with a data slot data processing method for a server, comprising:
sending report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
receiving response information returned by the server;
executing data slot data processing logic of a consumer according to the data slot consumption requirement, the held consumption information of the data slot and the response information received from the server so as to consume the held data slot data and generate corresponding report information;
and returning to execute the step of sending the report information to the server.
32. The data slot data processing method for a consumer of claim 31, wherein the consumer's data slot data processing logic further comprises:
if the report information includes consumption breakpoint update information, the consumption breakpoint update information includes: the data warehouse identification, the data slot identification and the consumption breakpoint position information of the data slot;
if the report information includes the consumption breakpoint acquisition information, the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
33. The data slot data processing method for a consumer of claim 32, wherein the consumer's data slot data processing logic further comprises:
and if the received response information shows that the identification set of the data slot which the server wishes to hold is smaller than the identification set of the data slot which the server actually holds, abandoning the consumption of the data slot which is actually held and the data slot which is supposed to hold, and generating heartbeat information of the identification of the data slot which is actually held and consumption breakpoint updating information of the data slot which is actually held and hopefully held.
34. The data slot data processing method for a consumer of claim 32, wherein the consumer's data slot data processing logic further comprises:
and if the received response information shows that the identification set of the data slot which the server wishes to hold is larger than the identification set of the data slot which is actually held, writing consumption breakpoint request information of the data slot which is hoped to hold and is actually held into the report information.
35. The data slot data processing method for a consumer of claim 32, wherein the consumer's data slot data processing logic further comprises:
if the received response message indicates that the server wishes to have the same set of identifications for the data slots as the set of identifications for the data slots actually held, then the held data slots continue to be consumed.
36. The data slot data processing method for a consumer of claim 32, wherein the consumer's data slot data processing logic further comprises:
and if the received response information comprises that the consumption breakpoint of the data slot is not successful, rewriting the consumption breakpoint updating information of the data slot into the report information.
37. A data slot data processing apparatus for a consumer for use with a data slot data processing apparatus for a server, comprising:
a report information sending unit, configured to send report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer;
a response information receiving unit, configured to receive response information returned by the server;
the execution data processing unit is used for triggering the data slot data processing logic unit of the consumer according to the data slot consumption requirement, the held data slot consumption information and the response information received from the server so as to consume the held data slot data and generate corresponding report information;
and the return triggering report sending information unit is used for returning and triggering the report sending information unit.
38. The data slot data processing apparatus for a consumer of claim 37, wherein the consumer's data slot data processing logic further comprises:
a consumption breakpoint updating unit, configured to, if the report information includes consumption breakpoint updating information, the consumption breakpoint updating information includes: the data warehouse identification, the data slot identification and the consumption breakpoint position information of the data slot;
a consumption breakpoint acquisition unit, configured to, if the report information includes the consumption breakpoint acquisition information, the consumption breakpoint acquisition information includes: an identification of a data warehouse and an identification of a data slot.
39. The data slot data processing apparatus for a consumer of claim 38, wherein the consumer's data slot data processing logic further comprises:
and the excess load balancing unit is used for giving up consumption of the difference set data slot which is actually held and hoped to hold if the received response information shows that the identification set of the data slot which the server hopes to hold is smaller than the identification set of the data slot which is actually held, and generating heartbeat information of removing the identification of the difference set data slot and consumption breakpoint updating information of the difference set data slot.
40. The data slot data processing apparatus for a consumer of claim 38, wherein the consumer's data slot data processing logic further comprises:
and the load balancing unit is used for writing consumption breakpoint request information of a data slot which is expected to hold and a difference set data slot which is actually held into the report information if the received response information shows that the identification set of the data slot which is expected to hold by the server is larger than the identification set of the data slot which is actually held.
41. The data slot data processing apparatus for a consumer of claim 38, wherein the consumer's data slot data processing logic further comprises:
and the same load balancing unit is used for continuing consuming the held data slots if the received response information indicates that the identification set of the data slots which the server wishes to hold is the same as the identification set of the data slots which are actually held.
42. The data slot data processing apparatus for a consumer of claim 38, wherein the consumer's data slot data processing logic further comprises:
and the re-breakpoint updating unit is used for rewriting the consumption breakpoint updating information of the data slot into the report information if the received response information includes that the consumption breakpoint of the data slot is not successful.
43. A data slot data processing electronic device for a consumer, the electronic device comprising:
a display;
a processor;
a memory for storing a program of data slot data for a consumer, the program when read executed by the processor performing the following: sending report information to a server; the report information at least comprises: heartbeat information transmitted at fixed time intervals; the heartbeat information at least comprises: an identification of a data slot currently held by the consumer; receiving response information returned by the server; executing data slot data processing logic of a consumer according to the data slot consumption requirement, the held consumption information of the data slot and the response information received from the server so as to consume the held data slot data and generate corresponding report information; and returning to execute the step of sending the report information to the server.
CN201610411103.6A 2016-06-13 2016-06-13 Data slot data processing method and device for server and consumer Active CN107491975B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610411103.6A CN107491975B (en) 2016-06-13 2016-06-13 Data slot data processing method and device for server and consumer

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610411103.6A CN107491975B (en) 2016-06-13 2016-06-13 Data slot data processing method and device for server and consumer

Publications (2)

Publication Number Publication Date
CN107491975A CN107491975A (en) 2017-12-19
CN107491975B true CN107491975B (en) 2021-02-23

Family

ID=60642419

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610411103.6A Active CN107491975B (en) 2016-06-13 2016-06-13 Data slot data processing method and device for server and consumer

Country Status (1)

Country Link
CN (1) CN107491975B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110062038A (en) * 2019-04-09 2019-07-26 网宿科技股份有限公司 A kind of data transmission scheduling method and system
CN110503466A (en) * 2019-08-15 2019-11-26 深圳市云积分科技有限公司 A kind of consumer demographics' acquisition methods and device based on interactive event
CN116881984B (en) * 2023-09-08 2024-02-23 云筑信息科技(成都)有限公司 Data monitoring method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008151082A2 (en) * 2007-05-31 2008-12-11 Microsoft Corporation Analytical framework for multinode storage reliability analysis
WO2013154135A1 (en) * 2012-04-10 2013-10-17 株式会社 東芝 Ultrasound diagnostic device, ultrasound image processing device, and medical image diagnostic device

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4668655B2 (en) * 2005-03-24 2011-04-13 株式会社バンダイナムコゲームス Program, information storage medium, and image generation system
CN101370032B (en) * 2008-09-09 2011-08-10 中兴通讯股份有限公司 Modification log maintenance method and apparatus for data synchronization
US9753884B2 (en) * 2009-09-30 2017-09-05 Microsoft Technology Licensing, Llc Radio-control board for software-defined radio platform
US9747600B2 (en) * 2012-03-30 2017-08-29 United State Poastal Service Item status tracking
CN104679772B (en) * 2013-11-29 2019-01-25 深圳市腾讯计算机***有限公司 Method, apparatus, equipment and the system of file are deleted in Distributed Data Warehouse
CN104793988B (en) * 2014-01-20 2019-01-22 阿里巴巴集团控股有限公司 The implementation method and device of integration across database distributed transaction
CN104539583B (en) * 2014-12-05 2018-01-30 浙江浙大中控信息技术有限公司 A kind of real-time data base ordering system and method

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2008151082A2 (en) * 2007-05-31 2008-12-11 Microsoft Corporation Analytical framework for multinode storage reliability analysis
WO2013154135A1 (en) * 2012-04-10 2013-10-17 株式会社 東芝 Ultrasound diagnostic device, ultrasound image processing device, and medical image diagnostic device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
IDC管理平台监控预警***的设计与实现;郭伟;《中国优秀硕士学位论文全文数据库 信息科技辑》;20131215;18-20 *

Also Published As

Publication number Publication date
CN107491975A (en) 2017-12-19

Similar Documents

Publication Publication Date Title
US9852204B2 (en) Read-only operations processing in a paxos replication system
CN108959510B (en) Partition level connection method and device for distributed database
CN108509462B (en) Method and device for synchronizing activity transaction table
US11868315B2 (en) Method for splitting region in distributed database, region node, and system
CN105630418A (en) Data storage method and device
CN107491975B (en) Data slot data processing method and device for server and consumer
CN107665219B (en) Log management method and device
CN111638995A (en) Metadata backup method, device and equipment and storage medium
CN104423982A (en) Request processing method and device
CN109033365B (en) Data processing method and related equipment
CN114827165A (en) Method and block link point for grouping multiple transactions
CN102959548A (en) Data storage method, search method and device
US7984072B2 (en) Three-dimensional data structure for storing data of multiple domains and the management thereof
CN114741449A (en) Object storage method and device based on distributed database
CN110019130B (en) Database updating method and device
CN109710698B (en) Data aggregation method and device, electronic equipment and medium
CN113760902A (en) Data splitting method, device, equipment, medium and program product
CN113157600A (en) Space allocation method of shingled hard disk, file storage system and server
KR20120082176A (en) Data processing method of database management system and system thereof
CN111161064A (en) Processing method and device for blockchain transaction records
CN110659303A (en) Read-write control method and device for database nodes
CN112860912B (en) Method and device for updating knowledge graph
US20210303339A1 (en) Data backup method, electronic device and computer program product
EP3951611A1 (en) Block verification method, apparatus and device
CN106959888B (en) Task processing method and device in cloud storage system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant