CN111190707B

CN111190707B - Data processing method and device

Info

Publication number: CN111190707B
Application number: CN201910712832.9A
Authority: CN
Inventors: 罗洪林; 肖晨光
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-08-02
Filing date: 2019-08-02
Publication date: 2023-04-28
Anticipated expiration: 2039-08-02
Also published as: CN111190707A

Abstract

A data processing method and device, comprising: acquiring a plurality of data and dividing the data into a plurality of data groups; distributing the data group to corresponding processing processes for processing, storing the group identification of the data group to the local and synchronizing the processing results of the processing processes to the monitoring process; triggering a monitoring process to determine unprocessed data corresponding to an abnormal processing process based on a monitoring side record file when the processing process is abnormal; when the abnormal processing process is not recovered in a preset time period, the unprocessed data is distributed to other processing processes for processing; in the process of distributing unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered, checking the current group identification of the unprocessed data; if the verification is not passed, the exception handling process is terminated to continue to process the unprocessed data, and other handling processes process the unprocessed data, so that the processing integrity is improved.

Description

Data processing method and device

Technical Field

The present invention relates to the field of communications, and in particular, to a data processing method and apparatus.

Background

The data processing process in the communication system comprises the following steps: a series of data flows from the upstream system into the data processing system, through the data processing system, and to the downstream system until the downstream system acquires the series of data. The circulation process is divided into two types, wherein the first type of data is published (published) by an upstream system, a receiver of a downstream system subscribes (subscriibe), a data processing system records one or more subscribers, and data conforming to rules is sent to the subscribers according to the subscription rules; the second type of data is produced (product) by the upstream system and consumed (Consume) by the downstream system consumer, and the data processing system receives and stores the data, waiting for a consumer's request to send the data to the consumer.

At present, no matter what kind of circulation process is adopted, in order to ensure high availability of a data processing system, a processing mode of a multi-node deployment and arbitration mechanism is generally adopted for processing, and the significance of the processing is that a plurality of nodes receive requests of users and are mutually active and standby or update processing progress information of data in a mode of an election decision maker (Leader). When a certain node fails, reinitiating election and electing a new Leader, thereby ensuring high availability of the data processing system, but when the failed node is the Leader, the data which should be processed originally is not processed, and further the integrity of data processing is reduced.

Disclosure of Invention

The embodiment of the application provides a data processing method and device, which can improve the integrity of data processing.

The embodiment of the application provides a data processing method, which comprises the following steps:

acquiring a plurality of data to be processed from a data source, dividing the plurality of data into a plurality of data groups, and storing the group identification of each data group to the local;

distributing each data group to a corresponding processing process in the processing process group, and updating the data state of the data in the data source to a state to be processed;

processing the data in the target data group according to the processing progress and the data state of the data in the distributed target data group; and synchronizing the processing result of the processing process to a monitoring process monitoring the processing process, wherein the processing result comprises the data state of the processed data;

when the processing process is detected to be abnormal, triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a monitoring side group identifier;

updating the data state of the unprocessed data in the data source to an unprocessed state through a monitoring process;

When the abnormal processing process is detected not to be recovered to normal within a preset time period, distributing the unprocessed data to other processing processes for processing, and adjusting the group identification of the unprocessed data to the group identification of a data group corresponding to the other processing processes, wherein the other processing processes are processing processes except the abnormal processing process in the processing process group;

in the process of distributing the unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered to be normal, checking the current group identifier of the unprocessed data;

if the verification is not passed, the exception processing process is terminated to continue to process the unprocessed data, and the other processing processes process the unprocessed data.

In some embodiments, the method further comprises:

when the abnormal processing process is detected to be recovered to be normal within a preset time period, the unprocessed data is distributed to the abnormal processing process; updating the data state of the unprocessed data into a state to be processed;

reprocessing the unprocessed data based on the exception handling process and the updated data state of the unprocessed data.

In some embodiments, the allocating the unprocessed data to other processing processes for processing, and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing processes includes:

updating the data state of the unprocessed data into a to-be-processed state, and adjusting the group identification of the unprocessed data into the group identification of the data group corresponding to the other processing processes;

triggering other processing processes to process the unprocessed data according to the data state and the group identification of the unprocessed data.

In some embodiments, the triggering the listening process to determine unprocessed data in the data set corresponding to the exception handling process based on the listening side record file includes:

triggering the monitoring process to determine the first data with the data state in the data group corresponding to the abnormal processing process as the in-process state and the second data with the data state as the to-be-processed state as the unprocessed data based on the monitoring side record file.

In some embodiments, in the process of allocating the unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is restored to be normal, checking the current group identifier of the unprocessed data includes:

In the process of distributing the unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the abnormal processing process;

if the verification is not passed, the exception processing process is terminated to continue processing the unprocessed data, and the other processing processes process the unprocessed data, including:

and when the current group identification of the first data is different from the group identification corresponding to the abnormal processing process, terminating the abnormal processing process to process the first data, and processing the first data by the other processing processes.

In some embodiments, the method further comprises:

and when the current group identifier of the first data is the same as the group identifier corresponding to the abnormal processing process, refusing to distribute the first data to the other processing processes, and triggering the abnormal processing process to continue processing the first data.

In some embodiments, the assigning each data group to a corresponding process in the set of process steps includes:

Group marking is carried out on each data group, so that a group identifier corresponding to each data group is obtained;

determining a target process identifier corresponding to the group identifier from a process identifier set, wherein the process identifier set comprises process identifiers of processing processes in the processing process group;

and distributing the data group corresponding to the group identifier to the processing process corresponding to the target process identifier.

In some embodiments, the processing the data in the target data set according to the processing process and the data state of the data in the allocated target data set includes:

determining target data which is required to be processed currently by the processing process from a target data group;

acquiring data processing expected information corresponding to the processing process and data attribute information of the target data, wherein the data processing expected information comprises a data group expected to be processed by the processing process and a data state expected to be processed, and the data attribute information comprises the current data state of the target data and the current data group;

verifying target data based on the data processing expected information and the data attribute information;

and when the verification is passed, triggering a processing process to process the target data.

Correspondingly, the embodiment of the application also provides a data processing device, which comprises:

the data dividing unit is used for acquiring a plurality of data to be processed from a data source, dividing the plurality of data into a plurality of data groups and storing the group identification of each data group to the local;

the first data distribution unit is used for distributing each data group to a corresponding processing process in the processing process group and updating the data state of the data in the data source to a to-be-processed state;

the first data processing unit is used for processing the data in the target data group according to the processing process and the data state of the data in the distributed target data group; and synchronizing the processing result of the processing process to a monitoring process monitoring the processing process, wherein the processing result comprises the data state of the processed data;

the data determining unit is used for triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file when the processing process is detected to be abnormal, wherein the monitoring side record file comprises a synchronized data state and a group identifier of a monitoring side;

The data updating unit is used for updating the data state of the unprocessed data in the data source into an unprocessed state through a monitoring process;

the second data distribution unit is used for distributing the unprocessed data to other processing processes for processing when the abnormal processing process is detected not to be recovered to normal within a preset time period, and adjusting the group identification of the unprocessed data to the group identification of a data group corresponding to the other processing processes, wherein the other processing processes are processing processes except the abnormal processing process in the processing process group;

the data verification unit is used for verifying the current group identification of the unprocessed data when the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal in the process of distributing the unprocessed data to other processing processes;

and the termination processing unit is used for terminating the abnormal processing process to continue processing the unprocessed data if the verification fails, and processing the unprocessed data by the other processing processes.

In some embodiments, the data processing apparatus further comprises:

A third data allocation subunit, configured to allocate the unprocessed data to the abnormal processing process when it is detected that the abnormal processing process returns to normal within a preset time period; updating the data state of the unprocessed data into a state to be processed;

and the data processing unit is used for reprocessing the unprocessed data based on the abnormal processing process and the updated data state of the unprocessed data.

Correspondingly, the embodiment of the application also provides a storage medium, wherein the storage medium stores instructions, and the instructions realize the data processing method provided by any one of the embodiments of the application when being executed by a processor.

According to the data processing method provided by the embodiment of the application, when data in the data groups are sent to the corresponding processing processes for processing, the group identification of each data group is stored locally, in the processing process, the processing result of each piece of data is synchronized to the monitoring process, and if the processing process is abnormal, the fact that data which are not processed by the abnormal processing process yet can be found is ensured according to the local and monitoring side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing by a monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type, and the exception processing process is recovered to be normal, checking the current group identification of the unprocessed data, if the checking is passed, the unprocessed data is not distributed to the other processing processes, the processing can be continued by the exception processing process, if the checking is not passed, the unprocessed data is distributed to the other processing processes, the exception processing process does not need to process the unprocessed data, repeated processing is avoided, and processing integrity is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1a is a schematic diagram of a first scenario of a data processing system according to an embodiment of the present application.

Fig. 1b is a schematic flow chart of a first flow chart of a data processing method according to an embodiment of the present application.

Fig. 2 is a second flowchart of a data processing method according to an embodiment of the present application.

FIG. 3a is a schematic diagram of a second scenario of a data processing system according to an embodiment of the present application.

FIG. 3b is a third flowchart illustrating a data processing method according to an embodiment of the present application

Fig. 4a is a schematic diagram of a first interaction among a processing server, a listening server, and a data source in the data processing method according to the embodiment of the present application.

Fig. 4b is a second interaction schematic diagram of a processing server, a listening server, and a data source in the data processing method according to the embodiment of the present application.

Fig. 5a is a first flowchart of a processing server and a listening server in processing an exception according to an embodiment of the present application.

Fig. 5b is a second flowchart of a processing server and a listening server in processing an exception according to an embodiment of the present application.

Fig. 5c is a third flowchart of a processing server and a listening server for processing an exception according to an embodiment of the present application.

Fig. 5d is a fourth flowchart of a processing server and a listening server for processing an exception according to an embodiment of the present application.

Fig. 6a is a schematic diagram of a first structure of a data processing apparatus according to an embodiment of the present application.

Fig. 6b is a schematic diagram of a second structure of the data processing apparatus according to the embodiment of the present application.

Fig. 7 is a schematic structural diagram of a server according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The embodiment of the application provides a data processing method, a data processing device, a storage medium and a server. Wherein the data processing means may be integrated in a server, which may refer to a single stand-alone server, e.g. the data processing means may be integrated in a data processing server; in addition, a server may also refer to a server cluster constructed from a plurality of servers, for example, the data processing apparatus may be integrated into a data processing server, a listening server, that is, the data processing server, the listening server may implement the methods of the present application,

Taking an example that the data processing apparatus is integrated in a single server, referring to fig. 1a, an embodiment of the present application provides a data processing system, including: the data source 10, the processing server 20 and the terminal 30, wherein the data source 10 may also be a third party service or a database disposed in the terminal 30, which is not limited herein. The terminal 30 may be a mobile phone, a tablet computer, a notebook computer, etc., and fig. 1a illustrates the terminal 30 as a mobile phone. The terminal 30 may have applications required by various users, such as applications with entertainment functions (e.g., video applications, audio playback applications, game applications, reading software), and applications with service functions (e.g., map navigation applications, dining applications, etc.).

Based on the system shown in fig. 1a, taking the case that the user uses shopping software to make a purchase, when a large number of users initiate a purchase request at the same time, the data source 10 includes a large number of request data initiated by the user through the terminal 30, the processing server 20 may acquire the data to be processed from the data source 10, and after the processing of the processing server 20 is completed, the processing server 20 resends the processing result of each piece of data to the data source 10, where the processing server 20 includes a processing process and a monitoring process for monitoring the processing process. Specifically, the processing server 20 acquires a plurality of data to be processed from the data source 10, and divides the plurality of data into a plurality of data groups; assigning each data group to a corresponding process in the process group, storing the group identification of each data group locally, and updating the data state of the data in the data source 10 to a to-be-processed state; processing the data in the target data group according to the processing progress and the data state of the data in the distributed target data group; and synchronizing the processing result of the processing process to the monitoring process, wherein the processing result comprises the data state of the processed data; when detecting that the processing process is abnormal, triggering a monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identifier of a monitoring side; updating the data state of the unprocessed data in the data source 10 to an unprocessed state by the listening process; when the abnormal processing process is detected not to be recovered to normal within a preset time period, the unprocessed data is distributed to other processing processes for processing, the group identification of the unprocessed data is adjusted to the group identification of the data group corresponding to the other processing processes, and the other processing processes are processing processes except the abnormal processing process in the processing process group; in the process of distributing unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered to be normal, checking the current group identification of the unprocessed data; if the verification is not passed, the normal processing process is terminated to continue to process the unprocessed data, and other processing processes process the unprocessed data.

Specifically, when the data acquired from the data source 10 is processed by the processing server 20, the processed data may be selectively sent to the terminal 30 or stored in the processing server 20 according to different processing circulation procedures, which is not limited herein.

The example of fig. 1a is merely an example of a system architecture for implementing embodiments of the present application, and embodiments of the present application are not limited to the system architecture shown in fig. 1a, and various embodiments of the present application are presented based on the system architecture.

In some embodiments, a data processing method is provided, which may be executed by a processor of a server, as shown in fig. 1b, and includes:

101. a plurality of data to be processed is acquired from a data source, and the plurality of data is divided into a plurality of data groups.

The data source is a source of data, and is a device or an original medium for providing some required data. The data source stores a plurality of data, and the data may include, for example, request data sent by a user for purchase or other data corresponding to the request data. The plurality of data to be processed may be collectively acquired in the form of a set, for example, the data set { data 1, data 2 … … data n }.

The data set may be divided into a plurality of data sets, for example, the data sets to be processed are divided by the total number of processing processes or the number of processing processes in idle state, for example, the total number of processing processes or the number of processing processes in idle state is 3, and n data sets to be processed are equally divided into 3 data sets.

102. And distributing each data group to a corresponding processing process in the processing process group, storing the group identification of each data group locally, and updating the data state of the data in the data source into a state to be processed.

Wherein the data set comprises at least one data from the data source, the processing process set comprises a plurality of processing processes, the processing processes being processes for processing data, such as response processes for responding to request data, etc. Data states the data states of data can be divided into 4 types according to the processing procedure of the processing procedure on the data: the first is an unprocessed state, wherein the unprocessed state refers to a state when data is not acquired by a processing process in a data source, and when an abnormality occurs in the processing process, the processing state of the data is updated from other states to the unprocessed state; the second is a state to be processed, wherein the state to be processed refers to a state that data is loaded by a processing process and is waiting for the processing process to process the data; the third is an in-process state, which refers to a state when a piece of data is being processed by a processing process, for the piece of data; the fourth is a processing completion state, which refers to a state after the data is processed by the processing process, as the name implies. The data state of the data can be a data state of each piece of data by means of carrying out state marking on each piece of data.

The group identifier corresponding to each data group can be stored locally, and the local can be a cache, a network disk and the like. The storage locally is not limited to the group identification, and may include information such as processing time for processing the group data.

Specifically, in order to send the data sets to the corresponding processing processes, each data set is allocated to a corresponding processing process in the processing process set, including:

Specifically, group marking each data set includes group marking each piece of data in the data set. The corresponding relation between the data group and the processing process can be processed by marking the data in the data group with the group identifier corresponding to the processing process, and transmitting the data with the same identifier as the corresponding identifier of the processing process to the processing process through the group identifier. The processing process processes each data according to time sequence when processing the data. The data set, the data line lock, the lock-free algorithm (CAS) And the like can be used for ensuring that each data is only monopolized by one processing thread, and avoiding that other processing threads read dirty so as to repeatedly process the same data.

103. Processing the data in the target data group according to the processing progress and the data state of the data in the distributed target data group; and synchronizing the processing result of the processing process to the monitoring process of the processing process of the monitoring process.

After receiving the data in each data group, each processing process determines the state of each data, and only receives and processes the data in the data group into unprocessed data. The meaning of the pending state is that the data in the data set has been sent to the corresponding processing process awaiting processing by the processing process.

Wherein. The processing result includes the current data state of each piece of data. It can be understood that when the processing process processes the data in the data set, the processing result of the data is synchronized to the monitoring process, the processing result may be the data state of the data, and the monitoring process may record the data state of the data in the data set corresponding to the processing process. For example: the data group comprises data 1, data 2 and data 3, after the processing process finishes processing the data 1, the data state of the data 1 is updated to a processing completion state and is synchronized to the monitoring process, the monitoring process records the data state of the data 1 in the data group as the processing completion state, the record can be stored in a record file at the monitoring side, and the record file can also be stored in a cache, a network disk and the like.

104. When detecting that the processing process is abnormal, triggering the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on the monitoring process and the monitoring side record file.

Specifically, according to the abnormal conditions actually occurring in the process, the anomalies are classified into two categories: the first category is to deal with the process itself exception, and specifically includes: the processing process crashes, is manually closed or exits the processing process when the processing process processes data; the second type is communication abnormality, that is, abnormality due to network reasons such as disconnection of communication between a processing process and a listening process. When the abnormality occurs, the monitoring process can be triggered to search whether the unprocessed data exists in the group identifier of the data group corresponding to the abnormal processing process according to the group identifier of the local record. The specific determination mode is as follows: the monitoring process searches data in the data group corresponding to the abnormal processing process from the data source, and then finds data with a data state being out of the processing completion state by combining the monitoring side record file generated by the monitoring processing process. I.e., the verification of the data state of each data in the data set corresponding to the exception handling process. If the data state of the data in the data group is the processing completion state, determining that the data is completed, and if the data state of the data is the processing state and the to-be-processed state, determining that the data is unprocessed. For example: the data group corresponding to the abnormal processing process comprises data 1, data 2 and data 3, the data state of the data 1 is a processing completion state, the data states of the data 2 and the data 3 are to-be-processed states, and when the abnormality occurs, the data 2 and the data 3 can be determined to be unprocessed, so that the data 2 and the data 3 are determined to be unprocessed data in the data group corresponding to the abnormal processing process; or data 1 is in-process state, data 2 and data 3 are in pending state, and data 1, data 2 and data 3 are determined as unprocessed state. If an exception occurs while processing data 2, the data state is also determined to be unprocessed data for data 2 under processing, so that unprocessed data is determined from the target data set based on the current data state of the data in the target data set, including:

And determining the first data with the data state being the in-process state and the second data with the data state being the to-be-processed state in the data group corresponding to the abnormal processing process as unprocessed data.

105. And updating the data state of the unprocessed data in the data source into an unprocessed state through the monitoring process.

When the monitoring process determines the unprocessed data, the data state of the unprocessed data is updated from the state to be processed or the state in process to the unprocessed state in the data source.

106. When the abnormal processing process is detected not to be recovered to be normal within the preset time period, the unprocessed data is distributed to other processing processes for processing, and the group identification of the unprocessed data is adjusted to the group identification of the data group corresponding to the other processing processes.

Specifically, in order to avoid that the abnormal processing process cannot be recovered for a long time, so that unprocessed data in a data set corresponding to the abnormal processing process cannot be processed, a preset time period can be set, whether the abnormal processing process is recovered within the preset time period is judged, if yes, the unprocessed data is distributed to the processing process for recovering the normal processing process to continue processing, the data state of the unprocessed data is updated to be in a to-be-processed state, and the unprocessed data is reprocessed based on the processing process and the updated data state of the unprocessed data. Thus, the method further comprises:

If the abnormal processing process does not recover to be normal within a preset time period, the unprocessed data is distributed to other processing processes for processing, wherein after the unprocessed data is distributed to the other processing processes, the other processing processes can adjust the group identification of the processed data to the group identification of the data group corresponding to the other processing processes; it is understood that the other processing processes are processing processes other than the exception processing process in the processing process group.

Reprocessing the unprocessed data based on other processing procedures and the updated data state of the unprocessed data.

107. In the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identification of the unprocessed data

Specifically, the preset exception type is communication exception, when the exception type of the exception handling process is communication exception, the exception handling process is in a state capable of handling data at the moment, when the exception occurs, the exception handling process is handling a certain piece of data, and based on a handling mechanism, the piece of data is handled after the exception occurs, but because the communication between the exception handling process and the monitoring process is not recovered, unprocessed data may be already distributed to other handling processes, so that the same piece of data is handled by the exception handling process and the other handling processes, namely repeated handling is caused. Therefore, in the process of distributing unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type, and the abnormal processing process is recovered to be normal, the current group identification of the unprocessed data needs to be checked;

when an exception occurs, the exception processing process is processing a certain piece of data, the current group identifier of the piece of data and the group identifier corresponding to the exception processing process can be compared, namely, the second data can be distributed to other processing processes, and the first data determines whether to process the piece of data or not through the group identifier. Therefore, in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is recovered to be normal, checking the current group identifier of the unprocessed data includes:

And in the process of distributing the unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the abnormal processing process.

108. If the verification is not passed, the exception handling process is terminated to continue to process the unprocessed data, and other handling processes process the unprocessed data.

Specifically, if the verification fails, it indicates that the unprocessed data has been allocated to another processing process, and the exception processing process is terminated to continue processing the unprocessed data, and the other processing process processes the unprocessed data.

And when the current group identification of the first data is different from the group identification corresponding to the abnormal processing process, terminating the abnormal processing process to process the first data, and processing the first data by other processing processes. Of course, since the group identification of the first data has been changed, it is explained that the group identification of the second data is also changed at the same time, and the second data is also allocated to other processing processes for processing.

In addition, since the processing process performs group identification on the data set before processing the data in the data set, if the group identification check passes, it indicates that the unprocessed data is not allocated to other processing processes, and the exception processing process can continue to process the unprocessed data. Thus, the method further comprises:

if the verification is passed, the unprocessed data is refused to be distributed to other processing processes, and the abnormal processing process is triggered to continue to process the unprocessed data. The method can also be as follows: when the current group identifier of the first data is the same as the group identifier corresponding to the abnormal processing process, the first data is refused to be distributed to other processing processes, and the abnormal processing process is triggered to continue processing the first data.

Specifically, after the processing process completes the data processing in the target data set, the data is sent to the downstream system in a form of a set or a stripe, and if the downstream system is unavailable for some reason, multiple retries are performed to ensure that the processed data is not missed when sent to the downstream system. And after each data group is processed, the processing process can delete the group identifier stored locally, and the monitoring side can delete the record information about the data group in the record file of the monitoring side at the same time.

As can be seen from the foregoing, in the embodiment of the present application, when data in a data set is sent to a corresponding processing process to perform processing, a group identifier of each data set is stored locally, in the processing process, a processing result of each piece of data is synchronized to a listening process, and if the processing process is abnormal, it is ensured that data that is not processed by the abnormal processing process can be found according to local and listening side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing by a monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type, and the exception processing process is recovered to be normal, checking the current group identification of the unprocessed data, if the checking is passed, the unprocessed data is not distributed to the other processing processes, the processing can be continued by the exception processing process, if the checking is not passed, the unprocessed data is distributed to the other processing processes, the exception processing process does not need to process the unprocessed data, repeated processing is avoided, and processing integrity is improved.

In some embodiments, taking a data processing apparatus as an example on a single server basis, as shown in fig. 2, the data processing method includes:

201. the server acquires a plurality of data to be processed from a data source and divides the plurality of data into a plurality of data groups.

The data source stores a plurality of data, and the data can comprise request data sent by a user or data corresponding to the request data. The plurality of data to be processed may be collectively acquired in the form of a set, for example, the data set { data 1, data 2 … … data n }.

The processing server is provided with a processing process group comprising a plurality of processing processes, and after receiving a plurality of data, the processing server divides the plurality of data to be processed into a plurality of data groups according to the plurality of processing processes in the processing process group.

The dividing manner may divide a plurality of data to be processed by a total number of processing processes or a number of processing processes in idle, for example: the total number of processing processes or the number of processing processes in idle is 3, dividing n data to be processed into 3 data groups equally.

202. And the server marks each data group to obtain a group identifier corresponding to each data group.

Specifically, when each data group is marked in a group mode, the group is marked corresponding to a plurality of processing processes in the processing process group, and each data can be ensured to be exclusive by only one processing thread by using a data set, a data line lock, a lock-free algorithm (CAS) And the like, so that other processing threads are prevented from dirty reading And repeatedly processing the same data.

203. The server determines a target process identifier corresponding to the group identifier from the process identifier set.

It will be appreciated that in order to determine that each group-tagged data group may be assigned to a corresponding process, a process identification set may be pre-established, which may include a process identification for each process, for example: if the process group includes process 1 and process 2 … … and process n, the process identifier set includes {1, 2 … … n }, and in step 203, the process group of each data group may be marked, where 1, 2 … … n in the process identifier set may be directly marked to the corresponding data group, or the identifier corresponding to the process identifier may be marked to the data group, which is not limited herein.

204. The server distributes the data group corresponding to the group identifier to the processing process corresponding to the target process identifier, and saves the group identifier of each data group to the local.

In step 204, it can be known that, since the process identifier corresponding to the process identifier or the identifier corresponding to the process identifier is marked to the data group when the group identifier is marked on each data group, the processing process corresponding to each data group can be found according to the group identifier obtained by the group identifier and the target process identifier, and the data in the data group can be distributed to the processing process corresponding to the target process identifier.

The group identifier corresponding to each data group can be stored locally, and the local can be a cache, a network disk and the like. The storage to the local is not limited to the group identification, and can also comprise information such as processing time for processing the group data

205. The server determines target data currently required to be processed by the processing process from the target data group.

It will be appreciated that after the data sets are assigned to the corresponding processing processes, each data process in the data set needs to be verified to determine whether the data set is the target data that the corresponding processing process currently needs to process.

The method specifically comprises the following steps: step 206, the server obtains the data processing expected information corresponding to the processing procedure and the data attribute information of the target data. Wherein the data processing desired information may include: information such as a group identification of a data group that the processing process desires to process, a data state of data that the processing process desires to process, whether the data state of data is rolled back, and the like; the data attribute information may include: information of actual group identification of data, data state, whether the data state is rolled back, etc., wherein the rollback is: the data state is updated from the other state to unprocessed.

Step 207, the server verifies the target data based on the data processing expected information and the data attribute information. If the verification is passed, step 208 is performed.

Specifically, the verification method may be to compare the data processing expected information with the data attribute information, and compare the group identifier of the data group expected to be processed by the processing process with the actual group identifier of the data in the data attribute information; the data state of the data expected to be processed by the processing process and the actual data state of the data in the data attribute information; the processing process expects a series of comparisons of whether the processed data is rolled back with whether the data is actually rolled back in the data attribute information. Step 208 is performed only if each verification condition must be passed.

Step 208, the server triggers a processing process to process the target data, and updates the data state of the corresponding data in the data source.

When the verification is passed, the target data is the data which is required to be processed currently by the processing process, and the data state of the target data in the data source is updated. The updated data state is the pending state, meaning that the pending process is processing the data. The updated data state may also be notified to the data source, and the data source updates the data state of the data, which is not described herein. If the verification is not passed, the data is ignored if the processing process is not required to process the data.

Step 209, synchronizing the processing result of the processing procedure to the listening process of the processing procedure of the listening process.

Wherein. The processing result includes a data state of the processed data. It can be understood that after each processing process completes one piece of data, the processing process synchronizes the processing result of the piece of data to the monitoring process, the processing result can be the data state of the piece of data, and the monitoring process can record the data state of the piece of data in the data group corresponding to the processing process. For example: the data group comprises data 1, data 2 and data 3, after the processing process finishes processing the data 1, the data state of the data 1 is updated to a processing completion state and is synchronized to the monitoring process, the monitoring process records the data state of the data 1 in the data group as the processing completion state, the record can be stored in a record file at the monitoring side, and the record file can also be stored in a cache, a network disk and the like.

And 210, when the server detects that the processing process is abnormal, triggering the monitoring process to determine that the data state in the data group corresponding to the abnormal processing process is first data in a processing state and the data state is second data in a to-be-processed state as unprocessed data based on the monitoring side record file.

Specifically, according to the abnormal conditions actually occurring in the process, the anomalies are classified into two categories: the first category is to deal with the process itself exception, and specifically includes: the processing process crashes, is manually closed or exits the processing process when the processing process processes data; the second type is communication abnormality, that is, abnormality due to network reasons such as disconnection of communication between a processing process and a listening process. When the abnormality occurs, the monitoring process can be triggered to search whether the unprocessed data exists in the group identifier of the data group corresponding to the abnormal processing process according to the group identifier of the local record. The specific determination mode is as follows: the monitoring process searches data in the data group corresponding to the abnormal processing process from the data source, and then finds data with a data state being out of the processing completion state by combining the monitoring side record file generated by the monitoring processing process. I.e., the verification of the data state of each data in the data set corresponding to the exception handling process. If the data state of the data in the data group is the processing completion state, determining that the data is completed, and if the data state of the data is the processing state and the to-be-processed state, determining that the data is unprocessed. For example: the data group corresponding to the abnormal processing process comprises data 1, data 2 and data 3, the data state of the data 1 is a processing completion state, the data states of the data 2 and the data 3 are to-be-processed states, and when the abnormality occurs, the data 2 and the data 3 can be determined to be unprocessed, so that the data 2 and the data 3 are determined to be unprocessed data in the data group corresponding to the abnormal processing process; or data 1 is in-process state, data 2 and data 3 are in pending state, and data 1, data 2 and data 3 are determined as unprocessed state. If an exception occurs while processing data 2, the data 2 in the data state being processed is also determined as unprocessed data

It will be appreciated that as each piece of data is processed, the processing process will update the data whose data state is pending to an in-process state, indicating that the piece of data is being processed. If the processing progress is abnormal, if the data in the processing state is not processed, determining the first data in the data state of being processed and the second data in the data state of being to-be-processed as unprocessed data.

If no exception occurs in the processing procedure, after the processing procedure finishes processing the data in the data set, the data state of the data in the data set is updated to be completed in the data source, and the information stored locally in step 203 is deleted.

Step 211, updating the data state of the unprocessed data in the data source to the unprocessed state through the monitoring process.

When the monitoring process determines the unprocessed data, the data state of the unprocessed data is updated from the state to be processed or the state in process to the unprocessed state in the data source. Specifically, the monitoring process may also be instructed to update the data state of the unprocessed data at the data source, and the number of times of changing the data state of the data to the unprocessed state may also be counted, that is, the rollback number of times is changed to the unprocessed state once, and the rollback number of times is increased by 1.

Step 212, the server judges whether the processing process is recovered to be normal within a preset time period, if so, step 213 is executed; if not, go to step 215.

However, in order to ensure the processing efficiency of the data, it is impossible to wait for the recovery of the abnormal processing process and then to process the corresponding data. Therefore, a preset time period can be preset, and whether the processing process is normal or not is judged in the preset time period. The preset time period can be 2s, 1min and the like, and different times are set according to different application scenes.

Step 213, the server distributes unprocessed data to the processing procedure; and updating the data state of the unprocessed data into a state to be processed.

When the abnormal processing process is restored to be normal within the preset time period, the unprocessed data of the abnormal processing process can be redistributed to the processing process for processing, and after the unprocessed data is distributed, the data state of the unprocessed data is updated from the unprocessed state to the to-be-processed state again.

Step 214, the server reprocesses the unprocessed data based on the processing procedure and the updated data state of the unprocessed data.

Specifically, the unprocessed data is redistributed, and the unprocessed data is reprocessed according to the updated data state of the unprocessed data. Raw data may also be retrieved from the new slave data sources and reassigned.

In step 215, the server distributes the unprocessed data to other processing processes, and updates the data state of the unprocessed data to the pending state.

Specifically, if the abnormal processing process does not recover to be normal within a preset time period, the unprocessed data is distributed to other processing processes for processing, wherein after the unprocessed data is distributed to the other processing processes, the other processing processes can adjust the group identifier of the processed data to the group identifier of the data group corresponding to the other processing processes; it is understood that the other processing processes are processing processes other than the exception processing process in the processing process group.

And step 216, in the process of distributing unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the abnormal processing process.

Specifically, the preset exception type is communication exception, when the exception type of the exception handling process is communication exception, the exception handling process is in a state capable of handling data at the moment, when the exception occurs, the exception handling process is handling a certain piece of data, and based on a handling mechanism, the piece of data is handled after the exception occurs, but because the communication between the exception handling process and the monitoring process is not recovered, unprocessed data may be already distributed to other handling processes, so that the same piece of data is handled by the exception handling process and the other handling processes, namely repeated handling is caused. When an abnormality occurs, the abnormal processing process is processing a certain piece of data, the current group identification of the piece of data and the group identification corresponding to the abnormal processing process are compared, namely, the second data can be distributed to other processing processes, and the first data determines whether to process the piece of data or not through the group identification.

And step 217, when the current group identification of the first data is different from the group identification corresponding to the exception handling process, terminating the exception handling process to handle the first data, and processing the unprocessed data by other handling processes.

Specifically, if the comparison result is that the current group identifier of the first data is different from the group identifier corresponding to the abnormal processing process, the first data is already allocated to other processing processes, the abnormal processing process is terminated to continue processing the unprocessed data, and the other processing processes process processes the unprocessed data.

Step 218, when the current group identifier of the first data is the same as the group identifier corresponding to the exception handling process, the first data is refused to be allocated to other handling processes, and the exception handling process is triggered to continue to process the first data.

Specifically, if the comparison result shows that the current group identifier of the first data is the same as the group identifier corresponding to the abnormal processing process, it indicates that the first data is not allocated to other processing processes, then the first data is refused to be allocated to other processing processes, and the abnormal processing process is triggered to continue processing the first data.

If other processing processes are processing other data and receive unprocessed data at the same time, the unprocessed data may be processed preferentially, or the unprocessed data may be processed after the other data processing is completed, which is not limited herein.

Taking the data processing device integrated in the data processing server and the monitoring server, that is, the data processing server and the monitoring server as examples, referring to fig. 3a, the embodiment of the application further provides a data processing system, which includes: data source 10, processing server 20, terminal 30, and listening server 40. The data source 10 may also be a third party service or a database provided in the terminal 30, which is not limited herein. The terminal 30 may be a mobile phone, a tablet computer, a notebook computer, etc., and fig. 3a is an example of the mobile phone as the terminal 30. The terminal 30 may have applications required by various users, such as applications with entertainment functions (e.g., video applications, audio playback applications, game applications, reading software), and applications with service functions (e.g., map navigation applications, dining applications, etc.). The listening server 40 is used to listen to the processing procedure of the processing server 20, and the listening server 40 and the processing server 20 may be connected through a network protocol (TCP/IP). The processing server 20 is connected to the listening server 40 before processing the data, or may be connected to the listening server 40 regardless of whether the processing server 20 processes the data, and when the processing server 20 is disconnected from the listening server 40, the processing server 20 no longer processes the data that was not processed. The processing server 20 includes a processing process group including a plurality of processing processes, the listening server 40 includes a listening process group including a plurality of listening processes, and the connection between the processing processes and the listening processes may be one-to-one, one-to-many, or many-to-many.

Specifically, the processing server 20 acquires a plurality of data to be processed from the data source 10, and divides the plurality of data into a plurality of data groups; distributing each data group to a corresponding processing process in a processing process group, storing the group identification of each data group to a local place, and updating the data state of the data in a data source to a state to be processed; processing data in the target data group according to the processing progress and the data state of the data in the distributed target data group, and updating the data state of corresponding data in the data source 10 and synchronizing the processing result of the processing progress to the monitoring server 40, wherein the processing result comprises the data state of the processed data; when the processing server 20 detects that the processing process is abnormal, the monitoring server 40 is triggered to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file comprising the synchronized data state and the group identification of the monitoring side, the data state of the unprocessed data in the data source 10 is updated to be an unprocessed state by the monitoring server 40, and the processing server 20 reprocesses the unprocessed data based on the updated data state of the unprocessed data.

Thus, referring to fig. 3b, the data processing method of the present application further includes:

step 301, a processing server obtains a plurality of data to be processed from a data source, and divides the plurality of data into a plurality of data groups.

Step 302, each data group is allocated to a corresponding processing process in the processing process group, and the data state of the data in the data source is updated to be in a to-be-processed state.

And 303, processing the data in the target data group according to the processing process and the data state of the data in the allocated target data group, and updating the data state of the corresponding data in the data source and synchronizing the processing result of the processing process to the monitoring server.

After receiving the data in each data group, each processing process determines the state of each data, and only receives and processes the data in the data group into unprocessed data. The meaning of the pending state is that the data in the data set has already been sent to the corresponding processing process waiting for the processing of the processing process.

After the processing process finishes processing one piece of data, the data source updates the data state of the piece of data into a processing completion state, and synchronizes the processing result of the piece of data to the monitoring server, wherein the processing result can comprise: the data state of the processed data.

And 304, when detecting that the processing process is abnormal, determining unprocessed data in a data group corresponding to the abnormal processing process.

When the processing server detects that the processing process is abnormal, the processing server determines which data in the data group corresponding to the abnormal processing process are not processed. Similarly, when the monitoring server detects that an abnormality occurs, the monitoring server also determines unprocessed data in the data group corresponding to the abnormal processing process. Specifically, the determining mode can determine whether the data in the data group is processed or not through the processing result of the synchronization of the processing server to the monitoring server.

For example: the monitoring server monitors the data 1 and the data 2 … … data n in the data group A, when the processing progress is abnormal, the monitoring server checks the state information of each data, if the data 1 is the processing completion state, the data 2 is the processing state … …, and the data n is the to-be-processed state, the data 2 to the data n are determined as the unprocessed data.

Therefore, specifically, when detecting that an abnormality occurs in a processing process, determining unprocessed data in a data group corresponding to the abnormal processing process includes:

when the abnormal processing progress is detected, triggering the monitoring server to determine unprocessed data in a data group corresponding to the abnormal processing progress based on the synchronized data state.

Specifically, determining unprocessed data in the data set corresponding to the exception handling process includes:

Step 305, updating the data state of the unprocessed data in the data source to an unprocessed state through a listening process.

When the listening server determines the unprocessed data, the data state of the data is updated to the unprocessed state in the data source.

And 306, in the process of distributing the unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is communication abnormality and the abnormal processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the abnormal processing process. If not, go to step 307; if so, step 308 is performed.

Specifically, the abnormal communication refers to the disconnection of network communication between the processing server and the listening server, that is, the TCP/IP disconnection, and since the processing process in the processing server has no problem, only the network between the processing server and the listening process is disconnected, in the process from the disconnection to the restoration of the normal data processing progress in the second preset time period, if the processing process is processing the data in the data state in the process during the disconnection, the processing of the data is continued, and the subsequent data in the data state to be processed is not processed. Therefore, the first data with the data state in the processing state and the second data with the data state in the pending state in the data group corresponding to the abnormal processing process are determined as unprocessed data, and then the current group identifier of the first data is compared with the group identifier corresponding to the abnormal processing process to determine whether the first data is allocated to other processes for processing. The second preset time period is longer than the first preset time period, that is, the exception handling process is not recovered in the first preset time period and is recovered in the second preset time period.

Step 307, when the current group identifier of the first data is different from the group identifier corresponding to the exception handling process, terminating the exception handling process to handle the first data.

It will be understood that if the current group identifier of the first data is different from the group identifier corresponding to the exception handling process, it is indicated that the group identifier of the first data has been updated by another process, and in order to avoid that the exception handling process and the other handling process repeat the processing of the first data, the exception handling process is terminated to continue the processing of the first data.

Step 308, when the current group identifier of the first data is the same as the group identifier corresponding to the exception handling process, refusing to distribute the first data to other handling processes, and triggering the exception handling process to continue to handle the first data process.

It can be understood that if the current group identifier of the first data is the same as the group identifier corresponding to the exception handling process, it is indicated that the group identifier of the first data is not updated by other processes, and the exception handling process recovered in the second preset time period is triggered to continue to process the first data process.

Specifically, when the processing process completes the data processing in the data group, the data is sent to the downstream system in the form of a group or a stripe, and if the downstream system is unavailable for some reason, multiple retries are performed to ensure that the processed data is not missed when sent to the downstream system.

The interaction method of the processing server, the listening server and the data source is shown in fig. 4a and fig. 4b.

In the data loading phase: before processing each group of data, the processing server stores the group identification of the data group locally, the monitoring server of the monitoring processing server and the data source synchronize the group identification of the data group, and the data source returns the group of data to the processing server according to the group identification;

in the data processing stage: the processing server processes after receiving the data, and synchronizes the processing result to the monitoring server and the data source when processing one piece of data, and the processing server and the monitoring server can update the data state of the processed data into the processing completion state in the data source; judging whether all data in the current data group are processed, if not, continuing processing, and if so, deleting the group identifier stored to the local, and ending the processing;

in the abnormal rollback phase: if the processing process is abnormal, the monitoring server finds the group identification of the data group which is processed correspondingly by the abnormal processing process, and updates the data state of the data which is not processed by the abnormal processing process into an unprocessed state in the data source;

In the data loading phase: the processing server acquires unprocessed data again from the data source, circularly verifies the group identification and the data state of each piece of data in the data group, judges whether the data is processed, skips the piece of data if the current data is processed, continues judging whether other data is processed, processes the piece of data if the current data is not processed, synchronizes the processing result of the piece of data to the monitoring server, and updates the data state of the piece of data in the data source to be processed. If all the data in the data group corresponding to the abnormal process are processed, the monitoring server processes the data group corresponding to the record group identification, and the processing is finished.

For a clearer understanding of the data processing method described herein, please refer to fig. 5a, 5b, 5c and 5d, wherein fig. 5a to 5d illustrate the data processing method from the point of view of the processing procedure and the listening procedure.

Wherein, BATCH is the group identification, disconnection is the processing progress abnormal, monitor progress A is used for monitor processing progress A, processing progress B is other processing progress, monitor progress B is other monitor progress.

Specifically, as shown in fig. 5a, when an exception occurs in the processing process a, the listening process a that listens to the processing process a will rollback the data with the group identified as a and the data state to be processed. If the processing process B loads unprocessed data and updates the BATCH=a to be BATCH=b, if the abnormal processing process is recovered to be normal within the second preset time period, the processing process a needs to be connected to a monitoring process, specifically, taking the connection to the monitoring process B as an example, after the monitoring process B establishes a connection with the processing process a, the processing process a triggers the monitoring process B to roll back the data of BATCH=a according to the information such as the local group identifier and the like, and since the processing process B has loaded and changed the group identifier from BATCH=a, the group identifier of the unprocessed data at this time is B, so that the number of lines actually affected by the roll-back operation at this time of the monitoring process B is zero. When the processing process A judges that the current group identifier of the unprocessed data and the group identifier of the corresponding data group, the processing process A does not process the unprocessed data any more because the group identifier of the unprocessed data is updated to B, the processing process B processes the unprocessed data, and after the processing process B finishes the data processing, the data state of the data is updated to a processing completion state.

Similarly, as shown in fig. 5b, when an exception occurs in the processing process a, the listening process a that listens to the processing process a will rollback the data with the group identified as a and the data state to be processed. If the processing process a is reconnected first, the processing process a triggers the monitoring process B to roll back the data of the BATCH=a according to the information such as the group identifier stored locally, because the data state of the unprocessed data is rolled back to the unprocessed state before, and the data state of the data which the monitoring process B expects to roll back is the to-be-processed state, therefore, because the data state is not matched, the number of lines actually affected by the rolling operation of the monitoring process B is zero, if the unprocessed data is still acquired by the processing process B, the group identifier of the unprocessed data is updated to BATCH=b, because the group identifier of the unprocessed data is already updated to B, the processing process a does not process the unprocessed data any more, and the processing process B processes the unprocessed data, and after the processing process B completes the data processing, the data state of the data is updated to the processing completion state, and after the processing process B completes the data processing the data state is updated to the processing completion state.

Similarly, as shown in fig. 5c, when an exception occurs in the processing process a, the listening process a that listens to the processing process a will rollback the data with the group identified as a and the data state to be processed. If the processing process B loads unprocessed data and updates the BATCH=a to be BATCH=b, if the abnormal processing process is recovered to be normal within the second preset time period, the processing process a needs to be connected to a monitoring process, specifically, taking the connection to the monitoring process B as an example, after the monitoring process B establishes a connection with the processing process a, the processing process a triggers the monitoring process B to roll back the data of BATCH=a according to the information such as the local group identifier and the like, and since the processing process B has loaded and changed the group identifier from BATCH=a, the group identifier of the unprocessed data at this time is B, so that the number of lines actually affected by the roll-back operation at this time of the monitoring process B is zero. When the processing process A judges that the current group identifier of the unprocessed data and the group identifier of the corresponding data group, the processing process A does not process the unprocessed data any more because the group identifier of the unprocessed data is updated to B, the processing process B processes the unprocessed data, and after the processing process B finishes the data processing, the data state of the data is updated to a processing completion state.

Similarly, as shown in fig. 5d, when an exception occurs in the processing process a, the listening process a that listens to the processing process a will rollback the data with the group identified as a and the data state to be processed. If the processing process a is reconnected first, the processing process a triggers the monitoring process B to roll back the data of the BATCH=a according to the information such as the local group identifier, and the processing process a will not process the unprocessed data any more because the data state of the unprocessed data is rolled back to the unprocessed state before, and the data state of the data which the monitoring process B expects to roll back is the to-be-processed state, so the number of lines actually affected by the rolling operation of the monitoring process B is zero because of the unmatched data, if the unprocessed data is still acquired by the processing process B, the group identifier of the unprocessed data is updated to BATCH=b, and the processing process a will not process the unprocessed data any more because the group identifier of the unprocessed data has been updated to B, and after the processing process B completes the processing of the data, the data state of the data is updated to the processing completion state.

If the processing process is to be performed with the processing process, the processing process may be configured to perform the processing of the data after the processing process is performed with the processing process.

As can be seen from the foregoing, in the embodiment of the present application, when data in a data set is sent to a corresponding processing process to perform processing, a group identifier of each data set is stored locally, in the processing process, a processing result of each piece of data is synchronized to a listening process, and if the processing process is abnormal, it is ensured that data that is not processed by the abnormal processing process can be found according to local and listening side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing by a monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type, and the exception processing process is recovered to be normal, checking the current group identification of the unprocessed data, if the checking is passed, the unprocessed data is not distributed to the other processing processes yet, the processing can be continued by the exception processing process, if the checking is not passed, the unprocessed data is distributed to the other processing processes, the exception processing process does not need to process the unprocessed data, the repeated processing is avoided, the processing integrity is improved, and the data which the processing process expects to process can be screened out through the verification of the data processing expected information and the data attribute information, so that unnecessary data is avoided to be processed.

In order to facilitate better implementation of the data processing method provided in the embodiments of the present application, in an embodiment, a data processing apparatus is also provided. Where the meaning of a noun is the same as in the data processing method described above, specific implementation details may be referred to in the description of the method embodiments.

In some embodiments, there is also provided a data processing apparatus, as shown in fig. 6a and 6b, which may include: a data dividing unit 401, a first data distributing unit 402, a first data processing unit 403, a data determining unit 404, a data updating unit 405, a second data distributing unit 406, a data verifying unit 407, and a termination processing unit 408.

A data dividing unit 401, configured to obtain a plurality of data to be processed from a data source, and divide the plurality of data into a plurality of data groups;

a first data allocation unit 402, configured to allocate each data group to a corresponding processing process in the processing process group, store a group identifier of each data group locally, and update a data state of the data in the data source to a to-be-processed state;

a first data processing unit 403, configured to process data in the target data set according to the processing procedure and the data state of the data in the allocated target data set; and synchronizing the processing result of the processing process to a monitoring process monitoring the processing process, wherein the processing result comprises the data state of the processed data;

The data determining unit 404 is configured to trigger the monitoring process to determine unprocessed data in a data set corresponding to the abnormal processing process based on a monitoring side record file when an abnormality occurs in the processing process, where the monitoring side record file includes a synchronized data state and a group identifier of the monitoring side;

a data updating unit 405, configured to update, by a listening process, a data state of the unprocessed data in the data source to an unprocessed state;

and the second data allocation unit 406 is configured to allocate the unprocessed data to other processing processes for processing when it is detected that the abnormal processing process is not restored within the preset time period, and adjust the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing processes, where the other processing processes are processing processes in the processing process group other than the abnormal processing process.

The data verification unit 407 is configured to, in a process of distributing the unprocessed data to other processing processes, verify a current group identifier of the unprocessed data if the exception type of the exception processing process is a preset exception type and the exception processing process is restored to be normal.

And a termination processing unit 408, configured to terminate the exception processing process to continue processing the unprocessed data if the check fails, and process the unprocessed data by the other processing process.

In some embodiments, the data processing apparatus further comprises:

a third data allocation unit 409 configured to allocate the unprocessed data to the abnormal processing process when it is detected that the abnormal processing process is restored to normal within a preset period of time; updating the data state of the unprocessed data into a state to be processed;

the data processing unit 410 is configured to reprocess the unprocessed data based on the exception processing procedure and the updated data status of the unprocessed data.

In some embodiments, the first data allocation unit 402 may include:

a marking subunit 4021, configured to perform group marking on each data group to obtain a group identifier corresponding to each data group;

a first determining subunit 4022, configured to determine, from a process identifier set, a target process identifier corresponding to the group identifier, where the process identifier set includes a process identifier of a processing process in the processing process group;

An allocation subunit 4023, configured to allocate a data group corresponding to the group identifier to a processing process corresponding to the target process identifier.

In some embodiments, the second data allocation unit 406 may further include:

the data adjustment subunit 4061 is configured to update the data state of the unprocessed data to a to-be-processed state, and adjust the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing process.

The data processing subunit 4062 is configured to trigger other processing procedures to process the unprocessed data according to the data state and the group identifier of the unprocessed data.

In some embodiments, the data determination unit 404 may further include:

the second data determining subunit 4041 is configured to trigger the listening process to determine, based on the listening-side record file, that the data state in the data set corresponding to the abnormal processing process is the first data in the in-process state, and that the data state is the second data in the pending state, as unprocessed data.

In some embodiments, the second data allocation unit 406 may further include:

a data comparison subunit 4063, configured to compare, in a process of allocating the unprocessed data to other processing processes, the current group identifier of the first data with the group identifier corresponding to the exception processing process if the exception type of the exception processing process is a preset exception type and the exception processing process is restored to be normal

The termination processing unit 408 may further include:

and a termination processing subunit 4081, configured to terminate the exception processing process to process the first data when the current group identifier of the first data is different from the group identifier corresponding to the exception processing process, and process the unprocessed data by the other processing processes.

And the rejecting allocation subunit 4082 is configured to reject allocation of the first data to the other processing process and trigger the exception processing process to continue processing the first data when the current group identifier of the first data is the same as the group identifier corresponding to the exception processing process.

In some embodiments, the first data processing unit 403 may further include:

a third determining subunit 4031, configured to determine, from the target data set, target data that needs to be processed currently by the processing procedure.

An obtaining subunit 4032, configured to obtain data processing expected information corresponding to the processing procedure, and data attribute information of the target data, where the data processing expected information includes a data set expected to be processed by the processing procedure and a data state expected to be processed, and the data attribute information includes a current data state of the target data and a data set to which the target data currently belongs.

And a verification subunit 4033, configured to verify the target data based on the data processing expected information and the data attribute information.

The triggering subunit 4034 is configured to trigger a processing procedure to process the target data when the verification passes.

In the implementation, each unit may be implemented as an independent entity, or may be implemented as the same entity or several entities in any combination, and the implementation of each unit may be referred to the foregoing method embodiment, which is not described herein again.

As can be seen from the above, the data processing method in the embodiment of the present application adopts the data dividing unit 401 to obtain a plurality of data to be processed from the data source, and divide the plurality of data into a plurality of data groups; the first data allocation unit 402 allocates each data group to a corresponding processing process in the processing process group, saves the group identifier of each data group to a local place, and updates the data state of the data in the data source to a to-be-processed state; the first data processing unit 403 processes the data in the target data group according to the processing process and the data state of the data in the allocated target data group; and synchronizing the processing result of the processing process to a monitoring process monitoring the processing process, wherein the processing result comprises the data state of the processed data; when detecting that the processing process is abnormal, the data determining unit 404 triggers the monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identifier of a monitoring side; the data updating unit 405 updates the data state of the unprocessed data in the data source to an unprocessed state through a listening process; when the second data distribution unit 406 detects that the abnormal processing process does not recover from the normal state within a preset time period, the unprocessed data is distributed to other processing processes for processing, and the group identifier of the unprocessed data is adjusted to the group identifier of the data group corresponding to the other processing processes; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type and the exception processing process is restored to be normal, the data verification unit 407 verifies the current group identifier of the unprocessed data; if the check is not passed, the termination processing unit 408 continues the processing of the unprocessed data by the termination exception processing process, and processes the unprocessed data by other processing processes. The missing data in the processing process can be avoided, each piece of data is guaranteed to be processed, and the integrity of the processed data is improved.

Referring to fig. 7, embodiments of the present application provide a server 800 that may include one or more processors 801 of a processing core, one or more memories 802 of a computer readable storage medium, radio Frequency (RF) circuitry 803, a power supply 804, an input unit 805, and a display unit 806. Those skilled in the art will appreciate that the server structure shown in fig. 1a and 3a is not limiting of the server and may include more or fewer components than shown, or may combine certain components, or may be arranged in different components. Wherein:

the processor 801 is a control center of the server, connects various parts of the entire server using various interfaces and lines, and performs various functions of the server and processes data by running or executing software programs and/or modules stored in the memory 802, and calling data stored in the memory 802. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor that primarily handles operating systems, user interfaces, applications, etc., with a modem processor that primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 801.

The memory 802 may be used to store software programs and modules, and the processor 801 executes various functional applications and data processing by executing the software programs and modules stored in the memory 802.

The RF circuitry 803 may be used to receive and transmit signals during the transmission and reception of information.

The server also includes a power supply 804 (e.g., a battery) for powering the various components, which may be logically connected to the processor 801 by a power management system, such as to perform charge, discharge, and power consumption management functions.

The server may further comprise an input unit 805, which input unit 805 may be adapted to receive input digital or character information and to generate keyboard, mouse, joystick, optical or trackball signal inputs in connection with user settings and function control.

The server may also include a display unit 806, which display unit 806 may be used to display information entered by or provided to a user as well as various graphical user interfaces of the server, which may be composed of graphics, text, icons, video, and any combination thereof. In this embodiment, the processor 801 in the server loads executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 executes the application programs stored in the memory 802, so as to implement various functions as follows:

Acquiring a plurality of data to be processed from a data source, dividing the plurality of data into a plurality of data groups, and storing the group identification of each data group to the local; distributing each data group to a corresponding processing process in the processing process group, and updating the data state of the data in the data source to a state to be processed; processing the data in the target data group according to the processing progress and the data state of the data in the distributed target data group; and synchronizing the processing result of the processing process to a monitoring process of the processing process; when detecting that the processing process is abnormal, triggering a monitoring process to determine unprocessed data in a data group corresponding to the abnormal processing process based on a monitoring side record file, wherein the monitoring side record file comprises a synchronized data state and a group identifier of a monitoring side; updating the data state of unprocessed data in the data source into an unprocessed state through a monitoring process; when the abnormal processing process is detected not to be recovered to be normal within the preset time period, the unprocessed data is distributed to other processing processes for processing, and the group identification of the unprocessed data is adjusted to the group identification of the data group corresponding to the other processing processes; in the process of distributing unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is a preset abnormal type and the abnormal processing process is recovered to be normal, checking the current group identification of the unprocessed data; if the verification is not passed, the exception handling process is terminated to continue to process the unprocessed data, and other handling processes process the unprocessed data.

As can be seen from the above, the server in the embodiment of the present application may store the group identifier of each data group to the local while sending the data in the data group to the corresponding processing process for processing, in the processing process, synchronize the processing result of each piece of data to the listening process, and if the processing process is abnormal, ensure that the data that is not processed by the abnormal processing process can be found according to the local and listening side record files; updating the data state of the unprocessed data into an unprocessed state to wait for processing by a monitoring process, and distributing the unprocessed data to other processing processes for processing when the abnormal processing process is not recovered within a preset time period, so as to avoid missing processing; in the process of distributing unprocessed data to other processing processes, if the exception type of the exception processing process is a preset exception type, and the exception processing process is recovered to be normal, checking the current group identification of the unprocessed data, if the checking is passed, the unprocessed data is not distributed to the other processing processes, the processing can be continued by the exception processing process, if the checking is not passed, the unprocessed data is distributed to the other processing processes, the exception processing process does not need to process the unprocessed data, repeated processing is avoided, and processing integrity is improved.

Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

The foregoing describes in detail a data processing method, apparatus, storage medium and server provided in the embodiments of the present application, and specific examples are applied to describe the principles and implementations of the present application, where the descriptions of the foregoing embodiments are only used to help understand the technical solutions and core ideas of the present application; those of ordinary skill in the art will appreciate that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of data processing, comprising:

acquiring a plurality of data to be processed from a data source, and dividing the plurality of data into a plurality of data groups;

distributing each data group to a corresponding processing process in the processing process group, storing the group identification of each data group to the local, and updating the data state of the data in the data source to a state to be processed;

In the process of distributing the unprocessed data to other processing processes, if the abnormal type of the abnormal processing process is communication abnormality and the abnormal processing process is recovered to be normal, comparing the current group identifier of the first data with the group identifier corresponding to the abnormal processing process;

2. The data processing method of claim 1, wherein the method further comprises:

3. The method for processing data according to claim 1, wherein said assigning the unprocessed data to other processing processes for processing and adjusting the group identifier of the unprocessed data to the group identifier of the data group corresponding to the other processing processes comprises:

4. A data processing method according to claim 3, wherein said triggering the listening process to determine unprocessed data in the data set corresponding to the exception handling process based on the listening side record file comprises:

triggering the monitoring process to determine the first data with the data state in the data group corresponding to the abnormal processing process as the processing state and the second data with the data state as the to-be-processed state as the unprocessed data based on the monitoring side record file.

5. The data processing method of claim 1, wherein the method further comprises:

6. A data processing method according to any one of claims 1 to 5, wherein said assigning each data group to a corresponding processing process of a group of processing processes comprises:

7. The method for processing data according to claim 1, wherein the processing the data in the target data group according to the processing progress and the data state of the data in the allocated target data group comprises:

8. A data processing apparatus, comprising:

the data dividing unit is used for acquiring a plurality of data to be processed from a data source and dividing the plurality of data into a plurality of data groups;

the first data distribution unit is used for distributing each data group to a corresponding processing process in the processing process group, storing the group identification of each data group to the local and updating the data state of the data in the data source into a state to be processed;

the data verification unit is used for comparing the current group identifier of the first data with the group identifier corresponding to the abnormal processing process when the abnormal type of the abnormal processing process is communication abnormality and the abnormal processing process is recovered to be normal in the process of distributing the unprocessed data to other processing processes;

and the termination processing unit is used for terminating the abnormal processing process to process the first data when the current group identification of the first data is different from the group identification corresponding to the abnormal processing process, and processing the first data by the other processing processes.

9. The data processing apparatus of claim 8, wherein the data processing apparatus further comprises:

a third data allocation unit, configured to allocate the unprocessed data to the abnormal processing process when it is detected that the abnormal processing process is restored to normal within a preset period of time; updating the data state of the unprocessed data into a state to be processed;

10. A storage medium storing instructions which, when executed by a processor, implement the steps in the data processing method of any one of claims 1 to 7.

11. A server comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, implements the steps of the data processing method of any of claims 1 to 7.