CN115495411A

CN115495411A - Data processing method, device, equipment and storage medium

Info

Publication number: CN115495411A
Application number: CN202211176676.7A
Authority: CN
Inventors: 罗绍军
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-09-26
Filing date: 2022-09-26
Publication date: 2022-12-20

Abstract

The disclosure provides a data processing method, a data processing device, a data processing apparatus and a storage medium, and relates to the technical field of computers, in particular to the technical fields of stream computing, distributed computing, big data and the like. The specific implementation scheme is as follows: determining watermark information corresponding to a job in a preset stream type calculation engine, wherein the watermark information comprises a job identifier and a watermark timestamp, and storing the watermark information into a preset storage system, wherein the watermark information stored in the preset storage system is used for determining a processing starting point of a data stream corresponding to the job when the job in the preset stream type calculation engine is recovered. By adopting the technical scheme, when the stream type calculation engine normally processes the operation, the watermark information containing the operation identification and the watermark timestamp is stored, so that the processing starting point of the data stream corresponding to the operation is determined when the operation is recovered.

Description

Data processing method, device, equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to the field of streaming computing, distributed computing, big data, and the like.

Background

Streaming computing engines (e.g., flink, etc.) typically generate a large number of intermediate results during the streaming computing process, and these intermediate results may be referred to as states (states), also called state data. The state plays an important role in fault recovery of the streaming computing engine, usually a large number of states are stored persistently in a snapshot form, the storage system has high performance and reliability requirements, the cost is high, and the large number of state snapshot storages have certain influence on the computing performance and stability of the streaming computing engine.

Disclosure of Invention

The disclosure provides a data processing method, a device, equipment and a storage medium.

According to an aspect of the present disclosure, there is provided a data processing method including:

determining watermark information corresponding to a job in a preset stream type calculation engine, wherein the watermark information comprises a job identifier and a watermark timestamp;

and storing the watermark information to a preset storage system, wherein the watermark information stored in the preset storage system is used for determining a processing starting point of a data stream of a corresponding job when the job in the preset stream type computing engine is recovered.

According to another aspect of the present disclosure, there is provided a data processing apparatus including:

the system comprises a watermark information determining module, a watermark information determining module and a watermark information processing module, wherein the watermark information determining module is used for determining watermark information corresponding to a job in a preset stream type computing engine, and the watermark information comprises a job identifier and a watermark timestamp;

and the watermark information storage module is used for storing the watermark information to a preset storage system, wherein the watermark information stored in the preset storage system is used for determining a processing starting point of a data stream corresponding to a job when the job in the preset stream type calculation engine is recovered.

According to another aspect of the present disclosure, there is provided an electronic device including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method of the embodiments of the present disclosure.

According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the respective steps of the method of the embodiments of the present disclosure.

According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, performs the respective steps of the method according to any of the embodiments of the present disclosure.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:

fig. 1 is a flow chart of a data processing method provided according to an embodiment of the present disclosure;

FIG. 2 is a flow chart of another data processing method provided in accordance with an embodiment of the present disclosure;

FIG. 3 is a flow chart of yet another data processing method provided in accordance with an embodiment of the present disclosure;

FIG. 4 is a flow chart of yet another data processing method provided in accordance with an embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a data processing process provided according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a data processing apparatus provided in accordance with an embodiment of the present disclosure;

fig. 7 is a block diagram of an electronic device for implementing a data processing method of an embodiment of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of embodiments of the present disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

Fig. 1 is a flowchart of a data processing method provided according to an embodiment of the present disclosure, where the embodiment of the present disclosure is applicable to a case where a streaming computation engine is used for data stream processing. The method may be performed by a data processing apparatus, which may be implemented in hardware and/or software and may be configured in an electronic device. Referring to fig. 1, the method specifically includes the following steps:

s101, determining watermark information corresponding to a job in a preset stream type calculation engine, wherein the watermark information comprises a job identifier and a watermark timestamp;

and S102, storing the watermark information to a preset storage system, wherein the watermark information stored in the preset storage system is used for determining a processing starting point of a data stream corresponding to a job when the job in the preset stream type computing engine is recovered.

In the embodiment of the present disclosure, the preset streaming calculation engine may include, for example, flink, storm, spark, or the like, and is not limited specifically. The computing framework based on the preset streaming computing engine may include a data input end (also called Source end) and a data output end (Sink end), where the Source end is responsible for acquiring data to be processed from a data stream Source (such as a data Source end message queue system), and after performing corresponding computation on the data to be processed by the preset streaming computing engine, the Source end outputs computation result data to the Sink end, and then the Sink end outputs the computation result data to a system or a device (such as a data output end message queue system) that really needs to use the computation result data. The data flow is generally from a system capable of storing a message queue, and may include Kafka, kinesis, or Elasticsearch, for example, and the system for finally receiving the calculation result data may also include Kafka, for example, without limitation in the embodiments of the present disclosure.

In the related art, a streaming computing engine generally executes corresponding computation by taking a Job (Job) as a unit, in order to ensure that the Job can be recovered when the Job (Job) exception occurs or the Job is resubmitted in the streaming computing engine, a large amount of state snapshot data needs to be stored in the computation process, when the Job processing is interrupted, the Job needs to be recovered, and a large amount of state snapshot data needs to be read for recalculation in the recovery process.

In the embodiment of the present disclosure, a scheme of storing watermark information may be adopted instead of storing state snapshot data. For example, in the preset streaming engine, a job identifier (such as jobId) can be used to uniquely identify the job. The data in the data stream generally carries an Event time (Event time) stamp, and the Event time stamp can be used to indicate the time of Event creation, that is, the generation time of the corresponding data. The watermark is also called watermark, and the watermark timestamp can be understood as a timestamp for advancing a clock performed by Job, and the generation mode of the watermark timestamp can be set according to actual requirements. Because of network transmission and the like, data in the data stream may arrive out of order, that is, the data with a larger event timestamp may arrive first, and the data with a smaller event timestamp may arrive later, the watermark timestamp may be a timestamp obtained by subtracting a preset value from the maximum event timestamp in the currently reached data, and it may be considered that the data of the corresponding event timestamp before the watermark timestamp is already calculated, that is, the data of the event timestamp before the watermark timestamp is already involved in the calculation of the current calculation result data.

In the embodiment of the disclosure, in the normal operation process of the preset streaming type calculation engine, the preset frequency may be adopted to determine the watermark timestamp corresponding to the operation in the preset streaming type calculation engine, watermark information is generated according to the operation identifier and the watermark timestamp, and different operations correspond to different watermark information. And the watermark time stamp corresponding to the operation can be read from the Sink terminal. The preset frequency can be set according to actual requirements.

Illustratively, after determining the watermark information, the watermark information is stored in a preset storage system. The type of the predetermined storage system is not limited, and may be, for example, redis or Kafka. Optionally, the data stream corresponding to the preset streaming type calculation engine comes from Kafka, and the preset storage system includes Kafka, so that multiplexing of the storage system can be performed, and the storage cost is further saved.

The watermark information stored in the preset storage system is used for determining the processing starting point of the data stream of a certain Job in the preset stream type calculation engine when the Job needs to be recovered due to situations such as Job abnormity or Job resubmission. The processing start point may be understood as resetting the consumption site time, that is, a position where to restart acquiring the to-be-processed data from the data stream when the job is resumed, or acquiring data with the event timestamp as the processing start point from the data stream as the to-be-processed data when the job is resumed. For example, in the case where the Source side acquires data to be processed from the data stream according to the offset (offset), the determination of the processing start point may be understood as resetting of the offset (offset).

According to the technical scheme, the watermark information corresponding to the operation in the preset streaming computation engine is determined, wherein the watermark information comprises the operation identification and the watermark timestamp, the watermark information is stored in the preset storage system, and the watermark information stored in the preset storage system is used for determining the processing starting point of the data stream corresponding to the operation when the operation in the preset streaming computation engine is recovered. By adopting the technical scheme, in the process of presetting the normal processing operation of the stream type computing engine, the watermark information containing the operation identification and the watermark time stamp is stored so as to determine the processing starting point of the data stream corresponding to the operation when the operation is recovered.

In an alternative embodiment, the method further comprises: responding to the preset streaming computing engine to recover target operation, and acquiring corresponding target watermark information from the preset storage system according to the target operation identification of the target operation; and determining a target time according to a target watermark time stamp in the target watermark information, wherein the target time corresponds to a processing starting point of the data stream when the target operation is recovered. The method has the advantages that when the preset streaming computing engine recovers the target operation, the corresponding target watermark information can be quickly acquired from the preset storage system, the processing starting point can be quickly and accurately determined based on the target watermark information, and the efficiency and the accuracy of operation recovery are favorably improved.

For example, the target job may be any job that has started to be executed in the preset streaming computing engine, and the target job may interrupt execution due to an exception or resubmission. The target job identifier may be understood as a job identifier corresponding to the target job, and when the target job is recovered, the target job identifier generally remains unchanged, and may obtain target watermark information including the target job identifier from a preset storage system, and determine a processing start point of the data stream when the target job is recovered, that is, a target time, according to a target watermark timestamp in the target watermark information.

In an optional implementation, after determining the target time according to the target watermark time stamp in the target watermark information, the method further includes: and acquiring the data to be processed corresponding to the target operation from the corresponding position in the data stream corresponding to the target operation according to the target time. The method has the advantages that after the target time is determined, the data to be processed can be accurately acquired, the target operation is continuously executed, and the efficiency and the accuracy of operation recovery are improved.

For example, the corresponding position of the target time in the data stream may be understood as a position where the event timestamp is equal to the target time, that is, data with an event timestamp greater than or equal to the target time in the data stream will be acquired as the to-be-processed data corresponding to the target job.

Fig. 2 is a flowchart of another data processing method according to an embodiment of the present disclosure, and this embodiment proposes an alternative scheme based on the above optional embodiments, and further explains a case where a job may include multiple concurrent tasks.

Illustratively, on the basis of the foregoing optional embodiments, the job includes multiple tasks, different tasks in the same job correspond to different watermark information, and the watermark information further includes a task identifier; wherein, the determining the target time according to the target watermark time stamp in the target watermark information includes: determining a minimum target watermark timestamp among a plurality of target watermark timestamps in the target watermark information; and determining the target time according to the minimum target watermark time stamp. The advantage of this arrangement is that the target time can be determined more reasonably and accurately for jobs in which concurrent processing situations exist.

Referring to fig. 2, the method includes:

s201, determining watermark information corresponding to the operation in the preset streaming calculation engine.

The operation comprises a plurality of tasks, different tasks in the same operation correspond to different watermark information, and the watermark information comprises an operation identifier, a task identifier and a watermark time stamp.

For example, in order to improve the execution efficiency of the job, the job may be executed in a concurrent manner, that is, one job may include multiple concurrent tasks (tasks) and the multiple tasks are processed in parallel, so each Task may correspond to its own watermark information.

And S202, storing the watermark information to a preset storage system.

And S203, responding to the target operation recovered by the preset stream type calculation engine, and acquiring corresponding target watermark information from a preset storage system according to the target operation identification of the target operation.

Illustratively, target watermark information corresponding to each target task in the target job is acquired from a preset storage system according to a target job identifier of the target job. The target task can be understood as a task in the target job, and the number of the acquired target watermark information is the same as the number of the target tasks, that is, the number of the target watermark timestamps is the same as the number of the target tasks.

S204, determining the minimum target watermark time stamp in the plurality of target watermark time stamps in the target watermark information.

And S205, determining a target time according to the minimum target watermark time stamp.

For example, since data from the data source is not fixed when being distributed to each task, which may be understood as a mesh correspondence relationship, when determining the target time, the target time stamp with the smallest value, that is, the target watermark time stamp with the earliest time, may be selected from the obtained plurality of target watermark time stamps, and then the target time may be determined according to the smallest target watermark time stamp, so as to avoid missing data that does not participate in the calculation.

And S206, acquiring the data to be processed corresponding to the target operation from the corresponding position in the data stream corresponding to the target operation according to the target time.

According to the data processing method provided by the embodiment of the disclosure, in the process of presetting the normal processing operation of the stream type calculation engine, watermark information including an operation identifier, a task identifier and a watermark timestamp is stored, when the operation is recovered, the target time is determined according to the minimum value in the watermark timestamps respectively corresponding to all tasks in the operation, and the data to be processed corresponding to the target operation is obtained from the position corresponding to the target time in the data stream, so that the stream type calculation engine can continue to calculate based on the data to be processed, the operation to be recovered is continuously executed, the efficiency and the accuracy of the operation recovery can be effectively improved on the basis of reducing the storage cost of a calculation state and reducing the influence on the calculation performance and the stability.

In an optional implementation manner, the watermark information further includes a job version identifier; the step of retrieving, in response to the preset stream computing engine, a target job, and acquiring, according to a target job identifier of the target job, corresponding target watermark information from the preset storage system includes: and responding to the preset streaming computing engine to recover the target operation, and acquiring target watermark information corresponding to the latest operation version identification of each target task in the target operation from the preset storage system according to the target operation identification and the corresponding target task identification of the target operation. The method has the advantages that in the execution process of the target operation or after the target operation is restarted, the number of concurrent tasks may be changed, and the like, and the target watermark information corresponding to the latest target task can be accurately identified by recording the version identification, so that the accuracy of operation recovery is further ensured.

The version information may include a version number, and the version number may be sequentially incremented along with the iteration of the version. For example, assuming that 200 target tasks in a target job are changed into 100 target tasks, and watermark information is stored in a preset storage system at a preset frequency, therefore, a situation that an old version of watermark information and a new version of watermark information exist at the same time may occur, and in order to avoid that a watermark timestamp in watermark information corresponding to an original 101-200 task participates in determining a target time, a target watermark timestamp to be read may be determined according to a latest version number, so as to accurately determine the target time.

Fig. 3 is a flowchart of another data processing method provided according to an embodiment of the present disclosure, and this embodiment proposes an alternative scheme based on the above optional embodiments, and further illustrates the determination of the target time.

S301, determining watermark information corresponding to the operation in the preset stream type calculation engine.

And S302, storing the watermark information to a preset storage system.

And S303, responding to the target operation recovered by the preset streaming computing engine, and acquiring target watermark information corresponding to the latest operation version identification of each target task in the target operation from the preset storage system according to the target operation identification and the corresponding target task identification of the target operation.

S304, determining the minimum target watermark time stamp in the plurality of target watermark time stamps in the target watermark information.

S305, determining target time according to the difference between the minimum target watermark time stamp and the preset number of window lengths, wherein the window lengths comprise the lengths of time windows adopted when the preset streaming computing engine executes target jobs.

For example, when the streaming computing engine is preset to execute the target job, a time window is usually used to batch process the data, and the time window is also called a maximum aggregation window, and the length of the time window may be understood as the time length of the streaming data covered by the time window.

The preset number may be set according to actual requirements, such as 1 or more. Assuming that the preset number is denoted as N, the window length is denoted as T, and the minimum target watermark timestamp is denoted as min _ watermark, the target time may be denoted as min _ watermark-N × T.

And S306, acquiring the data to be processed corresponding to the target operation from the corresponding position in the data stream corresponding to the target operation according to the target time.

According to the data processing method provided by the embodiment of the disclosure, on the basis of the optional embodiments, when the target time is determined, the window lengths of the preset number are traced forward, so that when the operation is recovered, data which do not participate in calculation are further prevented from being omitted, and the accuracy of the calculation result is effectively ensured.

In an optional implementation manner, after the acquiring, according to the target time, to-be-processed data corresponding to the target job from a corresponding position in a data stream corresponding to the target job, the method further includes: receiving calculation result data obtained after the preset streaming calculation engine calculates the data to be processed; and determining data to be output according to the calculation result data, wherein the data to be output comprises data of which the corresponding event timestamp is greater than the minimum target watermark timestamp in the calculation result data. This has the advantage that the output of inaccurate data can be reduced.

Illustratively, when the target time is determined, a preset number of window lengths are traced forward, so that the target time may fall into a middle position of a certain time window, and thus, more data already participating in calculation or incomplete data exist in the initial window, data with an event timestamp less than or equal to the minimum target watermark timestamp may be filtered out from the calculation result data, and when the event timestamp (which may be understood as an event timestamp corresponding to the time window end time) is greater than the minimum target watermark timestamp, output of data to be output is performed, so that inaccurate data may be effectively filtered out. Optionally, in order to ensure data reliability, the data to be output may also include data obtained by subtracting a set number (e.g., 1) of window lengths from the minimum target watermark timestamp, where the event timestamp corresponding to the data in the calculation result is greater than or equal to the minimum target watermark timestamp.

In an alternative embodiment, the pre-defined streaming engine comprises Flink. The Flink is a framework and a distributed processing engine, can be used for performing stateful computation on borderless and borderless data streams, can run in a common cluster environment, can perform computation at an internal memory speed and in any scale, effectively supports out-of-order and delayed events, and has better computation performance.

In an alternative embodiment, the predetermined storage system comprises Kafka, and the watermark information is stored in a topic (topic) in the predetermined storage system. Kafka is a high-throughput distributed publish-subscribe messaging system, which has the advantages of high performance, low latency, high availability and the like, and can be well used with Flink.

In an alternative embodiment, the watermark information is stored in the theme in the preset storage system based on a preset deduplication storage policy (compact). The method has the advantages that the duplicate removal processing can be effectively carried out on the watermark information, the watermark information with fixed time length is guaranteed to be stored, the storage resource is saved, and meanwhile, the efficiency of obtaining the target watermark timestamp is improved.

In an alternative embodiment, the data stream corresponding to the preset streaming calculation engine is from Kafka. That is, the data source message queue system is Kafka. The advantage of setting up like this is, can multiplex Kafka and carry out the storage of watermark information, further reduce the storage cost, improve the read-write efficiency of watermark information.

Fig. 4 is a flowchart of another data processing method provided according to an embodiment of the present disclosure, and fig. 5 is a schematic diagram of a data processing process provided according to an embodiment of the present disclosure, which can be understood with reference to fig. 4 and 5. On the basis of the above optional embodiments, the present embodiment proposes an alternative, as shown in fig. 4, including:

s401, watermark information corresponding to the operation in the preset streaming computing engine is determined through the data output end.

The watermark information comprises a watermark time stamp, jobId, taskId and a version number. Illustratively, the preset streaming computation engine is Flink.

S402, storing the watermark information to a preset storage system through a data output end.

Illustratively, the storage system is preset as Kafka, watermark information at a Sink end is periodically written into topic of Kafka in the normal operation process of the Flink, and the storage strategy of Kafka topic adopts compact to ensure that watermark information with fixed time length is stored. Referring to fig. 5, the data output end writes watermark information into the default storage system.

And S403, responding to the target job recovered by the preset streaming computing engine, and acquiring target watermark information corresponding to the latest job version identification of each target task in the target job from a preset storage system through a data output end and a data input end according to the target job identification and the corresponding target task identification of the target job.

Illustratively, when a target job in the Flink is subjected to abnormal job recovery or job re-lifting, the Sink end and the Source end respectively load watermark information from the Kafka topoic.

S404, determining the minimum target watermark time stamp in the plurality of target watermark time stamps in the target watermark information through the data output end and the data input end.

Exemplarily, the Source end calculates a minimum watermark timestamp of watermark timestamps of the latest version numbers recorded in all tasks in the same joba, and records the minimum watermark timestamp as min _ watermark for subsequent offset reset; and the Sink end also calculates the minimum watermark time stamp in the watermark time stamps of the latest version numbers recorded in all tasks in the same joba, and records the minimum watermark time stamp as min _ watermark for subsequent repeated data filtering. As shown in fig. 5, the data input terminal and the data output terminal respectively look up the minimum watermark (min _ watermark) in the preset storage system.

S405, determining target time through a data input end according to the difference between the minimum target watermark time stamp and the lengths of the windows with the preset number.

The window length comprises the length of a time window adopted when the streaming computation engine executes the target operation.

For example, the target time determined by the Source terminal can be represented as min _ watermark-N × T.

And S406, acquiring to-be-processed data corresponding to the target operation from a corresponding position in the data stream corresponding to the target operation according to the target time through the data input end.

Illustratively, assuming the data stream is also from Kafka, the Source end resets all partition consumption sites in the Source topic to min _ watermark-N × T. As shown in fig. 5, the data input end obtains the data to be processed from the data source end message queue system (Kafka), and resets the consumption point to the minimum watermark-N × T. And then, the preset streaming engine performs convergence calculation on the data to be processed and outputs calculation result data to the data output end.

And S407, receiving calculation result data obtained by calculating the data to be processed by the preset streaming calculation engine through the data output end, determining the data to be output according to the calculation result data, and outputting the data to be output.

And the data to be output comprises data of which the corresponding event time stamp is greater than the minimum target watermark time stamp in the calculation result data.

For example, due to the fact that a large amount of data or incomplete data exists in the initial window, inaccurate data can be filtered out at the Sink, that is, data with an event timestamp less than or equal to min _ watermark is filtered out at the Sink, and the inaccurate data is output after the event timestamp is greater than min _ watermark, for example, to Kafka later. As shown in fig. 5, the data output terminal outputs data of which event time stamp is larger than the minimum watermark among the result data to the data output terminal message queue system (Kafka).

According to the data processing method provided by the embodiment of the disclosure, the state storage of the streaming computing operation is irrelevant to the computed data, so that the limitation of the state storage on the cluster scale, the performance and other aspects of the storage system is reduced, the state storage is greatly simplified, a series of time delay, performance and stability problems caused by a large amount of state snapshot storage are avoided, the system stability is improved, the operation and maintenance cost is reduced, in addition, when the operation is recovered, the consumption point position resetting can be accurately determined according to the watermark information, the repeated data is effectively filtered, and the accuracy of the output data is ensured.

Fig. 6 is a schematic structural diagram of a data processing apparatus provided according to an embodiment of the present disclosure, where the embodiment of the present disclosure is applicable to a case where a streaming computation engine is used for data streaming processing, and the apparatus may be implemented in hardware and/or software and may be configured in an electronic device. Referring to fig. 6, the data processing apparatus 600 includes:

a watermark information determining module 601, configured to determine watermark information corresponding to a job in a preset streaming computing engine, where the watermark information includes a job identifier and a watermark timestamp;

a watermark information storage module 602, configured to store the watermark information to a preset storage system, where the watermark information stored in the preset storage system is used to determine a processing start point of a data stream of a corresponding job when a job in the preset streaming computation engine is resumed.

According to the technical scheme provided by the embodiment of the disclosure, watermark information corresponding to a job in a preset stream type calculation engine is determined, wherein the watermark information comprises a job identifier and a watermark timestamp, and the watermark information is stored in a preset storage system, wherein the watermark information stored in the preset storage system is used for determining a processing starting point of a data stream corresponding to the job when the job in the preset stream type calculation engine is recovered. By adopting the technical scheme, in the process of presetting the normal processing operation of the stream type calculation engine, the watermark information containing the operation identification and the watermark time stamp is stored, so that the processing starting point of the data stream corresponding to the operation is determined when the operation is recovered.

In an alternative embodiment, the method further comprises:

the target watermark information acquisition module is used for responding to the target operation recovered by the preset streaming calculation engine and acquiring corresponding target watermark information from the preset storage system according to the target operation identification of the target operation;

and the target time determining module is used for determining target time according to a target watermark time stamp in the target watermark information, wherein the target time corresponds to a processing starting point of the data stream when the target operation is recovered.

In an optional implementation manner, the job includes a plurality of tasks, different tasks in the same job correspond to different watermark information, and the watermark information further includes a task identifier;

wherein the target time determination module comprises:

a minimum time stamp determining unit configured to determine a minimum target watermark time stamp among a plurality of target watermark time stamps in the target watermark information;

and the target time determining unit is used for determining the target time according to the minimum target watermark time stamp.

In an optional implementation manner, the target time determination unit is specifically configured to:

and determining target time according to the difference between the minimum target watermark timestamp and the lengths of a preset number of windows, wherein the window length comprises the length of a time window adopted by the preset streaming computation engine when the target operation is executed.

In an optional implementation manner, the watermark information further includes a job version identifier;

the target watermark information obtaining module is specifically configured to:

and responding to the preset streaming computing engine to recover the target operation, and acquiring target watermark information corresponding to the latest operation version identification of each target task in the target operation from the preset storage system according to the target operation identification and the corresponding target task identification of the target operation.

In an alternative embodiment, the method further comprises:

and the to-be-processed data acquisition module is used for acquiring the to-be-processed data corresponding to the target operation according to the target time from the corresponding position in the data stream corresponding to the target operation after the target time is determined according to the target watermark time stamp in the target watermark information.

In an alternative embodiment, the method further comprises:

a calculation result data receiving module, configured to receive calculation result data obtained by the preset streaming calculation engine after obtaining to-be-processed data corresponding to the target job from a corresponding position in a data stream corresponding to the target job according to the target time and performing calculation on the to-be-processed data;

and the data to be output determining module is used for determining the data to be output according to the calculation result data, wherein the data to be output comprises data of which the corresponding event timestamp in the calculation result data is greater than the minimum target watermark timestamp.

In an alternative embodiment, the pre-defined streaming engine comprises Flink.

In an alternative embodiment, the predetermined storage system includes Kafka, and the watermark information is stored in a theme in the predetermined storage system.

In an optional implementation manner, the watermark information is stored in a theme in the preset storage system based on a preset deduplication storage policy.

In an alternative embodiment, the data stream corresponding to the preset streaming calculation engine is from Kafka.

In the technical scheme of the disclosure, the collection, storage, use, processing, transmission, provision, disclosure and the like of the personal information of the related user all accord with the regulations of related laws and regulations, and do not violate the common customs of public order.

The present disclosure also provides an electronic device, a readable storage medium, and a computer program product according to embodiments of the present disclosure.

FIG. 7 illustrates a schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 7, the device 700 comprises a computing unit 701 which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.

Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

Computing unit 701 may be a variety of general purpose and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 701 executes the respective methods and processes described above, such as the data processing method. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708. In some embodiments, part or all of a computer program may be loaded onto and/or installed onto device 700 via ROM 702 and/or communications unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Network (WAN) blockchain networks, and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome. The server may also be a server of a distributed system, or a server incorporating a blockchain.

Artificial intelligence is the subject of research that makes computers simulate some human mental processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), both at the hardware level and at the software level. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like; the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, a machine learning/deep learning technology, a big data processing technology, a knowledge map technology and the like.

Cloud computing (cloud computing) refers to a technology system that accesses a flexibly extensible shared physical or virtual resource pool through a network, where resources may include servers, operating systems, networks, software, applications, storage devices, and the like, and may be deployed and managed in a self-service manner as needed. Through the cloud computing technology, high-efficiency and strong data processing capacity can be provided for technical application such as artificial intelligence and block chains and model training.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and are not limited herein.

The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A method of data processing, comprising:

2. The method of claim 1, further comprising:

responding to the preset streaming computing engine to recover target operation, and acquiring corresponding target watermark information from the preset storage system according to the target operation identification of the target operation;

and determining target time according to a target watermark time stamp in the target watermark information, wherein the target time corresponds to a processing starting point of the data stream when the target operation is recovered.

3. The method according to claim 2, wherein the job comprises a plurality of tasks, different tasks in the same job correspond to different watermark information, and the watermark information further comprises a task identifier;

wherein, the determining the target time according to the target watermark time stamp in the target watermark information includes:

determining a minimum target watermark timestamp among a plurality of target watermark timestamps in the target watermark information;

and determining the target time according to the minimum target watermark time stamp.

4. The method of claim 3, wherein the determining a target time from the minimum target watermark timestamp comprises:

5. The method of claim 3, wherein the watermark information further comprises a job version identification;

wherein, the recovering the target job in response to the preset streaming computing engine and acquiring the corresponding target watermark information from the preset storage system according to the target job identifier of the target job includes:

6. The method of claim 4, further comprising, after determining a target time based on a target watermark timestamp in the target watermark information:

and acquiring the data to be processed corresponding to the target operation from the corresponding position in the data stream corresponding to the target operation according to the target time.

7. The method according to claim 6, further comprising, after the acquiring, according to the target time, the to-be-processed data corresponding to the target job from the corresponding position in the data stream corresponding to the target job, the step of:

receiving calculation result data obtained after the preset streaming calculation engine calculates the data to be processed;

and determining data to be output according to the calculation result data, wherein the data to be output comprises data of which the corresponding event timestamp is greater than the minimum target watermark timestamp in the calculation result data.

8. The method of any of claims 1-7, wherein the pre-defined streaming engine comprises Flink.

9. The method of any one of claims 1-7, wherein the predetermined storage system includes Kafka, and the watermark information is stored in a subject in the predetermined storage system.

10. The method of claim 9, wherein the watermark information is stored in a subject in the predetermined storage system based on a predetermined deduplication storage policy.

11. The method of claim 9, wherein the data stream corresponding to the pre-configured streaming engine is from Kafka.

12. A data processing apparatus comprising:

the system comprises a watermark information determining module, a watermark information determining module and a watermark information processing module, wherein the watermark information determining module is used for determining the watermark information corresponding to the operation in a preset stream type calculation engine, and the watermark information comprises an operation identifier and a watermark timestamp;

13. The apparatus of claim 12, further comprising:

the target watermark information acquisition module is used for responding to the target operation recovered by the preset streaming computation engine and acquiring corresponding target watermark information from the preset storage system according to the target operation identification of the target operation;

14. The apparatus according to claim 13, wherein the job includes a plurality of tasks, different tasks in the same job correspond to different watermark information, and the watermark information further includes a task identifier;

wherein the target time determination module comprises:

15. The apparatus according to claim 14, wherein the target time determination unit is specifically configured to:

16. The apparatus according to claim 14, wherein the watermark information further includes a job version identification;

and responding to the target operation recovered by the preset stream type calculation engine, and acquiring target watermark information corresponding to the latest operation version identification of each target task in the target operation from the preset storage system according to the target operation identification and the corresponding target task identification of the target operation.

17. The apparatus of claim 15, further comprising:

and the to-be-processed data acquisition module is used for acquiring the to-be-processed data corresponding to the target operation from the corresponding position in the data stream corresponding to the target operation according to the target time after the target time is determined according to the target watermark time stamp in the target watermark information.

18. The apparatus of claim 17, further comprising:

a calculation result data receiving module, configured to receive calculation result data obtained after the preset streaming calculation engine calculates the data to be processed according to the target time after the data to be processed corresponding to the target job is obtained from a corresponding position in a data stream corresponding to the target job according to the target time;

and the data to be output determining module is used for determining the data to be output according to the calculation result data, wherein the data to be output comprises data of which the corresponding event timestamp is greater than the minimum target watermark timestamp in the calculation result data.

19. The apparatus of any of claims 12-18, wherein the pre-defined streaming engine comprises Flink.

20. The apparatus according to any one of claims 12-18, wherein the predetermined storage system comprises Kafka, and the watermark information is stored in a subject in the predetermined storage system.

21. The apparatus of claim 20, wherein the watermark information is stored in a theme in the predetermined storage system based on a predetermined deduplication storage policy.

22. The apparatus of claim 20, wherein the data stream corresponding to the pre-configured streaming engine is from Kafka.

23. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-11.