CN118200405A

CN118200405A - Data transmission method and device, electronic equipment and storage medium

Info

Publication number: CN118200405A
Application number: CN202211600117.4A
Authority: CN
Inventors: 张德坤; 顾风桂; 李小华
Original assignee: Qingdao Zhongke Shuguang Technology Service Co ltd
Current assignee: Qingdao Zhongke Shuguang Technology Service Co ltd
Priority date: 2022-12-12
Filing date: 2022-12-12
Publication date: 2024-06-14

Abstract

The embodiment of the invention discloses a data transmission method, a data transmission device, electronic equipment and a storage medium. The data transmission method specifically comprises the following steps: determining node input data and node input data types of nodes to be received; under the condition that the node input data type is determined to be the target data type, determining a target data transfer mark corresponding to the data node to be received according to the node input data; and determining target subject data according to the target data transfer mark so as to consume the target subject data through the data node to be received. The technical scheme of the embodiment of the invention can reduce the complexity of data transmission, reduce the complexity of executing tasks by the nodes, and improve the execution efficiency of data processing operation.

Description

Data transmission method and device, electronic equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of computers, in particular to a data transmission method, a data transmission device, electronic equipment and a storage medium.

Background

With the advent of the data age, large data platforms are becoming more and more widely used in various fields. Tasks at various stages in the large data processing process, such as data ingestion, data loading, data conversion, data cleaning, data aggregation or data analysis, are not isolated, and all need to mutually cooperate in a dependent manner, and the dependent cooperation of the data processing process forms a DAG (directed acyclic graph). A node in the DAG represents a data processing task and is tasked to ultimately form a complete data processing job.

In a data processing job, it is sometimes necessary to transfer the processing result of a certain node in the DAG to another node for processing. However, in the existing data transmission method, a resource (for example, a database table) is usually applied for in a node, data to be transmitted is written into the whole table, and then other nodes apply for access to the resource, so that data transmission is realized. However, the method increases the complexity of data transmission and task execution of the nodes, wastes resources and reduces the execution efficiency of data processing jobs.

Disclosure of Invention

The embodiment of the invention provides a data transmission method, a data transmission device, electronic equipment and a storage medium, which can reduce the complexity of data transmission, reduce the complexity of executing tasks by nodes and improve the execution efficiency of data processing operation.

According to an aspect of the present invention, there is provided a data transfer method including:

determining node input data and node input data types of nodes to be received;

under the condition that the node input data type is determined to be the target data type, determining a target data transfer mark corresponding to the data node to be received according to the node input data;

and determining target subject data according to the target data transfer mark so as to consume the target subject data through the data node to be received.

In addition, the data transfer method according to the above embodiment of the present invention may further have the following additional technical features:

Optionally, before determining the node input data and the node input data type of the data node to be received, the method further includes:

Determining at least one data node to be transferred, and creating a node data transfer mark corresponding to each data node to be transferred;

The determining node input data of the data node to be received includes:

Determining an available mark list corresponding to the data node to be received according to the node data transfer marks;

And determining the node input data in the available mark list according to a node input data determining instruction.

One embodiment of the above invention has the following advantages or benefits: the node data transfer mark of the data node to be transferred is created to determine the available mark list of the data node to be received according to the node data transfer mark, so that the node input data is determined in the available mark list, and the complexity of data transfer can be further reduced.

Optionally, the determining, according to the data transfer labels of the nodes, the available label list corresponding to the data node to be received includes:

determining a node operation sequence list of each operation node;

determining a target sequence number of the data node to be received according to the node operation sequence list;

determining the operation node with the sequence number smaller than the target sequence number as a target operation node;

and under the condition that the target job node is determined to be the data node to be transferred, adding a node data transfer mark of the target job node into the available mark list.

One embodiment of the above invention has the following advantages or benefits: according to the method, the specific implementation mode of the available mark list corresponding to the data node to be received is determined according to the data transmission marks of each node, so that any node behind the data node to be transmitted can use the data to be transmitted, and the flexibility of data transmission is improved.

Optionally, after the creating the node data transfer flag corresponding to each data node to be transferred, the method further includes:

determining a node subscription theme corresponding to the data node to be transferred according to the node data transfer mark;

Determining node theme data of the node subscription theme corresponding to the data node to be transferred according to the output data of the data node to be transferred;

The determining the target subject data according to the target data transfer mark comprises the following steps:

And determining a matched data transfer mark matched with the target data transfer mark in the node data transfer marks, and determining node theme data corresponding to the matched data transfer mark as the target theme data.

One embodiment of the above invention has the following advantages or benefits: the node subscription subject and the node subject data of the data node to be transmitted are determined, so that the node subject data corresponding to the matched data transmission mark is determined to be target subject data, data transmission through the publishing and subscribing system is realized, and the execution efficiency of the data processing operation is further improved.

Optionally, the node theme data includes first identification data of the node theme, transfer data of the node theme and second identification data of the node theme;

The determining node theme data of the node subscription theme corresponding to the data node to be transferred according to the output data of the data node to be transferred comprises the following steps:

Determining the data subject to be transferred according to the output data of the data node to be transferred;

The consuming, by the data node to be received, the target subject data includes:

when the data node to be received identifies the first identification data of the node theme, starting to consume the data transmitted by the node theme;

And ending the consumption of the data transmitted by the node theme when the data to be received identifies the second identification data of the node theme.

One embodiment of the above invention has the following advantages or benefits: specific data constitution of the node theme data is provided so as to realize accurate consumption of the target theme data.

Optionally, the method further comprises:

Determining the node state of the data node to be received under the condition that the data node to be received does not recognize the second identification data of the node theme;

And under the condition that the node state is determined to be in a stop execution state, suspending consuming the node theme transfer data until the node state is determined to be in a start execution state, and continuing consuming the node theme transfer data.

One embodiment of the above invention has the following advantages or benefits: the specific application scene that the second identification data of the node theme is not identified is provided, when the node state is determined to be in the execution stopping state, consumption of the data transmitted to the node theme is suspended, and when the node state is determined to be in the execution starting state, consumption of the data transmitted to the node theme is continued, and the data transmission can be continued after the data node to be received is abnormal and restarted, so that reloading and breakpoint continuous transmission of the data are realized.

Optionally, the method further comprises:

And waiting for consumption of the target subject data until the target subject data is not empty under the condition that the target subject data is determined to be empty.

One embodiment of the above invention has the following advantages or benefits: the method has the advantages that the consumption of the target subject data is waited when the target subject data is empty, the node tasks of the data nodes to be received and the data nodes to be transmitted can be executed simultaneously, the data nodes to be received do not need to wait for the data nodes to be transmitted to finish the node tasks and then execute the node tasks, and therefore the execution efficiency of data processing operation is improved.

According to another aspect of the present invention, there is provided a data transfer apparatus comprising:

The input data determining module is used for determining node input data and node input data types of the data nodes to be received;

The data transfer mark determining module is used for determining a target data transfer mark corresponding to the data node to be received according to the node input data under the condition that the node input data type is determined to be the target data type;

and the data transfer module is used for determining target subject data according to the target data transfer mark so as to consume the target subject data through the data node to be received.

According to another aspect of the present invention, there is provided an electronic apparatus including:

At least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data transfer method of any one of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a data transfer method according to any one of the embodiments of the present invention.

According to the technical scheme, the node input data and the node input data type of the data node to be received are determined, and the target data transfer mark corresponding to the data node to be received is determined according to the node input data under the condition that the node input data type is determined to be the target data type, so that the target subject data is determined according to the target data transfer mark, consumption of the target subject data by the data node to be received is achieved, the problems of high complexity of executing tasks and low execution efficiency of data processing operation of the existing data transfer and the node are solved, the complexity of data transfer can be reduced, the complexity of executing tasks of the node is reduced, and the execution efficiency of the data processing operation is improved.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a data transfer method according to a first embodiment of the present invention;

FIG. 2 is a flow chart of a data transfer method according to a second embodiment of the present invention;

FIG. 3 is a schematic diagram of node theme data according to a second embodiment of the present invention;

FIG. 4 is a schematic diagram of a node job sequence list according to a second embodiment of the present invention;

FIG. 5 is a flow chart of a data transfer method according to a third embodiment of the present invention;

FIG. 6 is a schematic diagram of a data transmission device according to a fourth embodiment of the present invention;

fig. 7 is a schematic structural diagram of an electronic device implementing a data transfer method according to an embodiment of the present invention.

Detailed Description

In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

Fig. 1 is a flowchart of a data transmission method according to a first embodiment of the present invention, where the method may be applied to a case of reducing complexity of data transmission and improving execution efficiency of a data processing job, and the method may be performed by a data transmission device, where the device may be implemented by software and/or hardware, and may generally be directly integrated in an electronic device that performs the method, where the electronic device may be a terminal device or a server device, and the embodiment of the present invention does not limit types of electronic devices that perform the data transmission method. Specifically, as shown in fig. 1, the data transmission method specifically includes the following steps:

S110, determining node input data and node input data types of the data nodes to be received.

The node to be received may be any node waiting to receive data. The node input data may be any data that is input to the data node to be received. The node input data type may be a type of data input to the data node to be received. By way of example, the node input data types may include a string type, a numeric type, a tag type, etc., and embodiments of the present invention are not limited in this regard.

It should be noted that, the data to be received by the node to be received may be data transferred between nodes in the DAG, and not the node input data. Specifically, when a node (such as a data node to be received) in the DAG executes a node task, task processing needs to be performed according to result data obtained by executing tasks by other nodes, where the result data obtained by executing tasks by other nodes is to be received by the data node to be received, and the result data obtained by executing tasks by other nodes cannot be transmitted to the data node to be received as node input data. The data to be received by the data node to be received may be parameter data or temporary data among nodes in the DAG, which is not limited by the embodiment of the present invention.

In the embodiment of the invention, before the node task is executed by the data node to be received, the node input data and the node input data type of the data node to be received can be determined. It will be appreciated that some nodes in the DAG need to perform task processing based on node input data before performing node tasks.

And S120, under the condition that the node input data type is determined to be the target data type, determining a target data transfer mark corresponding to the data node to be received according to the node input data.

The target data type may be any target type of the node input data types. By way of example, the target data type may be a tag type, etc., and embodiments of the present invention are not limited in this regard. The target data transfer flag may be a flag corresponding to data to be received by the data node to be received, that is, a flag corresponding to data transferred to the data node to be received. It will be appreciated that the target data transfer indicia may be a tag for uniquely identifying the data being transferred. The target data transfer flag may be any type of flag, for example, a preset string type flag, a node parameter type flag, etc., which is not limited in this embodiment of the present invention.

In the embodiment of the invention, after the node input data and the node input data type of the data node to be received are determined, whether the node input data type is the target data type or not can be further determined, and when the node input data type is determined to be the target data type, the target data transfer mark corresponding to the data node to be received is determined according to the node input data.

S130, determining target subject data according to the target data transfer mark so as to consume the target subject data through the data node to be received.

The target theme data may be theme data corresponding to the target data transfer mark. It will be appreciated that the target subject data is the data to be received by the data node to be received, i.e. the data to be transferred. It is further understood that different data transfer markers may correspond to different subject data.

In the embodiment of the invention, after the target data transfer mark corresponding to the data node to be received is determined according to the node input data, the target subject data can be further determined according to the target data transfer mark, so that the target subject data is consumed through the data node to be received, and the data transfer of the target subject data is realized.

According to the technical scheme, the node input data and the node input data type of the data node to be received are determined, and the target data transfer mark corresponding to the data node to be received is determined according to the node input data under the condition that the node input data type is determined to be the target data type, so that the target subject data is determined according to the target data transfer mark, consumption of the target subject data by the data node to be received is achieved, the problems that the complexity of the existing data transfer and node execution task is high and the execution efficiency of the data processing operation is low are solved, the data transfer complexity can be reduced, the complexity of the node execution task is reduced, and the execution efficiency of the data processing operation is improved.

Example two

Fig. 2 is a flowchart of a data transfer method provided by a second embodiment of the present invention, where the foregoing technical solutions are further refined, and various specific alternative implementations of determining node input data of a data node to be received, determining target subject data according to the target data transfer flag, and consuming the target subject data by the data node to be received are provided. The technical solution in this embodiment may be combined with each of the alternatives in one or more embodiments described above. As shown in fig. 2, the method may include the steps of:

s210, determining at least one data node to be transferred, and creating a node data transfer mark corresponding to each data node to be transferred.

The data node to be transferred may be any node waiting to transfer data. By way of example, assuming that node a and node B are included in the DAG, node a obtains result data a after performing a node task, and node B needs to perform processing of the node task according to the result data a when performing the node task, node a may be determined as a data node to be transferred, and node B may be determined as a data node to be received. The node data transfer markers may be markers corresponding to the data nodes to be transferred. It will be appreciated that different nodes of data to be transferred may have different corresponding node data transfer labels.

In the embodiment of the invention, at least one data node to be transferred is determined so as to create a node data transfer mark corresponding to each data node to be transferred. It should be noted that, the embodiment of the present invention does not limit a specific implementation manner of creating the node data transfer flag corresponding to each data node to be transferred, as long as the creation of the node data transfer flag can be achieved.

S220, determining a node subscription theme corresponding to the data node to be transferred according to the node data transfer mark.

The node subscription topic may be a topic of any publishing and subscribing system corresponding to the data node to be transferred, for example, may be a topic of a Kafka publishing and subscribing system, or may be a topic of another publishing and subscribing system, which is not limited in the embodiment of the present invention.

In the embodiment of the invention, after the node data transfer marks corresponding to the data nodes to be transferred are created, the node subscription subject corresponding to the data nodes to be transferred can be determined according to the node data transfer marks.

S230, determining node theme data of the node subscription theme corresponding to the data node to be transferred according to the output data of the data node to be transferred.

The output data of the data node to be transferred may be result data obtained by executing the node task. It will be appreciated that the output data of the data node to be transferred may be data for which data transfer is to take place. The node topic data may be data corresponding to a node subscription topic. It can be understood that the publish-subscribe system can classify the node topic data by using the node subscription topic of the publish-subscribe system, and the data node to be received can consume the node topic data in the node subscription topic by using different node subscription topics.

In the embodiment of the invention, after determining the node subscription subject corresponding to each data node to be transferred according to the data transfer mark of each node, the node subject data of the node subscription subject corresponding to the data node to be transferred can be further determined according to the output data of the data node to be transferred.

For example, when the data node to be transferred executes the node task, a topic of the Kafka publish-subscribe system corresponding to the node data transfer mark may be created, so as to publish output data obtained by the data node to be transferred executing the node task into the topic, thereby obtaining node topic data.

Alternatively, the node topic data may include node topic delivery data. Correspondingly, determining the node topic data of the node subscription topic corresponding to the data node to be transferred according to the output data of the data node to be transferred may include: and determining the data subject to be transferred according to the output data of the data node to be transferred.

The node theme transfer data may be data for data transfer in the node theme data. It is to be appreciated that the node topic data can include a plurality of node topic delivery data. Fig. 3 is a schematic diagram of node theme data according to the second embodiment of the present invention, and as shown in fig. 3, the node theme delivery data may be "data" in fig. 3.

Specifically, after determining the node subscription subject corresponding to each data node to be transferred according to the data transfer mark of each node, the node subject transfer data of the node subscription subject corresponding to the data node to be transferred can be further determined according to the output data of the data node to be transferred.

According to the technical scheme, the specific data composition of the node theme data is provided, so that the target theme data can be accurately consumed.

S240, determining node input data and node input data types of the data nodes to be received.

It should be noted that fig. 2 is only a schematic diagram of one implementation manner, and steps S220-S230 and step S240 have no sequence relationship, and steps S220-S230 may be implemented first, then step S240 may be implemented, and then steps S220-S230 may be implemented, or both may be implemented in parallel or alternatively.

Optionally, determining node input data of the data node to be received may include: determining an available mark list corresponding to the data node to be received according to the data transfer marks of each node; and determining the node input data in the available mark list according to the node input data determining instruction.

The available mark list may be a list formed by available node data transfer marks corresponding to the data nodes to be received. The node input data determination instruction may be an instruction to determine node input data.

Specifically, after the node data transfer marks corresponding to the data nodes to be transferred are created, an available mark list corresponding to the nodes to be received can be further determined according to the node data transfer marks, so that the node input data can be determined in the available mark list according to the node input data determining instruction. It is to be appreciated that the list of available tags may include a plurality of node data transfer tags, and the node input data may be determined from the plurality of node data transfer tags based on the node input data determination instruction, i.e., the node input data may be the node data transfer tag.

Illustratively, an input parameter (i.e., node input data) is defined in the data node to be received, the data type of the input parameter is defined as a tag type, and a node data transfer tag is selected from the available tag list as the node input data.

According to the technical scheme, the node data transfer marks of the data nodes to be transferred are created, so that the available mark list of the data nodes to be received is determined according to the node data transfer marks, and therefore the node input data is determined in the available mark list, and the complexity of data transfer can be further reduced.

Optionally, determining the available mark list corresponding to the data node to be received according to the data transfer mark of each node may include: determining a node operation sequence list of each operation node; determining a target sequence number of a data node to be received according to the node operation sequence list; determining the operation node with the sequence number smaller than the target sequence number as a target operation node; and in the case that the target job node is determined to be the data node to be transferred, adding the node data transfer mark of the target job node into the available mark list.

The job node may be any node in the DAG that performs a node task. It may be understood that the job node may be a data node to be transferred, a data node to be received, or a node that does not need to transfer data, which is not limited by the embodiment of the present invention. The node job order list may be a list of the order in which the job nodes perform the node tasks. It will be appreciated that each job node needs to execute according to a set rule sequence when executing the node task. The target sequence number may be a sequence number of a node task to be performed by the data node to be received. The target job node may be a node having a sequence number less than the sequence number of the node task to be performed by the data node to be received.

Fig. 4 is a schematic diagram of a node Job sequence list provided in the second embodiment of the present invention, as shown in fig. 4, job1 may include Job nodes 1-6, job node 2 is a data node to be transferred, flag1 is a node data transfer flag of Job node 2, the node Job sequence list is ① Job node 1- > ② Job node 2- > ③ Job node 3- > ④ Job node 4- > ⑤ Job node 5- > ⑥ Job node 6, assuming that the data node to be received is the job node 4, the target sequence number is ④, and the target job node includes the job node 1, the job node 2 and the job node 3, since the job node 2 is the data node to be transferred, the flag1 can be added to the available flag list. Job2 may include an operation node a-e, an operation node c is a data node to be transferred, a flag2 is a node data transfer flag of the operation node c, a node operation sequence list is ① operation node a- > ② operation node b- > ③ operation node c- > ④ operation node d- > ⑤ operation node e, if the data node to be received is the operation node e, a target sequence number is ⑤, the target operation node includes operation nodes a-d, since the job node c is a data node to be transferred, flag2 may be added to the available flag list. It will be appreciated that in the node Job sequence list, the nodes in the same Job are put together in the list, and this Flag can be used by all nodes after the node where the Flag is present, such as Flag1 in fig. 4 marked with Flag2, and Flag1 can be used by all of the 3, 4, 5, and 6 nodes.

Specifically, after the node data transfer marks corresponding to the data nodes to be transferred are created, a node operation sequence list of each operation node can be further determined, and a target sequence number of the data node to be received is determined according to the node operation sequence list, so that the operation node with the sequence number smaller than the target sequence number is determined to be the target operation node, and the node data transfer mark of the target operation node is added to the available mark list when the target operation node is determined to be the data node to be transferred.

According to the technical scheme, the specific implementation mode of determining the available mark list corresponding to the data node to be received according to the data transmission marks of each node is provided, so that any node behind the data node to be transmitted can use the data to be transmitted, and the flexibility of data transmission is improved.

Optionally, determining a node job sequence list of each job node, and determining the node job sequence list of each job node in the DAG according to the traversing sequence by using a breadth-first traversing algorithm of the graph. It will be appreciated that in the node job sequence list, each job node is fetched and sent to the executor of the task orchestration framework for starting in a first-in first-out manner. When each job node executes the node task, if the node data transfer mark is found, a node subscription theme corresponding to the data node to be transferred can be created.

S250, under the condition that the node input data type is determined to be the target data type, determining a target data transfer mark corresponding to the data node to be received according to the node input data.

S260, determining a matched data transfer mark matched with the target data transfer mark in the node data transfer marks, and determining node theme data corresponding to the matched data transfer mark as the target theme data so as to consume the target theme data through the data node to be received.

The matching data transfer mark may be one node data transfer mark matched with the target data transfer mark in the node data transfer marks. It will be appreciated that the matching with the target data transfer tag may be the same as the target data transfer tag, or may correspond to the target data transfer tag, which is not limited in this embodiment of the present invention.

In the embodiment of the invention, after the target data transfer mark corresponding to the data node to be received is determined according to the node input data, the matched data transfer mark matched with the target data transfer mark can be further determined in the node data transfer marks, so that the node theme data corresponding to the matched data transfer mark is determined as the target theme data, and the target theme data is consumed through the data node to be received.

According to the technical scheme, the node subscription subject and the node subject data of the data node to be transmitted are determined, so that the node subject data corresponding to the matched data transmission mark is determined as the target subject data, and therefore data transmission through the publishing and subscribing system is achieved, and the execution efficiency of data processing operation is further improved.

Optionally, the node theme data may include first identification data of the node theme and second identification data of the node theme. Accordingly, consuming the target subject data by the data node to be received may include: when the data node to be received is determined to recognize the first identification data of the node theme, starting to consume the data transmitted by the node theme; and when the data node to be received is determined to recognize the second identification data of the node theme, ending the consumption of the data transmitted by the node theme.

The first identifying data of the node theme may be one of the data of the node theme for identification, for example, may be the data for identifying that the transfer of the data is started. The node topic second identification data may be another identification data in the node topic data, for example, may be data for identifying the end of the transfer data.

Illustratively, as shown in fig. 3, the first identification data of the node topic may be "S" in fig. 3: beginning ", the node topic second identification data may be" S "in fig. 3: successful "or" S: failure). It may be appreciated that if the data node to be transferred can successfully output all the data to be transferred when outputting the data, the second identification data of the node theme may be determined as "S: successful). If the data node to be transferred fails to successfully output all the data to be transferred when outputting the data, that is, if output failure or output error occurs in the process of outputting the data, the second identification data of the node theme can be determined as "S: failure).

For example, before the first data is written after the data node to be transferred is started, a special identification data, namely, the first identification data of the node theme, may be written in the node theme data, then the output data is written in the node theme data, namely, the node theme transfer data, after the data is completely written in the node theme data, and when the node task is finished, a special identification data, namely, the second identification data of the node theme, is written in the node theme data.

Specifically, after the node theme data corresponding to the matching data transfer mark is determined as the target theme data, consumption of the node theme transfer data can be started when the data node to be received is determined to recognize the first identification data of the node theme, and consumption of the node theme transfer data can be ended when the data node to be received is determined to recognize the second identification data of the node theme.

Optionally, the method may further include: under the condition that the data node to be received does not recognize the second identification data of the node theme, determining the node state of the data node to be received; and under the condition that the node state is determined to be in the execution stopping state, suspending consuming the node theme transfer data until the node state is determined to be in the execution starting state, and continuing consuming the node theme transfer data.

The node state may be a state in which the node executes the node task, for example, a state in which the node pauses executing the node task, a state in which the node ends executing the node task, or a state in which the node continues executing the node task after stopping executing the node task, etc., which is not limited in the embodiment of the present invention. The execution stop state may be a state in which the node pauses executing the node task, or may be a state in which the node ends executing the node task, which is not limited in the embodiment of the present invention. The start execution state may be a state in which the node continues to execute the node task after stopping executing the node task, which is not limited by the embodiment of the present invention.

Specifically, after the target subject data is consumed by the data node to be received, it may be further determined whether the data node to be received recognizes the second identification data of the node subject, and when the data node to be received does not recognize the second identification data of the node subject, determine a node state of the data node to be received, if the node state is a stop execution state, suspend the consumption of the data transferred by the node subject, until the node state is a start execution state, and continue the consumption of the data transferred by the node subject.

For example, assuming that the data to be received includes data a, data b and data c in the data to be transmitted, the node to be received fails or stops the node task for some reason, and the node to be received consumes the data a, but does not consume the data b and the data c, when the node to be received resumes executing the node task, the consumption of the data b and the data c may be continued.

According to the technical scheme, a specific application scenario is provided, in which the second identification data of the node theme is not identified, consumption of the data transmitted to the node theme is suspended when the node state is determined to be in a stop execution state, and consumption of the data transmitted to the node theme is continued when the node state is determined to be in a start execution state, and the data transmission can be continued after the data node to be received is abnormal and restarted, so that reloading and breakpoint continuous transmission of the data are realized.

Optionally, the method may further include: and waiting for consumption of the target subject data until the target subject data is not empty under the condition that the target subject data is determined to be empty.

Specifically, when the target subject data is consumed through the data node to be received, whether the target subject data is empty or not can be determined, and consumption of the target subject data is waited when the target subject data is empty until the consumption of the target subject data is started when the target subject data is not empty. It will be appreciated that if the target subject data is not null, the target subject data may be consumed directly.

According to the technical scheme, the consumption of the target subject data is waited when the target subject data is empty, the node tasks of the data node to be received and the data node to be transmitted can be executed simultaneously, the data node to be received does not need to wait for the data node to be transmitted to complete the node tasks and then execute the node tasks, and therefore the execution efficiency of data processing operation is improved.

Optionally, the method may further include: and deleting the node subscription theme and the node theme data on all the Job nodes when the node tasks of all the Job nodes in Job are successfully executed. If Job needs to be deleted, the node subscription topic and node topic data on all Job nodes in Job can be deleted. Job, i.e., a Job, may be a task orchestration made up of DAGs.

In the node job sequence list, a certain job node is determined, and the determination preceding the job node may be regarded as an upstream node, and the determination following the job node may be regarded as a downstream node. In the prior art, the start-up execution of the downstream task is typically triggered only after the end of the upstream task.

In the embodiment of the invention, each job node in the node job sequence list can be distributed to the distributed environment in sequence for simultaneous execution, and can continuously receive data streams from upstream tasks; the upstream data is not required to be produced and then sent to the downstream task, and the upstream task and the downstream task can be continuously sent and received through a data flow pipeline; the downstream task can start the breakpoint continuous transmission of the support data after stopping; the data of the upstream task is visible not only to the downstream tasks of the same branch of the DAG, but also to the downstream tasks of different branches; when the data of the upstream task is transmitted to the downstream task, the data is transmitted in real time and received in real time, so that the downstream task receives the data and processes the data in parallel with the upstream task in the process of continuously transmitting the upstream task to the downstream, and the execution efficiency of the whole job is improved; the unified data transfer interface makes the data transfer between the upstream and downstream tasks not need to pay attention to the transfer mode.

Alternatively, the data transfer method may be applied to any task orchestration framework known in the art. Task orchestration frameworks common in the prior art include Oozie, azkaban, airflow and Dolphinscheduler, etc., which are mainly focused on: the scheduling mode of each node of the DAG is triggered at fixed time, distributed scheduling is supported, single-point faults are avoided through decentralization, rich task types are supported, and good visualization is achieved. But in data transfer, data that is typically defined on metadata of the task instance or external is directly transferred to the entire DAG. The data is transferred by writing the data into the metadata of the task instance, so that the task loading is too slow when the data is large, and the whole DAG flow is affected. When data is transferred by directly transferring external data to the whole DAG, the execution result of a certain task in the DAG cannot be transferred to other downstream tasks. Existing partial DAG task orchestration tools can support data transfer, but cannot transfer to an indirect downstream task or another branch of the DAG at the time of transfer, but can only transfer to its direct downstream task.

According to the technical scheme, by applying any data transmission method disclosed by the embodiment to the task orchestration framework, data transmission among all task nodes in the DAG can be realized in the DAG task orchestration framework flow, the task nodes do not need to process, and the data transmission is not used as logic to be developed for a certain task.

According to the technical scheme, at least one data node to be transmitted is determined, a node data transmission mark corresponding to each data node to be transmitted is created, node subscription subject is determined according to the node data transmission mark, node subject data is determined according to output data of the data node to be transmitted, node input data and node input data types of the data node to be received are determined, when the node input data types are determined to be target data types, the target data transmission mark corresponding to the data node to be received is determined according to the node input data, and therefore the matched data transmission mark is determined in the data transmission marks of each node, the node subject data corresponding to the matched data transmission mark is determined to be target subject data, consumption is conducted on the target subject data through the data node to be received, the problems that the complexity of existing data transmission and node execution tasks is high and the execution efficiency of data processing operation is low are solved, the complexity of data transmission is reduced, the complexity of the node execution tasks is reduced, and the execution efficiency of the data processing operation is improved.

Example III

In order to better understand the data transmission method of the present embodiment by those skilled in the art, a specific example is described below, and fig. 5 is a flowchart of a data transmission method provided by the third embodiment of the present invention, and as shown in fig. 5, a specific process may include:

S510, determining at least one data node to be transferred, and creating a node data transfer mark corresponding to each data node to be transferred.

S520, determining a node subscription theme corresponding to the data node to be transferred according to the node data transfer mark.

And S530, determining the node theme transfer data of the node subscription theme corresponding to the data node to be transferred according to the output data of the data node to be transferred.

S540, determining a node operation sequence list of each operation node.

S550, determining the target sequence number of the data node to be received according to the node operation sequence list.

S560, determining the operation node with the sequence number smaller than the target sequence number as the target operation node.

S570, adding the node data transfer mark of the target job node to the available mark list under the condition that the target job node is determined to be the data node to be transferred.

S580, determining the node input data in the available mark list according to the node input data determining instruction.

S590, determining the node input data type of the data node to be received.

S5100, under the condition that the node input data type is determined to be the target data type, determining a target data transfer mark corresponding to the data node to be received according to the node input data.

S5110, determining a matched data transfer mark matched with the target data transfer mark in each node data transfer mark, and determining node theme transfer data corresponding to the matched data transfer mark as the target theme data.

S5120, when the data node to be received identifies the first identification data of the node theme, consumption of the data transmitted by the node theme is started.

And S5130, waiting for consumption of the target subject data until the target subject data is not empty under the condition that the target subject data is determined to be empty.

S5140, determining whether the data node to be received identifies second identification data of the node theme; if yes, execute S5150; otherwise, S5160 is performed.

S5150, the consumption of the data transmitted by the node theme is ended.

S5160, determining the node state of the data node to be received, and suspending consuming the data transmitted by the node theme under the condition that the node state is determined to be in a stop execution state until the node state is determined to be in a start execution state, and continuing consuming the data transmitted by the node theme.

According to the technical scheme, the task output is marked and then transmitted to all downstream tasks, and the data represented by the marks are used after being analyzed by the downstream tasks, so that the transmission of a large amount of data can be supported, the complexity of data transmission is reduced, the complexity of executing the tasks by the nodes is reduced, and the execution efficiency of data processing operation is improved.

Example IV

Fig. 6 is a schematic diagram of a data transmission device according to a fourth embodiment of the present invention, as shown in fig. 6, where the device includes: an input data determination module 610, a data transfer indicia determination module 620, and a data transfer module 630, wherein:

An input data determining module 610, configured to determine node input data and a node input data type of a data node to be received;

A data transfer flag determining module 620, configured to determine, according to the node input data, a target data transfer flag corresponding to the data node to be received, if it is determined that the node input data type is a target data type;

And the data transfer module 630 is configured to determine target topic data according to the target data transfer flag, so as to consume the target topic data through the data node to be received.

Optionally, the input data determining module 610 may be specifically configured to: determining at least one data node to be transferred, and creating a node data transfer mark corresponding to each data node to be transferred; determining an available mark list corresponding to the data node to be received according to the data transfer marks of each node; and determining the node input data in the available mark list according to the node input data determining instruction.

Optionally, the input data determining module 610 may be further specifically configured to: determining a node operation sequence list of each operation node; determining a target sequence number of a data node to be received according to the node operation sequence list; determining the operation node with the sequence number smaller than the target sequence number as a target operation node; and in the case that the target job node is determined to be the data node to be transferred, adding the node data transfer mark of the target job node into the available mark list.

Optionally, the data transfer module 630 may be specifically configured to: determining a node subscription theme corresponding to a data node to be transferred according to the node data transfer mark; determining node theme data of node subscription theme corresponding to the data node to be transferred according to the output data of the data node to be transferred; and determining a matched data transfer mark matched with the target data transfer mark in the node data transfer marks, and determining the node theme data corresponding to the matched data transfer mark as target theme data.

Optionally, the node theme data may include first identification data of the node theme, transfer data of the node theme, and second identification data of the node theme; accordingly, the data transfer module 630 may be further specifically configured to: determining the data subject to be transferred according to the output data of the data node to be transferred; when the data node to be received is determined to recognize the first identification data of the node theme, starting to consume the data transmitted by the node theme; and when the data node to be received is determined to recognize the second identification data of the node theme, ending the consumption of the data transmitted by the node theme.

Alternatively, the data transfer device may be specifically configured to: under the condition that the data node to be received does not recognize the second identification data of the node theme, determining the node state of the data node to be received; and under the condition that the node state is determined to be in the execution stopping state, suspending consuming the node theme transfer data until the node state is determined to be in the execution starting state, and continuing consuming the node theme transfer data.

Optionally, the data transfer device may further be specifically configured to: and waiting for consumption of the target subject data until the target subject data is not empty under the condition that the target subject data is determined to be empty.

The data transmission device provided by the embodiment of the invention can execute the data transmission method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

Example five

Fig. 7 shows a schematic diagram of the structure of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Electronic equipment may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 7, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from the storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as data transfer methods.

In some embodiments, the data transfer method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. One or more of the steps of the data transfer method described above may be performed when the computer program is loaded into RAM 13 and executed by processor 11. Alternatively, in other embodiments, the processor 11 may be configured to perform the data transfer method in any other suitable way (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for carrying out methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.

The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims

1. A method of data transfer, comprising:

determining node input data and node input data types of nodes to be received;

2. The method of claim 1, further comprising, prior to said determining the node input data and the node input data type for the data node to be received:

The determining node input data of the data node to be received includes:

3. The method according to claim 2, wherein determining the list of available labels corresponding to the data node to be received according to the data transfer labels of the nodes comprises:

determining a node operation sequence list of each operation node;

4. The method according to claim 2, further comprising, after said creating a node data transfer flag corresponding to each of said data nodes to be transferred:

5. The method of claim 4, wherein the node topic data comprises node topic first identification data, node topic transfer data, and node topic second identification data;

6. The method according to claim 5, further comprising:

7. The method according to claim 1, characterized in that the method further comprises:

8. A data transfer device, comprising:

9. An electronic device, the electronic device comprising:

At least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data transfer method of any one of claims 1-7.

10. A computer readable storage medium storing computer instructions for causing a processor to perform the data transfer method of any one of claims 1-7.