CN116781703A

CN116781703A - Data processing method, device, computer equipment and readable medium

Info

Publication number: CN116781703A
Application number: CN202210232264.4A
Authority: CN
Inventors: 张启明; 熊先奎; 蔡伟博; 杨树林; 虞红芳; 朱炫鹏; 姚海东
Original assignee: ZTE Corp; University of Electronic Science and Technology of China
Current assignee: ZTE Corp; University of Electronic Science and Technology of China
Priority date: 2022-03-09
Filing date: 2022-03-09
Publication date: 2023-09-19
Also published as: WO2023168937A1

Abstract

The disclosure provides a data processing method, which is used for responding to a strategy update request message which is sent by a working node and carries network throughput information of the working node, and updating global network throughput information according to the network throughput information; under the condition that strategy updating request messages sent by all working nodes are received, determining data distribution information according to global network throughput information; and sending a strategy response message carrying the data allocation information to each working node in response to determining that the preset strategy updating condition is met according to the preset first threshold value, the data allocation information and the historical data allocation information. The data aggregation and distribution process of the distributed application carried by the computing power network based on the wide area network is accelerated, and the communication efficiency is improved; and dynamically distributing data volumes of different scales for different sink nodes, so that the bandwidth resources of the current wide area network are efficiently utilized. The present disclosure also provides a data processing method, a data processing apparatus, a computer device, and a readable medium.

Description

Data processing method, device, computer equipment and readable medium

Technical Field

The present disclosure relates to the field of communications technologies, and in particular, to a data processing method, an apparatus, a computer device, and a readable medium.

Background

The distributed machine learning is a typical service in a computing power network scene, the model aggregation and updating process is a representative example of data aggregation and distribution in the computing power network, and the important focus on the transmission scheduling optimization related work of communication in the distributed machine learning model aggregation and updating process is to explore a specific and effective view of the communication primitive optimization work in the computing power network scene. The distributed machine learning transmission scheduling optimization work may be done from a calculation and communication overlap perspective. A priority-based transmission scheme is proposed in the related art, which mainly considers the contradiction between the gradient generation order in machine learning and the order of demand of global gradients (parameters) in forward propagation, and considers giving higher priority to the parameters of the former layer model so as to be acquired earlier by each node, thereby starting forward propagation earlier. In the process, the scheme also considers that the parallelism of uplink and downlink utilization is improved by applying fine granularity transmission, and the core idea is that the smaller granularity can effectively reduce the waiting time delay of merging operation in the data aggregation process, but the specific design is limited to the fixed value segmentation of the data blocks with larger granularity, and how to process the data blocks with too small granularity is not considered. The related art focuses on the fact that the parameter scale of many layers in deep learning neural networks is small, especially CNN (Convolutional Neural Networks, convolutional neural network) networks, and in this case, if the transmission of parameters or gradients is performed with granularity of layers, the transmission efficiency becomes low in consideration of other overhead in each transmission.

Regarding the load balancing work of data aggregation and distribution tasks under a multi-center node, in the related art, mainly aiming at the load balancing problem of a plurality of servers under a distributed machine learning PS architecture, the performance of each server is determined by a statistical and predictive method, and then a part of servers are selected to be allocated with a certain amount of parameters by referring to the performance. In view of the randomness of the dynamic changes of the network, the prediction method is not in fact consistent with the reality in many cases, so that the grasped network information cannot be used well. Furthermore, this scheme does not deeply discuss the problem of fine granularity transmission and merging of too fine granularity in transmission.

A computing network generally needs a wide area network to open a communication channel among computing nodes distributed across domains, and inherent isomerism, scarcity and dynamic characteristics of a wide area network bandwidth cause frequent occurrence of bottleneck links, so that a communication process of distributed services carried by the computing network is significantly blocked and slowed down. In the related art, a scheme of multiple central nodes is adopted, which cannot sense the dynamically changed network state, so that the communication task born by each central node is not matched with the communication capability of the central node, and the overall communication efficiency is low.

Disclosure of Invention

The present disclosure provides a data processing method, apparatus, computer device, and readable medium.

In a first aspect, an embodiment of the present disclosure provides a data processing method applied to a scheduling node in a distributed cluster system of a computing power network, where the method includes:

in response to receiving a policy update request message which is sent by a working node and carries network throughput information of the working node, updating global network throughput information according to the network throughput information, wherein the global network throughput information is used for recording network throughput between any two working nodes;

under the condition that strategy updating request messages sent by all working nodes are received, determining data distribution information according to the global network throughput information, wherein the data distribution information is used for recording the data distribution proportion of each working node;

and transmitting a strategy response message carrying the data distribution information to each working node in response to determining that a preset strategy updating condition is met according to a preset first threshold value, the data distribution information and the historical data distribution information.

In some embodiments, the global network throughput information is a matrix of n x n, the element a of the matrix _ij For the network throughput from node i to node j, i is more than or equal to 1 and less than or equal to n, j is more than or equal to 1 and less than or equal to n, and n is the total number of working nodes in the distributed cluster system;

the determining data allocation information according to the global network throughput information includes:

respectively calculating the sum R of each row element in the matrix _i And the sum C of the column elements in the matrix _j ；

According to the sum R of the elements of each row in the matrix _i Sum C of the elements of each row _j Determining the effective throughput of each working node;

and determining data allocation information according to the effective throughput of each working node and the minimum value of the effective throughput of each working node.

In some embodiments, the meeting the preset policy update condition includes:

the change rate of the data distribution proportion of any one of the working nodes in the data distribution information is larger than a preset first threshold value; the change rate of the data distribution proportion of the working node is calculated according to the data distribution proportion of the working node in the data distribution information and the data distribution proportion of the working node in the historical data distribution information.

In some embodiments, after determining data allocation information from the global network throughput information, the method further comprises:

And responding to the fact that the preset strategy updating condition is met according to the first threshold value, the data distribution information and the historical data distribution information, and updating the historical data distribution information according to the data distribution information aiming at the working node with the change rate larger than the first threshold value.

In yet another aspect, an embodiment of the present disclosure further provides a data processing method applied to a working node in a distributed cluster system of a computing power network, where the method includes:

dividing the data to be processed into a plurality of data fragments, wherein the data volume of each data fragment is smaller than or equal to a second threshold value;

determining a target receiving node of a data fragment to be distributed according to locally stored data distribution information, wherein the locally stored data distribution information is data distribution information carried in a strategy response message sent by a scheduling node;

and sending a second forenotice message and a local data message carrying the data fragments to be distributed to each destination receiving node.

In some embodiments, the splitting the data to be processed into a plurality of data slices includes:

under the condition that the data to be processed is data with an internal structure, the internal structure of the data to be processed, the data quantity of which is larger than a second threshold value of 1/4, is divided into a plurality of data blocks, wherein the data quantity of each data block is smaller than or equal to the second threshold value of 1/4; sequentially merging the data blocks according to the generation sequence of the data blocks to generate data fragments; or splitting the data to be processed into a plurality of data fragments according to the second threshold value under the condition that the data to be processed is the data without the internal structure.

In some embodiments, the second threshold is determined according to a total amount of data of the data to be processed by the working node and a total amount of the working nodes.

In some embodiments, after splitting the data to be processed into a plurality of data slices, further comprising:

and assigning a priority to each data fragment, and generating a priority pair column, wherein the priority pair column comprises each data fragment arranged from high to low.

In some embodiments, the determining the destination receiving node of the data fragment to be allocated according to the locally stored data allocation information includes:

calculating the sum of data distribution proportions of the working nodes according to the locally stored data distribution information;

determining a working node with the data distribution ratio of more than or equal to 1 in the data distribution information as a target receiving node in response to the sum of the data distribution ratios of the working nodes being more than or equal to the number of the data fragments;

and determining all the working nodes as destination receiving nodes in response to the sum of the data distribution ratios of the working nodes being smaller than the number of the data fragments.

In some embodiments, when the sum of the data allocation ratios of the working nodes is greater than or equal to the number of data fragments, the number of data fragments to be received by each destination receiving node is a first number, where the first number is a difference between the data allocation ratio of each destination receiving node and a data fragment adjustment amount, where the data fragment adjustment amount is determined according to the sum of the data allocation ratios of the working nodes, the number of data fragments, and the data allocation ratio of each destination receiving node;

And under the condition that the sum of the data distribution proportion of the working nodes is smaller than the number of the data fragments, the number of the data fragments to be received by each destination receiving node is a second number, and the second number is determined according to the data distribution proportion of each destination receiving node.

calculating the load of each working node according to the total data amount of the data to be processed of each working node and the locally stored data distribution information;

and determining the target receiving node of the data fragment to be distributed according to the load of each working node and the data quantity of the data fragment to be distributed.

In some embodiments, the method further comprises:

responding to a first advance notice message and a global data message sent by other working nodes, and acquiring global data carried in the global data message, wherein the global data is data obtained by performing data processing on data fragments distributed to the other working nodes;

and calculating the network throughput between the working node and the other working nodes according to the data volume of the global data, the receiving time of the first forenotice message and the receiving time of the global data message.

In some embodiments, the method further comprises:

and processing all the data fragments distributed to the working node in response to receiving the all the data fragments.

In some embodiments, the method further comprises:

responding to the received second advance notice message and local data message, and acquiring the data fragments carried in the local data message;

and calculating the network throughput between the corresponding working nodes according to the data volume of the data fragments, the receiving time of the second forenotice message and the receiving time of the local data message.

In some embodiments, the method further comprises:

for the same data fragment, under the condition that the data fragments sent by all working nodes are received, carrying out data merging on the data fragments to obtain global data of the data fragments;

and sending a first advance notice message and a global data message to all working nodes, wherein the global data message carries global data of the data fragments.

In yet another aspect, an embodiment of the present disclosure further provides a data processing apparatus, which is applied to a scheduling node in a distributed cluster system of a computing power network, and includes a throughput information sensing module, a policy making module, and a policy issuing module;

The throughput information sensing module is used for responding to a strategy update request message which is sent by a working node and carries network throughput information of the working node, updating global network throughput information according to the network throughput information, wherein the global network throughput information is used for recording network throughput between any two working nodes;

the policy making module is used for determining data distribution information according to the global network throughput information under the condition that policy update request messages sent by all working nodes are received, and the data distribution information is used for recording the data distribution proportion of each working node;

the policy issuing module is used for sending a policy response message carrying the data allocation information to each working node in response to determining that a preset policy updating condition is met according to a preset first threshold value, the data allocation information and the historical data allocation information.

In yet another aspect, the disclosed embodiments also provide a data processing apparatus, which is applied to a working node in a distributed cluster system of a computing power network, including a data segmentation module, a destination receiving node determining module and a data distribution module,

The data segmentation module is used for segmenting data to be processed into a plurality of data fragments, wherein the data volume of each data fragment is smaller than or equal to a second threshold value;

the destination receiving node determining module is used for determining a destination receiving node of the data fragment to be distributed according to locally stored data distribution information, wherein the locally stored data distribution information is data distribution information carried in a strategy response message sent by a scheduling node;

the data distribution module is used for sending a second advance notice message and a local data message carrying the data fragments to be distributed to each destination receiving node.

In yet another aspect, the disclosed embodiments also provide a computer device, comprising: one or more processors; a storage device having one or more programs stored thereon; the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data processing method as described above.

In yet another aspect, the disclosed embodiments also provide a computer readable medium having a computer program stored thereon, wherein the program when executed implements the data processing method as described above.

The embodiment of the disclosure provides a data processing method, which comprises the following steps: in response to receiving a policy update request message which is sent by a working node and carries network throughput information of the working node, updating global network throughput information according to the network throughput information; under the condition that strategy updating request messages sent by all working nodes are received, determining data distribution information according to global network throughput information; and sending a strategy response message carrying the data allocation information to each working node in response to determining that the preset strategy updating condition is met according to the preset first threshold value, the data allocation information and the historical data allocation information. The embodiment of the disclosure can obviously accelerate the data gathering and distributing process of the distributed application carried by the computing power network based on the wide area network, thereby effectively reducing the communication time of data gathering and distributing and improving the communication efficiency; based on the network throughput sensing technology, data volumes with different scales are dynamically distributed to different sink nodes through a node performance evaluation algorithm, so that a bottleneck link can be effectively avoided, a high-bandwidth resource link is more effectively utilized, and the high-efficiency utilization of the bandwidth resource of the current wide area network is realized.

Drawings

FIG. 1 is a schematic diagram of a system architecture of an embodiment of the present disclosure;

FIG. 2 is a flowchart of a data processing method with a scheduling node as an execution body according to an embodiment of the present disclosure;

fig. 3 is a schematic flow chart of determining data allocation information according to an embodiment of the present disclosure;

FIG. 4 is a flowchart of a data processing method with a working node as an execution body according to an embodiment of the present disclosure;

fig. 5 is a schematic diagram one of a flow of determining a destination receiving node of a data fragment to be allocated according to an embodiment of the present disclosure;

fig. 6 is a schematic diagram two of a flow of determining a destination receiving node of a data fragment to be allocated according to an embodiment of the present disclosure;

FIG. 7 is a flow chart illustrating the calculation of network throughput between working nodes according to an embodiment of the present disclosure;

FIG. 8 is a schematic flow chart of data merging according to an embodiment of the disclosure;

FIG. 9 is a flowchart of a data processing method with a working node as an execution body according to an embodiment of the disclosure;

FIG. 10 is a schematic diagram of a data processing apparatus (scheduling node) according to an embodiment of the present disclosure;

FIG. 11 is a schematic diagram of a data processing apparatus (working node) according to an embodiment of the present disclosure;

FIG. 12 is a second schematic diagram of a data processing apparatus (working node) according to an embodiment of the present disclosure;

FIG. 13 is a third schematic diagram of a data processing apparatus (working node) according to an embodiment of the present disclosure;

fig. 14 is a schematic structural diagram of a data processing apparatus (working node) according to an embodiment of the present disclosure.

Detailed Description

Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

Embodiments described herein may be described with reference to plan and/or cross-sectional views with the aid of idealized schematic diagrams of the present disclosure. Accordingly, the example illustrations may be modified in accordance with manufacturing techniques and/or tolerances. Thus, the embodiments are not limited to the embodiments shown in the drawings, but include modifications of the configuration formed based on the manufacturing process. Thus, the regions illustrated in the figures have schematic properties and the shapes of the regions illustrated in the figures illustrate the particular shapes of the regions of the elements, but are not intended to be limiting.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The embodiment of the disclosure provides a data processing method, which is applied to a distributed cluster system of a computing power network, as shown in fig. 1, wherein the system comprises a scheduling node and a plurality of working nodes, and the scheduling node is responsible for sensing network throughput information and formulating and issuing a strategy according to the network throughput information; the working node is responsible for executing specific data transmission control according to the strategy information. In system deployment, since the scheduling node needs to make a decision according to throughput information of all other working nodes, the scheduling node is deployed on one network device separately. The worker nodes are deployed to all network devices that actually perform computing tasks. The scheduling nodes and all the working nodes form a star logic network, and scheduling information is interacted through a wide area network so as to ensure that the overall decision and control are smoothly carried out; and a logical full-connection network is formed among all the working nodes, and data information is transmitted through a wide area network so as to meet the data requirements of calculation tasks. In the embodiment of the disclosure, the working node needs to send the calculated local data, needs to receive global data sent by other working nodes, and needs to perform data aggregation and distribution on the local data of other working nodes.

The dispatching node internally comprises three functional partitions of network throughput information sensing, strategy making and strategy issuing. The network throughput information sensing partition is responsible for sensing network throughput information among the working nodes executing the distributed tasks, wherein the network throughput information is determined by the working nodes during data transmission and is informed to the scheduling node in a strategy update request message. The network throughput information sensing partition maintains global network throughput information, the global network throughput information is used for recording network throughput between any two working nodes, and when a scheduling node receives a strategy updating request message of a working node, the network throughput information sensing partition updates a value of a corresponding position in the global network throughput information by using the latest network throughput information of the received working node. The policy making partition is a core working area of the scheduling node, and can quantitatively evaluate the communication performance of all the working nodes according to a performance evaluation algorithm based on network throughput information provided by current global network throughput information, then compare the evaluation result with the last round of evaluation result, and trigger policy reset and need to release a new policy if the performance of the working nodes is greatly changed. The policy issuing partition is used for distributing the performance evaluation result to all working nodes. In the distributed service, in order to ensure the synchronization of all working node policies, the working node needs to send a policy update request message to the scheduling node, and after receiving the policy update request message of all working nodes, the scheduling node writes the latest policies into a policy response message and sends the latest policies to the working node.

The functions of the working node mainly comprise policy local updating and data transmission control. Before the working node needs to start a new round of data transmission, a strategy update request message needs to be sent to the dispatching node to inquire whether the local transmission strategy needs to be updated. After receiving the strategy response message of the dispatching node, the working node strategy locally updates the strategy in the partition reading strategy response message, and then redistributes the data block scale sent to different working nodes so as to minimize the transmission delay and prepare for the subsequent data transmission control. The data transmission control partition is responsible for optimizing granularity fragmentation of data according to a current strategy, distributing sending priority to each fragmentation according to the internal relation of the data, and then sending and receiving the data. Thus, the send thread, receive thread, and service thread are created at the worker node, respectively, to perform this portion of the functionality. For the sending thread of the working node, the data transmission control partition is responsible for data slicing and assigns priority to each slice, and then sends local data messages (one or more data slices) of different sizes to different working nodes according to a given strategy. For the receiving thread, the data transmission control partition is responsible for receiving the global data message after merging of the working nodes and carrying out local subsequent processing. Whereas for the service threads of the working nodes, the data transfer control partition is responsible for receiving local data from other working nodes and distributing global data to other working nodes, where the global data is typically merged from the received local data.

An embodiment of the present disclosure provides a data processing method, where the method is applied to a scheduling node in a distributed cluster system of a computing power network, as shown in fig. 2, and the data processing method includes the following steps:

and step 21, in response to receiving a policy update request message which is sent by the working node and carries network throughput information of the working node, updating global network throughput information according to the network throughput information.

The global network throughput information (THRUPUT_TABLE) is used for recording the network throughput between any two working nodes, and the network throughput information between the working nodes can reflect the communication performance and the network state of the working nodes. In this step, the policy update request message sent by each working node to the scheduling node carries the network throughput information of the working node, and the scheduling node updates the corresponding location of the global network throughput information based on the network throughput information of each working node.

Step 22, determining data allocation information according to the global network throughput information when the policy update request message sent by all the working nodes is received.

The data distribution information (route_list) is used for recording the data distribution proportion of each working node, wherein the data distribution proportion of the working node is the proportion of the data scale of the working node responsible for aggregation and distribution to the whole data scale.

Step 23, in response to determining that the preset policy update condition is met according to the preset first threshold, the data allocation information and the historical data allocation information, sending a policy response message carrying the data allocation information to each working node.

The data distribution information (routing_list) is a data processing policy of the whole power network, and the history data distribution information (LOG) is used for recording the data distribution proportion of the history of each working node. In this step, the scheduling node determines whether a preset policy update condition is satisfied according to a preset first THRESHOLD (partition_threshold), data allocation information (partition_list) and history data allocation information (LOG), and when determining that policy update is required, the scheduling node issues a new policy to each working node, where the new policy is the data allocation information (partition_list) obtained in step 22. When it is determined that policy update is not required, the scheduling node returns a null policy response message to each working node.

In some embodiments, the global network throughput information (THRUPUT_TABLE) is a matrix of n, element a of the matrix _ij For the network throughput from node i to node j, 1.ltoreq.i.ltoreq.n, 1.ltoreq.j.ltoreq.n, where n is the total number of working nodes in the distributed cluster system, the matrix may be expressed as:

wherein a is _ij (i=j) =0 means that node i itself has no network throughput, initially a _ij (i+.j) random.

As shown in fig. 3, the determining data allocation information according to the global network throughput information includes the following steps:

step 221, calculating the sum R of the elements of each row in the matrix _i And the sum C of the elements of each column in the matrix _j 。

In this step, the sum R of the elements of each row in the matrix is calculated _i ，And calculates the sum C of the elements of each column in the matrix _j ，/>Exemplary, R ₁ ＝a ₁₁ +a ₁₂ +…+a _1n ,R ₂ ＝a ₂₁ +a ₂₂ +…+a _2n ；C _1＝ a ₁₁ +a ₂₁ +…+a _n1 ,C ₂ ＝a ₁₂ +a ₂₂ +…+a _n2 。

Step 222, according to the sum R of the elements of each row in the matrix _i Sum C of the elements of each row _j The effective throughput of each working node is determined.

The effective throughput of the working node i is SUM_C [ i ]]，SUM_C[i]＝MIN(R _i ，C _j ) Where j=i. That is, R is taken _i And C _j Effective throughput sum_c [ i ] with minimum value in (i=j) as working node i]. Illustratively, the effective throughput of the working node 1 is MIN (R ₁ ，C ₁ ) The effective throughput of the working node 2 is MIN (R ₂ ，C ₂ )。

Step 223, determining data allocation information according to the effective throughput of each working node and the minimum value of the effective throughput of each working node.

Data distribution ratio b of data distribution information (route_list) for each working node _i Is set of (b) _i Indicating that the working node i should be responsible for aggregating and distributing the proportion of the data size to the total data size, the corresponding data allocation information (route_list) may be expressed as: [ b ] ₁ b ₂ b ₃ … b _n ]。

Data distribution ratio b of working node i _i The method can be obtained by calculation according to the following formula (1):

where MIN (SUM_Cj) represents the minimum of the effective throughput of each worker node, ROUND represents the nearest rounding, and SUM_Ci represents the effective throughput of each worker node.

After the data allocation proportion of each working node is obtained according to the formula (1), the data allocation information (ROPORTION_LIST) can be obtained.

In some embodiments, the meeting the preset policy update condition includes: the RATE of CHANGE (change_rate) of the data allocation proportion of any one of the working nodes in the data allocation information (route_list) is greater than a preset first THRESHOLD (allocation_threshold). The CHANGE RATE (change_rate) of the data distribution ratio of the working node is calculated according to the data distribution ratio (report_list i) of the working node in the data distribution information and the data distribution ratio (DICITION_LOG i) of the working node in the history data distribution information (DICITION_LOG).

That is, the CHANGE RATE change_rate [ i ] of the data allocation ratio of the working node i is compared with the first THRESHOLD (discard_threshold) of the set trigger policy issue in the traversal order, and the policy update and issue is triggered as long as one value of change_rate [ i ] is greater than the first THRESHOLD.

The RATE of CHANGE of the data allocation ratio of the working node i, change_rate [ i ], can be calculated by the following formula (2):

wherein DICISION_LOG [ i ] represents the historical data distribution information of the working node i (namely the historical data distribution PROPORTION of the working node i), and PROPORTION_LIST [ i ] represents the data distribution PROPORTION of the working node i.

In some embodiments, after determining the data allocation information (i.e., step 22) from the global network throughput information, the data processing method may further include the steps of: in response to determining that the preset policy update condition is satisfied according to the first THRESHOLD (partition_threshold), the data allocation information (partition_list), and the history data allocation information (LOG), the history data allocation information (LOG) is updated according to the data allocation information (partition_list) for the working node whose CHANGE RATE (change_rate) is greater than the first THRESHOLD. That is, when it is determined that policy update is required, the scheduling node locally updates the history data allocation information of the working node with a large change in communication performance.

The embodiment of the disclosure also provides a data processing method, which is applied to a working node in a distributed cluster system of a computing power network, as shown in fig. 4, and comprises the following steps:

and step 41, segmenting the data to be processed into a plurality of data fragments, wherein the data volume of each data fragment is smaller than or equal to a second threshold value.

The working node segments the computationally generated data based on a second threshold (SLICE_SIZE) to obtain a data SLICE. The second threshold (slicesize) may be determined according to the total amount of data of the data to be processed by the working node and the total amount of the working nodes. In some embodiments of the present invention, in some embodiments,m is the total data amount of the data to be processed of the working node i (the working node i is the working node for data segmentation), N is the total number of the working nodes in the distributed cluster system of the computing power network, α is a coefficient, α=1.2x10 ⁵ 。

In this step, different data slicing schemes are adopted for different data types. Under the condition that the data to be processed is the data with the internal structure, the internal structure of which the data volume is larger than a second threshold value of 1/4 in the data to be processed is divided into a plurality of data blocks, wherein the data volume of each data block is smaller than or equal to the second threshold value of 1/4, and the data blocks are combined in sequence according to the generation sequence of the data blocks to generate the data fragments. And under the condition that the data to be processed is the data without the internal structure, dividing the data to be processed into a plurality of data fragments according to a second threshold value.

And step 42, determining a target receiving node of the data fragment to be distributed according to the locally stored data distribution information, wherein the locally stored data distribution information is the data distribution information carried in the strategy response message sent by the scheduling node.

In case of policy update, the scheduling node issues a new policy to each of the working nodes, i.e., sends current data allocation information (route_list) to each of the working nodes through a policy response message, and each of the working nodes locally stores the data allocation information (route_list). In this step, each working node determines a destination receiving node of the data fragment to be allocated according to locally stored data allocation information (reporting_list), and its specific implementation will be described in detail later.

Step 43, sending the second advance notice message and the local data message carrying the data fragment to be allocated to each destination receiving node.

In the step, all the data fragments to be distributed are sent to each destination receiving node by each working node, so that the data distribution is realized. For each data fragment to be allocated, the sending thread judges a target receiving node of the data fragment, if the target receiving node is self, the sending thread sequentially sends a second advance notice message and a local data message to a service thread of the working node, and if the target receiving node is other working nodes, the sending thread sequentially sends the second advance notice message and the local data message to each target receiving node.

According to the data processing method provided by the embodiment of the disclosure, certain independence in distributed application data is considered, and the data with better granularity is transmitted by splitting the larger data block before data transmission, so that the data fragments with the optimized granularity are independently subjected to subsequent data operation after being collected by the aggregation nodes, the problem that all data operations are blocked for a long time is solved, and the acquisition and utilization of the data by the application are accelerated.

In some embodiments, after splitting the data to be processed into a plurality of data slices (i.e. step 41), the data processing method may further comprise the steps of:

step 42' assigns priorities to the data chunks and generates a priority pair column that includes each of the data chunks arranged from high to low.

In this step, priority labels are added to each data slice, and the working node may allocate priorities (may also be determined according to specific requirements) to each data slice according to the sequence in which the data slices are generated, where the priority of the data slice that is generated first is the lowest and the priority of the data slice that is generated last is the highest. The working node maintains a priority queue, and the data fragments marked with the priority labels are pushed into the priority queue to enter a waiting state. By adding different priority labels to different data fragments, the priority transmission of more important data in a transmission queue is realized, and the acquisition and utilization of important data by application are further accelerated.

Steps 41-43 and step 42' are processed by the sending thread of the working node, and after all the data fragments to be allocated are distributed, the sending thread ends.

Two specific implementations of determining the destination receiving node of the data fragment to be allocated are described below with reference to fig. 5 and 6, respectively.

In some embodiments, as shown in fig. 5, the determining the destination receiving node of the data fragment to be allocated according to the locally stored data allocation information (i.e. step 42) includes the following steps:

step 421, calculating the sum of the data distribution ratios of the working nodes according to the locally stored data distribution information.

In this step, the sending thread of the work node calculates the SUM of the elements (i.e., the data allocation ratio of each work node) in the data allocation information (route_list), denoted by sum_p.

Step 422, determining whether the sum of the data allocation ratios of the working nodes is greater than or equal to the number of data slices, if yes, executing step 423, otherwise, executing step 424.

In the step, judging the size relation between SUM_P and the number S of the local data fragments, if SUM_P is more than or equal to S, indicating that the data fragments are less, and selecting part of working nodes to distribute data to the working nodes; if SUM_P < S, the data fragments are more, and the data is distributed to all working nodes.

Step 423, determining that the working node with the data allocation ratio of greater than or equal to 1 in the data allocation information is the destination receiving node.

In the case where SUM_P is equal to or greater than S, the working nodes with a data allocation ratio >1 are determined based on the data allocation information (ROPORTION_LIST), and these working nodes are used as destination receiving nodes.

In step 424, all working nodes are determined as destination receiving nodes.

In the case of sum_p < S, all working nodes are the destination receiving node.

In some embodiments, when the SUM (sum_p) of the data allocation ratios of the working nodes is greater than or equal to the number (S) of data fragments, the number of data fragments to be received by each destination receiving node is a first number, and the first number is a difference between the data allocation ratio (report_list [ i ]) of each destination receiving node and the data fragment adjustment amount (REDUCTION), and the data fragment adjustment amount (REDUCTION) is determined according to the SUM (sum_p) of the data allocation ratios of the working nodes, the number (S) of data fragments, and the data allocation ratio of each destination receiving node.

In some embodiments, the difference (P_S) between the SUM of the data allocation ratios (SUM_P) of each working node and the number of data fragments (S) is calculated, and the SUM of the difference (P_S) and the SUM of the data allocation ratios (Sigma) of each destination receiving node is calculated _i PROPORTION_LIST[i]) And, expressed by SUM_MORE_1, wherein each destination receiving node is the working node with the data allocation ratio greater than or equal to 1. Based on the difference (P_S), the data distribution ratio (processor_LIST [ i ]]) And SUM_MORE_1, respectively calculating the data fragment adjustment (REDUCTION) of each destination receiving node. The first number of data fragments to be received by each destination receiving node is the data allocation PROPORTION (report_list [ i ] of each destination receiving node]) Data fragment adjustment amount (REDUCTION [ i ]]) I.e. the first number of data fragments to be received by the destination receiving node i = processing_list [ i ]]-REDUCTION[i]。

Wherein the data fragment adjustment amount (REDUCTION [ i ]) of the destination receiving node i can be calculated according to the following formula (3):

REDUCTION[i]＝CEIL(P_S*PROPORTION_LIST[i]/SUM_MORE_1) (3)

in some embodiments, in a case where the SUM (sum_p) of the data allocation ratios of the working nodes is smaller than the number (S) of data fragments, the number of data fragments to be received by each destination receiving node is a second number, which is determined according to the data allocation ratio of each destination receiving node. It should be noted that, when all the working nodes are used as destination receiving nodes and all the data fragments to be allocated cannot be completely allocated, the data fragments which are not allocated may be allocated again according to the data allocation proportion of each working node in the data allocation information (request_list) until the allocation of all the data fragments to be allocated is completed.

Exemplary, a destination receiving node LOOKUP table LOOKUP_LIST INDEX may be set _i ]=i, wherein,

INDEX _i ∈[BEGIN _i ,END _i ]，i∈(1,n)。

BEGIN＝{0,PROPORTION_LIST[0],PROPORTION_LIST[:1],…,PROPORTION_LIST[:i-1],…,PROPORTION_LIST[:n-1]}；

END＝{PROPORTION_LIST[0]-1,PROPORTION_LIST[:1]-1,…,PROPORTION_LIST[:i]-1,…,PROPORTION_LIST[:n]-1}。

wherein, the process_list [: j ] =process_list [0] +process_list [1] + … +process_list [ j ].

And sequentially distributing corresponding data fragments to be distributed (the number of the distributed data fragments is the data distribution proportion value of the current working node) for the elements in LOOKUP_LIST (namely all the working nodes) according to the sequence of the data fragments to be distributed by the sending thread in a circulating traversing mode until all the data fragments to be distributed are distributed.

In some embodiments, as shown in fig. 6, the method for determining the destination receiving node of the data fragment to be allocated according to the locally stored data allocation information (i.e. step 42) includes the following steps:

step 421', calculating the load of each working node according to the data total amount of the data to be processed of each working node and the locally stored data distribution information.

In this step, the load of the working node i can be calculated according to the following formula:wherein M is the total data amount of the data to be processed of the working node i, and is the data amount of the working node i]The data of the working node i is allocated a proportion i epsilon (1, n).

Step 422' determines the destination receiving node of the data fragment to be allocated according to the load of each working node and the data amount of the data fragment to be allocated.

In this step, the differences between the LOAD of each working node and the data amount of the data fragment to be currently allocated are calculated, and the maximum value MAX (LOAD) _i -PARAS _INDEX ) And taking the working node corresponding to the maximum value as a target receiving node of the data fragment to be distributed.

The difference between the load of the working node i and the data amount of the current data slice to be allocated can be calculated according to the following formula (4):

difference value _i ＝LOAD _i -PARAS _INDEX (4)

Wherein INDEX represents the identification of the data fragment to be allocated currently, PARAS _INDEX LOAD for data volume of data fragment INDEX _i Is the load of the working node i.

In the embodiment of the disclosure, when the working node determines the destination receiving node of each local data fragment, the working node adopts the cyclic traversal determination method under the guidance of the bandwidth maximization idea under the minimum competition, so that the data can be distributed as soon as possible under the condition that the generation speed of the data is not far greater than the transmission speed of the data. However, if the data generation speed is far greater than the data transmission speed in practice, a simple naive round robin arrangement method (determining the number of slices that each working node can take charge of based on the total number of slices, then sequentially selecting each node as the destination receiving node of the current slice in a single sequential round robin manner, and skipping a certain node when the accumulated number of slices of the node reaches the number of slices that each node can take charge of, then sequentially allocating a sufficient number of data slices to each working node according to the order of the working nodes) and a target node sequential arrangement method (determining the number of data slices that each working node can take charge of based on the total number of data slices) may be adopted for simplification.

The embodiment of the disclosure provides a data slicing allocation scheme based on bandwidth maximization under minimum competition, and periodically allocates proportional data slicing amounts to each working node, so that the problem of early blocking of low communication capacity nodes under a naive cyclic arrangement method can be avoided, and the problem that bandwidth resources of each working node cannot be simultaneously utilized under a target receiving node sequential arrangement method can be avoided, thereby improving the overall utilization rate of a network and enabling a transmission process to be completed faster and earlier.

For each working node, a receiving thread and a service thread are created in addition to a sending thread, and the sending thread, the receiving thread and the service thread can be executed in parallel.

The processing procedure of the reception thread is described in detail below. As shown in fig. 7, the data processing method may further include the steps of:

in step 71, global data carried in the global data message is obtained in response to receiving the first advance notice message and the global data message sent by other working nodes, where the global data is data obtained after data processing is performed on the data fragments sent to the other working nodes.

In this step, the receiving thread of the working node sequentially receives the first advance notice message and the global data message, records the receiving time t1 of the first advance notice message and the receiving time t2 of the global data message, and obtains global data in the global data message. The working node performs merging processing such as aggregation on the distributed data fragments (local data) to obtain global data, and sends the global data to other working nodes through global data messages.

Step 72, calculating the network throughput between the working node and other working nodes according to the data volume of the global data, the receiving time of the first forenotice message and the receiving time of the global data message.

The receiving thread of the worker node determines the network throughput between worker node i and worker node j that sent the global data message according to the following equation (5):

wherein S is _ij The amount of global data sent to working node i for working node j.

It should be noted that, the network throughput between the same nodes needs to be cumulatively calculated as an average value.

In the disclosed embodiment, the measurement of the throughput of the network between the nodes is determined by locating the start and stop time of the data transmission and combining the data amount contained in the data message. Wherein the start time of the data transmission is determined by the time when the working node receiving the message receives the advance notice message and the finish time of the data transmission is determined by the time when the working node receiving the message receives the data message, which is arranged for the convenience of the upper layer timing operation. One possible, more preferable, approach is to time directly based on the protocol of the transport layer, for example TCP (Transmission Control Protocol ), and to locate the timestamp time-counting based on the ACK (response) acknowledgement of the completion of the entire data stream transmission. However, the implementation process of the scheme is more complex, the difficulty is greater, the modification of the underlying base library is related, the compatibility is difficult to achieve, and the system integration is inconvenient.

In some embodiments, the data processing method further comprises the steps of: and processing all the data fragments in response to receiving all the data fragments distributed to the working node. In the step, a receiving thread of a working node judges whether all data fragments distributed to the working node are received or not, if all the data fragments are received, all the data fragments are processed, namely, the follow-up operation of local data is executed, and then the receiving thread is ended; if not, step 71 is performed.

The processing procedure of the service thread is described in detail below.

The data processing method may further include the steps of: and responding to the received second forenotice message and the local data message, and acquiring the data fragments carried in the local data message. The data fragments carried in the local data message are local data, and are distributed data fragments. According to the data volume S of the data fragments _ij The network throughput between the corresponding working nodes is calculated according to the', the receiving time t1' of the second advance notice message and the receiving time t2' of the local data message. The particular implementation of the service thread of the working node to calculate network throughput is similar to the particular implementation of the receiving thread to calculate network throughput, except that: data volume S using local data _ij ' data amount S instead of global data _ij The reception time t1 'of the second advance notice message is used for replacing the reception time t1 of the first advance notice message, and the reception time t2' of the local data message is used for replacing the reception time t2 of the global data message.

In some embodiments, as shown in fig. 8, the data processing method further includes the steps of:

and 81, carrying out data merging on the data fragments to obtain global data of the data fragments under the condition that the data fragments transmitted by all working nodes are received for the same data fragments.

In this step, when the service thread of the working node receives the same data fragment sent by all the working nodes, data merging processing such as data aggregation is performed on the data fragment, so as to obtain global data of the data fragment.

After receiving the data, the service thread of the working node usually needs to execute a data merging operation to merge the same data fragments from all the working nodes, and it should be noted that the merging operation is optional or not necessary, so as to realize the effect of AllGather (global collection), and after receiving the local data of other working nodes, each working node can directly distribute the local data to each working node in turn without any operation, thus completing broadcasting, and the working node only plays a role of central summarization.

Step 82, sending a first advance notice message and a global data message to all working nodes, wherein the global data message carries global data of the data fragments.

Judging whether all data fragments responsible for the working node are processed by the service thread of the working node, and if all the processing is finished, ending the service thread; if not, continuing to process the data fragments responsible for the working node in the mode of steps 81-82.

In order to clearly illustrate the technical solution of the embodiments of the present disclosure, the following details of the process of performing data processing on a working node are described with reference to fig. 9 and a specific example. As shown in fig. 9, the working node performs the following data processing procedure:

step 1: after the working node is ready, the data is subjected to slicing processing, a priority label is added, and a strategy update request message is sent to the scheduling node to request strategy update.

Step 2: the working node waits for receiving the strategy response message from the dispatching node, when receiving the strategy response message which is not empty, executing strategy updating, and then turning to the step 3; if the policy response message is empty, step 3 is directly executed.

Step 3: the working nodes respectively create a sending thread, a receiving thread and a service thread, wherein the sending thread is responsible for sending local data to other nodes for gathering respectively, the receiving thread is responsible for receiving global data for next calculation, the service thread is responsible for receiving and merging the local data sent by other working nodes so as to obtain global data, and then the global data is distributed to other working nodes. The sending thread performs steps 4-5 (i.e., the left column of steps in fig. 9), the receiving thread performs steps 6-7 (i.e., the middle column of steps in fig. 9), the serving thread performs steps 8-10 (i.e., the right column of steps in fig. 9), and the sending thread, the receiving thread, and the serving thread may be executed synchronously.

Step 4: the sending thread selects one data fragment and determines its destination receiving node and then sends a second advance notice message and a local data message (carrying the selected data fragment) to the destination receiving node.

Step 5: and (4) judging whether all the data fragments are transmitted by the transmitting thread, if so, ending the transmitting thread, and otherwise, turning to the step (4).

Step 6: the receiving thread waits to receive the first advance notice message and the global data message from the other working nodes, calculates the throughput in case of sequential reception, wherein the throughput between the same nodes needs to be cumulatively averaged, and then goes to step 7.

Step 7: and the receiving thread judges whether all the data fragments are received, if so, the receiving thread executes the follow-up operation of the local data, and then the receiving thread is ended, otherwise, the step 6 is carried out.

Step 8: the service thread waits to receive the second advance notice message and the local data message from the working node, calculates the throughput in the case of sequential reception (the specific procedure is the same as in step 6), and then goes to step 9.

Step 9: the service thread judges whether the current data fragment sent by all the working nodes is received, if yes, the data merging of the data fragment is completed, then a first advance notice message and a message containing global data of the data fragment merged data (namely global data) are sequentially sent to all other working nodes, and the step 10 is carried out; otherwise, go to step 8.

Step 10: and (3) judging whether all the data fragments responsible for the working node are processed by the service thread, if so, ending the service thread, otherwise, turning to the step (8).

The embodiment of the disclosure is established on the premise that the network cannot change in a short time, and the next iteration strategy is planned by using the network state of the last iteration, so that the working node defaults to inquire about strategy updating when each round of data transmission starts. Considering a specific use scenario, if the network state of the environment is stable, the policy update inquiry can be executed again after a certain number of rounds.

According to the embodiment of the disclosure, based on multi-center communication task load balancing, each working node is treated as a center node by default, and in fact, if a real scene is not needed, only part of working nodes can be selected as the center nodes to participate in data slicing aggregation, and the working nodes of non-center nodes are not required to be opened up with service threads and calibrated at a scheduling node.

According to the embodiment of the disclosure, the problem of different time delays caused by independent transmission of the data with the large granularity is solved by simultaneously dividing the data with the large granularity and combining the data with the small granularity, and the transmission priority strategy is used to ensure early transmission of more important data, so that effective calculation and communication overlap are realized. In addition, the data volume sent to each sink node (namely the destination receiving node) is dynamically adjusted according to the network throughput, so that load balancing of sink tasks aiming at real-time network throughput is realized in a more general dynamic network and multi-sink node scene, the generation of bottleneck links is avoided, the utilization rate of the network is improved, and the sink and distribution of the dynamic network are more efficient.

The key points of the embodiment of the disclosure are network throughput monitoring, a policy formulation optimization algorithm based on network information, an optimization granularity priority data transmission mechanism, a bandwidth maximization fragmentation allocation method under minimum competition and the like. The network throughput monitoring mechanism determines uplink and downlink throughput among nodes in the network in the data transmission process, and provides a basis for policy making. The policy making optimization algorithm evaluates the communication performance of the working nodes based on the uplink and downlink throughput of the network, dynamically distributes the data which each node is responsible for, and realizes the load balancing of the aggregation and distribution tasks. The optimized granularity priority data transmission mechanism reduces the long blocking time in data merging under larger granularity through granularity refinement, avoids the time delay caused by single undersize data transmission by merging a plurality of undersize data, and sends important data earlier in a transmission priority mode, thereby facilitating early start of later work and accelerating global process. The method for distributing the fragments with the maximized bandwidth under the minimum competition can further effectively improve the utilization rate of network bandwidth resources and accelerate the completion of transmission.

The embodiment of the disclosure optimizes transmission scheduling aiming at the characteristics of the wide area network. Because of the scarcity of wide area network bandwidth, the necessity of optimizing transmission efficiency by transmission scheduling is only present; due to the heterogeneity of wide area network bandwidths, the feasibility of avoiding bottleneck links through proper transmission scheduling optimization is only achieved; because of the bandwidth dynamics of wide area networks, there is a need to sense the network state in real time through network sensing and adjust the transmission policy in time. The scheme of the embodiment of the disclosure completely covers three characteristics of the wide area network and realizes the transmission scheduling optimization with strong pertinence. The embodiment of the disclosure can sense the current network state through the measurement of the current network, and on the premise that the network state is kept basically stable in a short time, the current network state is used for deciding the data distribution proportion of each central node, and the decision result is used in a future period of time. Therefore, the data aggregation and distribution of the distributed service realized by using the embodiment of the disclosure has obvious iterative characteristics, and can be applied for a sufficient time for executing network sensing and decision results, so that the dynamic decision achieves an ideal effect.

Based on the same technical concept, the embodiment of the present disclosure further provides a data processing apparatus, as shown in fig. 10, where the data processing apparatus is applied to a scheduling node in a distributed cluster system of a computing power network, and includes a throughput information sensing module 101, a policy formulation module 102, and a policy issuing module 103.

The throughput information sensing module 101 is configured to, in response to receiving a policy update request message sent by a working node and carrying network throughput information of the working node, update global network throughput information according to the network throughput information, where the global network throughput information is used to record network throughput between any two working nodes.

The policy making module 102 is configured to determine, when a policy update request message sent by all working nodes is received, data allocation information according to the global network throughput information, where the data allocation information is used to record a data allocation proportion of each working node.

The policy issuing module 103 is configured to send a policy response message carrying the data allocation information to each working node in response to determining that a preset policy update condition is met according to a preset first threshold, the data allocation information and the historical data allocation information.

In some embodiments, the global network throughput information is a matrix of n x n, the element a of the matrix _ij For the network throughput from node i to node j, i is more than or equal to 1 and less than or equal to n, j is more than or equal to 1 and less than or equal to n, and n is the total number of working nodes in the distributed cluster system; the policy making module 102 is configured to calculate a sum R of elements of each row in the matrix _i And the sum C of the column elements in the matrix _j The method comprises the steps of carrying out a first treatment on the surface of the According to the sum R of the elements of each row in the matrix _i Sum C of the elements of each row _j Determining the effective throughput of each working node; and determining data allocation information according to the effective throughput of each working node and the minimum value of the effective throughput of each working node.

In some embodiments, the meeting the preset policy update condition includes:

In some embodiments, the throughput information sensing module 101 is further configured to, after the policy making module 102 determines the data allocation information according to the global network throughput information, update the historical data allocation information according to the data allocation information for the working node with the change rate greater than the first threshold in response to determining that a preset policy update condition is met according to the first threshold, the data allocation information and the historical data allocation information.

Based on the same concept, the embodiment of the present disclosure further provides a data processing apparatus, as shown in fig. 11, where the data processing apparatus is applied to a working node in a distributed cluster system of a computing power network, and includes a data splitting module 201, a destination receiving node determining module 202, and a data distributing module 203.

The data slicing module 201 is configured to slice data to be processed into a plurality of data slices, where a data amount of each data slice is less than or equal to a second threshold.

The destination receiving node determining module 202 is configured to determine a destination receiving node of a data partition to be allocated according to locally stored data allocation information, where the locally stored data allocation information is data allocation information carried in a policy response message sent by a scheduling node.

The data distribution module 203 is configured to send a second advance notice message and a local data message carrying the data fragments to be allocated to each destination receiving node.

In some embodiments, the data slicing module 201 is configured to slice, in a case where the data to be processed is data having an internal structure, the internal structure of the data to be processed having a data size greater than a second threshold of 1/4 into a plurality of data blocks, where the data size of each data block is less than or equal to the second threshold of 1/4; sequentially merging the data blocks according to the generation sequence of the data blocks to generate data fragments; or splitting the data to be processed into a plurality of data fragments according to the second threshold value under the condition that the data to be processed is the data without the internal structure.

In some embodiments, the data slicing module 201 is configured to determine the second threshold according to a total amount of data of the data to be processed by the working node and a total amount of the working nodes.

In some embodiments, as shown in fig. 12, the data processing apparatus further includes a priority module 204, where the priority module 204 is configured to assign priorities to the data slices, and generate a priority pair column, where the priority pair column includes the data slices arranged from high to low.

In some embodiments, the destination receiving node determining module 202 is configured to calculate a sum of data allocation proportions of the working nodes according to the locally stored data allocation information; determining a working node with the data distribution ratio of more than or equal to 1 in the data distribution information as a target receiving node in response to the sum of the data distribution ratios of the working nodes being more than or equal to the number of the data fragments; and determining all the working nodes as destination receiving nodes in response to the sum of the data distribution ratios of the working nodes being smaller than the number of the data fragments.

In some embodiments, the data distribution module 203 is configured to, when the sum of the data distribution ratios of the working nodes is greater than or equal to the number of data slices, determine, for each destination receiving node, a first number of data slices to be received by each destination receiving node, where the first number is a difference between the data distribution ratio of each destination receiving node and a data slice adjustment amount, where the data slice adjustment amount is determined according to the sum of the data distribution ratios of the working nodes, the number of data slices, and the data distribution ratio of each destination receiving node; and under the condition that the sum of the data distribution proportion of the working nodes is smaller than the number of the data fragments, the number of the data fragments to be received by each destination receiving node is a second number, and the second number is determined according to the data distribution proportion of each destination receiving node.

In some embodiments, the destination receiving node determining module 202 is configured to calculate a load of each of the working nodes according to a total data amount of data to be processed by each of the working nodes and locally stored data allocation information; and determining the target receiving node of the data fragment to be distributed according to the load of each working node and the data quantity of the data fragment to be distributed.

In some embodiments, as shown in fig. 13, the data processing apparatus further includes a network throughput computing module 205, where the network throughput computing module 205 is configured to, in response to receiving a first advance notice message and a global data message sent by another working node, obtain global data carried in the global data message, where the global data is data obtained by performing data processing on data fragments sent to the other working node; and calculating the network throughput between the working node and the other working nodes according to the data volume of the global data, the receiving time of the first forenotice message and the receiving time of the global data message.

In some embodiments, as shown in fig. 14, the data processing apparatus further includes a data processing module 206, where the data processing module 206 is configured to process all the data slices distributed to the working node in response to receiving the all the data slices.

In some embodiments, the network throughput computing module 205 is further configured to, in response to receiving the second advance notice message and the local data message, obtain a data fragment carried in the local data message; and calculating the network throughput between the corresponding working nodes according to the data volume of the data fragments, the receiving time of the second forenotice message and the receiving time of the local data message.

In some embodiments, the data processing module 206 is further configured to, for the same data slice, perform data merging on the data slice to obtain global data of the data slice when the data slice sent by all working nodes is received; and sending a first advance notice message and a global data message to all working nodes, wherein the global data message carries global data of the data fragments.

The embodiment of the disclosure is applicable to data aggregation and distribution scenes in various large-scale or large-data-volume distributed services carried by a wide area network-based computing network. From the perspective of comprehensive utilization of the solution, the solution of the embodiments of the present disclosure is particularly suitable for global model convergence for distributed machine learning and federal learning. On the technical application scene, the model convergence process and the data collection process of the control module based on the machine learning algorithm in the intelligent factory, the internet of vehicles and the Cloud VR scene can use the scheme of the embodiment of the disclosure, such as the collection of a large amount of sensor data in the intelligent factory, the information of different devices (such as traffic lights, cameras and intelligent mark identifiers) on the road side needed to be acquired by an edge server in the internet of vehicles, the unified scheduling and the acquisition of intermediate data of each working node by a Cloud node in the Cloud VR scene.

The disclosed embodiments also provide a computer device comprising: one or more processors and a storage device; wherein the storage device stores one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the data processing method provided in the foregoing embodiments.

The disclosed embodiments also provide a computer readable medium having a computer program stored thereon, wherein the computer program, when executed, implements the data processing method as provided by the foregoing embodiments.

Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, functional modules/units in the apparatus disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between the functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed cooperatively by several physical components. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as known to those skilled in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Furthermore, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted in a generic and descriptive sense only and not for purpose of limitation. In some instances, it will be apparent to one skilled in the art that features, characteristics, and/or elements described in connection with a particular embodiment may be used alone or in combination with other embodiments unless explicitly stated otherwise. It will therefore be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the scope of the present invention as set forth in the following claims.

Claims

1. A data processing method, applied to a scheduling node in a distributed cluster system of a computing power network, the method comprising:

2. The method of claim 1 wherein the global network throughput information is a matrix of n x n, element a of the matrix _ij For the network throughput from node i to node j, i is more than or equal to 1 and less than or equal to n, j is more than or equal to 1 and less than or equal to n, and n is the total number of working nodes in the distributed cluster system;

3. The method of claim 1, wherein the meeting of the preset policy update condition comprises:

4. The method of claim 3, wherein after determining data allocation information from the global network throughput information, the method further comprises:

5. A data processing method, applied to a working node in a distributed cluster system of a computing power network, the method comprising:

6. The method of claim 5, wherein slicing the data to be processed into a plurality of data slices comprises:

7. The method of claim 5, wherein the second threshold is determined based on a total amount of data to be processed by the worker node and a total amount of worker nodes.

8. The method of claim 5, further comprising, after slicing the data to be processed into a plurality of data slices:

9. The method of claim 5, wherein the determining the destination receiving node for the data fragment to be allocated based on the locally stored data allocation information comprises:

10. The method of claim 9, wherein in the case where a sum of the data allocation ratios of the working nodes is greater than or equal to the number of data fragments, the number of data fragments to be received by each of the destination receiving nodes is a first number, the first number being a difference between the data allocation ratio of each of the destination receiving nodes and a data fragment adjustment amount determined according to the sum of the data allocation ratios of the working nodes, the number of data fragments, and the data allocation ratio of each of the destination receiving nodes;

11. The method of claim 5, wherein the determining the destination receiving node for the data fragment to be allocated based on the locally stored data allocation information comprises:

12. The method of claim 5, wherein the method further comprises:

13. The method of claim 12, wherein the method further comprises:

14. The method of claim 5, wherein the method further comprises:

and calculating the network throughput between the corresponding working nodes according to the data volume of the data fragments and the receiving time of the second forenotice message and the local data message.

15. The method of claim 14, wherein the method further comprises:

16. The data processing device is characterized by being applied to a scheduling node in a distributed cluster system of a computing power network and comprising a throughput information sensing module, a strategy making module and a strategy issuing module;

17. A data processing device is characterized in that the device is applied to working nodes in a distributed cluster system of a computing power network and comprises a data segmentation module, a destination receiving node determining module and a data distribution module,

18. A computer device, comprising:

one or more processors;

a storage device having one or more programs stored thereon;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data processing method of any of claims 1-15.

19. A computer readable medium having stored thereon a computer program, wherein the program when executed implements the data processing method according to any of claims 1-15.