CN114465941B - Cluster computing flow simulation method, system and device based on packet receiving and transmitting cooperation - Google Patents
Cluster computing flow simulation method, system and device based on packet receiving and transmitting cooperation Download PDFInfo
- Publication number
- CN114465941B CN114465941B CN202210383737.0A CN202210383737A CN114465941B CN 114465941 B CN114465941 B CN 114465941B CN 202210383737 A CN202210383737 A CN 202210383737A CN 114465941 B CN114465941 B CN 114465941B
- Authority
- CN
- China
- Prior art keywords
- flow
- packet
- message
- task
- simulation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
Abstract
The invention discloses a cluster computing flow simulation method, a system and a device based on packet receiving and sending cooperation. The invention adopts the directed acyclic graph to model the calculation process and the communication process of the cluster calculation, the data characteristics of the calculation stage and the communication stage can be defined by a user according to the requirement, and simultaneously, the system supports the batch generation of experimental samples under the constraint condition set by the user, including the generation of the directed acyclic graph and the communication characteristics thereof. In addition, the invention defines a flow transmission protocol based on a cooperative mechanism of a packet sender and a packet receiver, can realize dynamic simulation task scheduling besides generating flow data according with data characteristics, and more truly simulates the transmission process of cluster calculation flow in a network while ensuring verification efficiency.
Description
Technical Field
The invention belongs to the technical field of computer network flow simulation, and particularly relates to a cluster computing flow simulation method, system and device based on packet sending and receiving cooperation.
Background
The traffic scheduling technology of the data center network is receiving more and more attention at present, and especially with the development of artificial intelligence and big data intelligence technology, the amount of data transmission in the data center network is larger and larger, and the transmission speed and timeliness required by application are higher and higher, even under the condition that network hardware equipment is continuously updated, the transmission speed of data in the network still becomes one of bottlenecks which restrict large-scale data application.
The optimization of the network transmission capability is mainly started from two aspects, one is that the research directions such as intra-network calculation and the like transmit data by aggregation so as to achieve the purpose of reducing the data volume. Secondly, the flow with the dependency relationship is forwarded according to the sequence through a flow scheduling strategy, so as to achieve global optimization.
Both of the above-mentioned research techniques require flow data as well as a simulation environment to support the technical research and to get effect feedback. If communication codes are split from the existing cluster computing framework for modification, a great deal of time and energy are required for adding feature labels required by research, and even for project frameworks which are not open to source, the scheme of modifying the source codes is not feasible. The flow distinguishing precision and timeliness achieved by adopting a clustering mode are difficult to guarantee.
Disclosure of Invention
The invention aims to provide a cluster computing flow simulation method, a system and a device based on packet receiving and sending cooperation aiming at the defects in the prior art. The invention fills the technical blank in the field of flow scheduling simulation with dependency relationship, dynamically updates the global dependency relationship through the cooperation of the packet sender and the packet receiver, and more truly and timely starts different flow simulation task stages. The statistics of the packet receiving stage can be used as the effect feedback of the scheduling strategy to realize the closed loop of the experiment. The SDN switching equipment can support a more flexible forwarding strategy.
The invention is realized by the following technical scheme: a first aspect of an embodiment of the present invention provides a cluster computing traffic simulation method based on packet forwarding and receiving coordination, where the method includes:
(1) constructing a simulation task by self-defining or constraint conditions based on the directed acyclic graph;
(2) the packet sender detects the integrity of task description data, selects a simulation task from a simulation task library and starts the simulation task, transmits a preset scheduling strategy to a software defined network component, and then selects a communication stage task without pre-dependency to construct transmission simulation of flow; meanwhile, feedback information from the packet receiver assembly is received through the monitoring port;
(3) the software defines a network receiving and scheduling strategy, analyzes the message sent by the packet sender component, and then recombines and forwards the message to realize the network transmission of the message;
(4) the packet receiving device receives the message forwarded by the software defined network and analyzes the message; and after receiving the flow transmission end message transmitted by the packet transmitter, the packet receiver transmits a flow end confirmation message to the monitoring port of the packet transmitter.
Further, the self-defining simulation task is specifically as follows: defining the dependency relationship and data characteristics between different calculation stages and communication stages, detecting whether a topological structure describing the dependency relationship has an illegal structure of an outlier and a loop, associating a flow template with each communication stage, outputting the topological structure of the simulation task, and obtaining the self-defined simulation task.
Further, the flow template is constructed by self-definition or based on constraint conditions; the process of constructing the flow template through the constraint conditions specifically comprises the following steps: and outputting a flow template meeting the constraint condition under the constraint condition comprising the flow size, the parallel flow quantity range, the parallel flow variance, the mean value of the size of the single message and the variance thereof.
Further, the simulation task is specifically constructed through constraint conditions; under the constraint conditions including the flow size, the range of the number of parallel flows, the variance of the parallel flows, the mean value and the variance of the size of a single message, the number of computing nodes and the number of communication stages, generating a task topological structure without outliers and loops based on the inverse process of topological sorting, generating a flow template according with the flow constraint conditions according to the number of the communication stages, and outputting the topological structure describing the tasks, so that the simulation task constructed based on the constraint conditions is obtained.
Further, the step (2) is specifically:
(2.1) checking whether simulation task description data including topological structure data, mapping relation and associated flow template data output by task construction are complete or not; if the flow template data is complete, transmitting a preset scheduling strategy to the software defined network;
(2.2) processing matrix data describing a topological structure in a simulation task to obtain a communication stage without pre-dependency at present for task scheduling, and obtaining flow characteristics; starting a simulation task of the communication stage according to the flow size, the flow parallelism and the variance thereof, and the average message size and the variance in the flow characteristics; generating the flow of the cooperative flow neutron flow based on the flow, the number of the packet sending threads and the parallel flow variance, and generating the size of each message based on the message mean value and the variance value by Gaussian distribution; starting threads corresponding to the number of the packet sending threads, and performing packet packaging; if all the packet sending threads are finished, sending a flow transmission representation message at the communication stage;
and (2.3) monitoring the state feedback port in real time, deleting the dependency represented by the communication stage to update the topological structure of the simulation task if a flow end confirmation message of the packet receiver is received, and repeating the step (2.2) until all elements in the matrix data of the topological structure are zero and the simulation task is ended.
Further, the step (3) is specifically: after receiving the scheduling strategy, a control plane in the software defined network generates a flow table and issues the flow table to a data plane according to a strict priority multi-queue mechanism; after receiving the message sent by the packet sender, the data plane analyzes the message to obtain the task number and the flow number of the message so as to match with the forwarding strategy in a strict priority mode, and the message is retransmitted to a corresponding port after being recombined, so that the network transmission of the message is realized.
Further, the step (4) comprises the following substeps:
(4.1) receiving a message forwarded from a software defined network by monitoring a port in a specified range in real time in a packet receiver, and analyzing the content of the message to obtain information including a task number and flow rate coding;
(4.2) the packet receiver counts the transmission completion time of each flow and the total task completion time according to the task number-flow number as a statistic dimension;
and (4.3) after receiving the flow transmission end message transmitted by the packet transmitter, the packet receiver transmits a flow transmission end confirmation message containing the task number, the flow end identifier and the task end identifier to a packet transmitter feedback port.
A second aspect of the embodiments of the present invention provides a cluster computing traffic simulation system based on packet sending and receiving coordination, including a task construction component, a packet sender component, a packet receiver component, a software defined network component, an inter-component communication module, a packet encapsulation module, and a packet unpacking module;
the task construction component is used for constructing a simulation task through self definition or constraint conditions;
the packet sending component is used for detecting the integrity of task description data, selecting a simulation task from a simulation task library, starting the simulation task, transmitting a preset scheduling strategy to the software defined network component, and then selecting a communication stage task without pre-dependency to construct transmission simulation of flow;
the packet receiving component is used for receiving the message forwarded by the software defined network;
the software defined network component is used for receiving the scheduling strategy, analyzing the message sent by the packet sender component, and then recombining and forwarding the message to realize the network transmission of the message;
the inter-component communication module is used for receiving feedback information from the packet receiver component through the monitoring port; transmitting a preset scheduling strategy to a software defined network; after receiving the flow transmission end message transmitted by the packet transmitter, the packet receiver transmits a flow end confirmation message to a monitoring port of the packet transmitter;
the message packaging module is used for packaging the message;
the message unpacking module is used for unpacking the message.
A third aspect of the embodiments of the present invention provides a cluster computing traffic simulation apparatus based on packet transceiving coordination, including one or more processors, configured to implement the above cluster computing traffic simulation method based on packet transceiving coordination.
A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, on which a program is stored, where the program, when executed by a processor, is configured to implement the method for computing traffic simulation based on a cluster under cooperation of transceiving packets.
The beneficial effects of the invention are: (1) the packet sending component monitors the state feedback of the packet receiving device, and the actual packet receiving conditions of the communication link and the receiving end are brought into the scheduling process of the packet sending task by dynamically updating the global topological structure of the task and acquiring the communication stage without the dependency relationship in real time. The mode of the cooperative work of the receiving and sending packet ends enables the simulation environment to be closer to the real environment. (2) The user can flexibly make and test the flow scheduling strategy, and the SDN switching equipment configures the corresponding strategy, so that the statistical result of the packet receiving end can effectively measure the effect of the scheduling strategy, and the method is helpful for helping a researcher to adjust the scheduling strategy in real time and verify the effectiveness of the adjustment. (3) The system supports automatic generation of a large batch of simulation tasks, the generated result strictly conforms to the dependency relationship of different computing stages of cluster computing, and the system has strong support capability on model training and reasoning verification of a scheduling algorithm. (4) The system is used for cluster calculation flow scheduling research with dependency and upper application decoupling, researchers do not need to invest in the improvement of upper application services, and the scheduling effect of a specific experimental case verification scheduling strategy in a specific environment can be designed independently.
Drawings
FIG. 1 is a block diagram of a cluster computation flow simulation system based on the cooperation of a sending packet and a receiving packet;
FIG. 2 is a workflow block diagram of a task building component;
FIG. 3 is a block diagram of the workflow of the wrapper component;
figure 4 is a workflow block diagram of SDN components;
FIG. 5 is a block diagram of the workflow of the receive wrapper component;
FIG. 6 is a flow diagram for automatically building a simulation task based on constraints;
FIG. 7 is a flow diagram of generating a directed acyclic graph;
FIG. 8 is a protocol message structure;
fig. 9 is a schematic structural diagram of a cluster computing traffic simulation apparatus based on packet coordination between transmission and reception according to the present invention.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
The following describes a cluster computation flow simulation system and method based on packet coordination in detail with reference to the accompanying drawings. The features of the following examples and embodiments may be combined with each other without conflict.
As shown in fig. 1, the present invention provides a cluster computation flow simulation system based on packet sending and receiving coordination, which includes a task building component, a packet sender component, a packet receiver component, and an SDN component.
The task construction component is composed of a task self-defining module or a batch task generating module based on constraint conditions.
The task self-defining module defines the dependency relationship and the data characteristics between different calculation stages and communication stages in a visual mode by a user, firstly checks whether flow template data meeting simulation parameters designed by the user exist in a flow template base or not, generates a flow template if the flow template data do not exist, creates the flow template, and stores the flow template in the flow template base. Detecting whether a topological structure describing the dependency relationship in the user-defined task has an illegal structure of an outlier and a loop, and outputting a mapping relationship between a communication stage and a flow template library; and mapping the flow template base to a communication stage, and outputting a topological structure of the simulation task to obtain the self-defined simulation task. And stores the simulation task in a simulation task library, i.e., a directed acyclic graph library (DAG library). The DAG library completely describes the topological relation and the communication phase mapping relation required by the whole simulation task.
As shown in fig. 6 and 7, the flow template specifically includes: the flow template automatically generates the flow template through self-definition or under a first constraint condition, wherein the first constraint condition comprises flow size (minSize-maxSize), parallel flow quantity range (minWidth-maxWidth), parallel flow variance (flow _ var), and mean value (pkt _ size) and variance (pkt _ var) of single message size, and the flow template meeting the constraint condition is output and stored in a flow template library.
The automatic generation of the simulation task according to the set constraint conditions specifically comprises the following steps: automatically generating a simulation task under a second constraint condition, wherein the second constraint condition adds the number of computing nodes (node _ num) and the number range of communication stages (node _ num-1 to edge _ num) except for the first constraint condition automatically generated by the flow template, generating a task topological structure without outliers and loops based on the inverse process of topological sorting, storing the topological structure into a directed acyclic graph library to obtain an index identifier dag _ id of the topological structure, generating flow templates meeting flow constraint conditions according to the number of the communication stages, storing the flow templates into a flow template library to obtain index identifications flow _ id of the flow templates, outputting topological structures describing tasks and mapping relations between the communication stages and the flow templates, that is, the topology index dag _ id, the edge name edge _ name in the topology and the flow template index coflow _ id are stored in the mapping table t _ edge.
The packet sending component is used for detecting the integrity of task description data, selecting a simulation task from a simulation task library, starting the simulation task, transmitting a preset scheduling strategy to the software defined network component, and then selecting a communication stage task without pre-dependency to construct and transmit simulation of flow. The system comprises a task scheduling module.
The task scheduling module acquires a current calculation stage without a dependent item through the dependency relationship between different calculation stages and communication stages described by a topological structure, and outputs a communication stage list with the calculation stage as a starting point; in addition, when a communication stage is sensed to finish a transmission task at a packet receiver, the updating operation of the topological structure is finished.
The packet sending module realizes the construction and sending functions of the flow by calling the packet module according to the flow characteristics described by the flow template mapped in the communication stage. The task scheduling component of the wrapper-issuing machine updates the topology in order to direct the wrapper-issuing machine to perform the wrapping of a specific flow.
The packet receiving component is used for receiving the message forwarded by the software defined network and comprises a packet receiving module and a counting module. The packet receiving module receives a message through a monitoring appointed port; the statistical module acquires the task number, the flow number and the ending identification of the message by calling the message unpacking module, and performs statistics and storage on the number and time of message transmission in an iterative updating mode;
the SDN component is composed of a control plane and a data plane. The control plane receives a forwarding strategy transmitted by a task scheduling component of the packet sender, determines the priority of message forwarding according to the task number and the flow number carried in the message as unique identifiers, and generates a flow table according to a strict priority forwarding strategy and issues the flow table to the data plane. And the data plane completes the actual forwarding work of the message in the simulation system according to the flow table issued by the control plane.
The inter-component communication module is used for monitoring the packet receiving state of the packet receiver component, and a confirmation message of the packet receiver for completing message reception is the only trigger factor for updating the topological structure; and the second is a control plane used for sending the scheduling strategy set by the user to the SDN component. After the packet receiver receives the flow transmission end identifier, a message transmission task end confirmation message is generated by calling a message packaging module, and the task completion state is fed back to the packet receiver.
The message packaging module packages the task description into a message format specified by a system flow protocol, wherein the message format comprises a task number, a flow number, a sub-flow label, a message sequence number, a flow end identifier, a task end identifier and an additional load.
The message unpacking module is the reverse process of message packing and is used for analyzing the communication messages among the components.
The invention provides a cluster computing flow simulation method based on packet receiving and sending cooperation, which is applied to the system and specifically comprises the following steps:
(1) as shown in FIG. 2, the simulation tasks are selected by customization or by defining a topology system using a task building component.
The simulation task construction element comprises a topological structure for describing the dependency relationship between the calculation stage and the communication stage and a flow template for describing flow characteristics, the topological structure is independently constructed on the bottom layer, and the multiplexing of the flow template is realized in a mapping association mode. The task construction component supports two construction modes: firstly, selecting the number of points representing a calculation stage and the front-back dependence between the two points by using a self-defined construction mode, and selecting a flow template associated with mapping for each communication stage after detecting the legality of a topological structure; and secondly, automatically completing task construction by the system by defining the number of points and the upper limit of the number of edges in the topological structure and the flow template attribute of the associated mapping in the communication stage.
The user-defined construction mode specifically comprises the following steps: firstly, whether flow template data which accords with simulation parameters designed by a user exist in a flow template library or not is checked, if not, a flow template is generated, the flow template is created, and the flow template is stored in the flow template library. The flow template data comprises a flow name, a flow size, a parallel flow quantity range, a parallel flow size variance, and a mean value and variance of a single message size. The user then determines the dependencies between the different computation phases and communication phases, which can be represented by directed acyclic graphs, by defining the computation phases and the communication phases connecting the different computation phases. In order to ensure that the tasks cannot be triggered or interdependent, the directed acyclic graph generated by the user in a self-defined way needs to be subjected to outlier and loop detection. If the communication stage passes the detection, a flow template is further selected from the flow template library for each communication stage to carry out mapping association, a topological structure of the simulation task is obtained, and the user-defined simulation task is obtained.
The generation of the flow template specifically comprises the following steps: the flow template automatically generates the flow template by self-defining or under a first constraint condition, wherein the first constraint condition comprises the flow size, the range of the quantity of the parallel flows, the variance of the parallel flows, the mean value and the variance of the size of a single message, and the flow template which meets the constraint condition is output and stored in a flow template library.
The constraint condition-based batch task automatic construction method specifically comprises the following steps: the number of computation stages N, the upper limit of the number of communication stages L and the traffic template data are defined. For a directed acyclic graph, L ranges from N-1 and
in between. The user-defined parameters need to be validated legally. After the parameter validity is checked, a directed acyclic graph without outliers and loops is generated according to the reverse process of the topological sorting, as shown in fig. 6 and 7. And after the construction of the topological structure is completed, automatically generating a flow template according to the quantity L of the communication phases and the constraint conditions of the flow template, mapping and associating to obtain the topological structure of the simulation tasks, namely obtaining the simulation tasks generated in batch, and using a simulation task library generated in batch, namely a directed acyclic graph library (DAG library).
(2) As shown in fig. 3, firstly, a packet sender detects the integrity of task description data, selects a simulation task from a simulation task library and starts the simulation task, transmits a preset scheduling policy to an SDN component, and then selects a communication stage task without pre-dependency to perform traffic construction and transmission simulation. And simultaneously opening the monitoring port to receive the feedback information from the packet receiver assembly. The method specifically comprises the following substeps:
and (2.1) before starting the traffic simulation task, the packet sending component checks whether the task description data output by the task construction is complete, and specifically checks whether the topological structure data, the mapping relation and the associated traffic template data are complete. And if the data of the flow template is complete, transmitting a preset scheduling strategy to a Software Defined Network (SDN) component.
And (2.2) processing matrix data describing a topological structure in the simulation task through a task scheduling module to obtain a current communication stage without pre-dependency, obtaining flow characteristics through a mapping relation, and starting the simulation task in the communication stage according to the flow size, the flow parallelism and the variance thereof, and the average message size and the variance in the flow characteristics. The traffic parallelism represents the number of packetization threads. And generating the flow of the collaborative flow neutron flow based on the flow, the number of the packet sending threads and the parallel flow variance, generating the size of each message based on the mean value and the variance value of the message, and generating the flow and the message sent by each thread according to the mean value and the variance by Gaussian distribution. And starting threads corresponding to the number of the packet sending threads according to the number of the parallel streams, and performing packet by using a packet packaging module, wherein the protocol message structure is shown in fig. 8. The cooperative flow header comprises a task end identifier (4 bytes), a flow end identifier (4 bytes), a sub-flow identifier (4 bytes), a message sequence number (4 bytes), a task identifier (4 bytes) and a flow identifier (4 bytes). And if all the packet sending threads are finished, sending the traffic transmission representation message at the communication stage through the message packet sending module. The end-of-flow flag is set to 1.
And (2.3) monitoring the state feedback port in real time through the inter-component communication module, deleting the dependency represented by the communication stage to update the topological structure of the simulation task if a flow end confirmation message of the packet receiver is received, and repeating the step (2.2) until all elements in the matrix data of the topological structure are zero and the simulation task is ended.
(3) As shown in fig. 4, after receiving a scheduling policy sent by a packet sender component, a controller plane of the SDN generates a flow table according to a strict priority multi-queue mechanism and sends the flow table to a data plane. After receiving a message sent by a packet sender, a data plane of the SDN analyzes the message to obtain a task number and a flow number of the message, so that a forwarding strategy under a strict priority mode is matched, the message is recombined and then forwarded to a corresponding port of the packet receiver, and network transmission of the message is realized.
(4) As shown in fig. 5, after receiving the packet forwarded by the SDN component, the packet receiver performs packet analysis according to a protocol format. And counting the completion time, the packet receiving quantity and the whole task completion time at the flow number level of each task. And after the packet receiver receives the flow transmission end message transmitted by the packet transmitter, replying a confirmation message of flow transmission end.
And after receiving the flow end confirmation message transmitted by the packet receiver, the packet transmitter updates the topological structure, acquires the latest communication stage in the independent state and starts a flow simulation task of the packet transmitter. And when no communication stage in the topological structure is in the running state or the to-be-run state, ending the task.
The method specifically comprises the following substeps:
and (4.1) monitoring a port in a specified range in real time through a packet receiving module in the packet receiver, receiving a message forwarded by the SDN component, and analyzing the content of the message according to a protocol format by using a message analysis module. And analyzing and acquiring information such as task numbers, flow numbers and the like.
And (4.2) the packet receiver counts the transmission completion time of each flow and the total task completion time according to the task number-flow number as a statistic dimension, and persists the transmission completion time and the total task completion time into a database after the simulation task is finished to form a closed loop of the task.
(4.3) the packet receiving device receives a packet flow transmission end identification message, namely the message with the flow end identification set to be 1 in the protocol, and sends the flow transmission end confirmation message to the packet sending device feedback port, wherein the flow transmission end confirmation message comprises a task number, a flow end identification and a task end identification.
(5) And after receiving the flow end confirmation message transmitted by the packet receiver, the packet transmitter updates the topological structure, acquires the latest communication stage in the independent state and starts the flow simulation task. And when no communication stage in the topological structure is in the running state or the state to be run, ending the task.
Corresponding to the embodiment of the cluster calculation flow simulation method based on the cooperation of the receiving and sending packets, the invention also provides an embodiment of a cluster calculation flow simulation device based on the cooperation of the receiving and sending packets.
Referring to fig. 9, an embodiment of the present invention provides a cluster computing traffic simulation apparatus based on packet coordination for receiving and sending, which includes one or more processors, and is configured to implement the cluster computing traffic simulation method based on packet coordination for receiving and sending in the foregoing embodiment.
The embodiment of the cluster computing flow simulation device based on the cooperation of the sending and receiving packets can be applied to any equipment with data processing capability, and the any equipment with data processing capability can be equipment or devices such as computers. The device embodiments may be implemented by software, or by hardware, or by a combination of hardware and software. The software implementation is taken as an example, and as a logical device, the device is formed by reading corresponding computer program instructions in the nonvolatile memory into the memory for running through the processor of any device with data processing capability. In terms of hardware, as shown in fig. 9, a hardware structure diagram of any device with data processing capability where the cluster computing traffic simulation apparatus based on cooperation of the transceiving packets according to the present invention is located is shown, in addition to the processor, the memory, the network interface, and the nonvolatile memory shown in fig. 9, in the embodiment, any device with data processing capability where the apparatus is located may further include other hardware generally according to an actual function of the any device with data processing capability, which is not described again.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiment, since it basically corresponds to the method embodiment, reference may be made to the partial description of the method embodiment for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the invention. One of ordinary skill in the art can understand and implement without inventive effort.
The embodiment of the present invention further provides a computer-readable storage medium, on which a program is stored, where the program, when executed by a processor, implements the method for simulating cluster computing traffic based on packet coordination in the foregoing embodiments.
The computer readable storage medium may be an internal storage unit, such as a hard disk or a memory, of any data processing device described in any previous embodiment. The computer readable storage medium may also be any external storage device of a device with data processing capabilities, such as a plug-in hard disk, a Smart Media Card (SMC), an SD Card, a Flash memory Card (Flash Card), etc. provided on the device. Further, the computer readable storage medium may include both an internal storage unit and an external storage device of any data processing capable device. The computer-readable storage medium is used for storing the computer program and other programs and data required by the arbitrary data processing capable device, and may also be used for temporarily storing data that has been output or is to be output.
In summary, the method monitors the status feedback of the packet receiver through the packet transmitter, dynamically updates the global topology of the task, and obtains the communication stage without dependency relationship in real time, so as to incorporate the actual packet receiving conditions of the communication link and the receiving end into the scheduling process of the packet transmitting task. The mode of the cooperative work of the receiving and sending packet ends enables the simulation environment to be closer to the real environment. In the system, the user can flexibly make and test the flow scheduling strategy, and the SDN switching equipment is configured with the corresponding strategy, so that the statistical result of the packet receiving end can effectively measure the effect of the scheduling strategy, and the system is helpful for helping a researcher to adjust the scheduling strategy in real time and verify the effectiveness of the adjustment. The system supports automatic generation of a large batch of simulation tasks, the generated result strictly conforms to the dependency relationship of different computing stages of cluster computing, and the system has strong support capability on model training and reasoning verification of a scheduling algorithm. The system is used for decoupling cluster calculation flow scheduling research and upper application with dependency relationship, a user does not need to invest in the improvement of upper application service, and the scheduling effect of a specific experimental case verification scheduling strategy in a specific environment can be independently designed.
In addition, the directed acyclic graph models the calculation process and the communication process of cluster calculation, data characteristics of a calculation stage and a communication stage can be defined by a user according to needs, and meanwhile, the system supports batch generation of experimental samples under set constraint conditions, including generation of the directed acyclic graph and the communication characteristics thereof. In addition, the invention defines a flow transmission protocol based on a cooperative mechanism of a packet sender and a packet receiver, can realize dynamic simulation task scheduling besides generating flow data according with data characteristics, and more truly simulates the transmission process of cluster calculation flow in a network while ensuring verification efficiency.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (9)
1. A cluster computing flow simulation method based on packet receiving and sending cooperation is characterized by comprising the following steps:
(1) constructing a simulation task by self-defining or constraint conditions based on the directed acyclic graph;
(2) the packet sender detects the integrity of task description data, selects a simulation task from a simulation task library and starts the simulation task, transmits a preset scheduling strategy to a software defined network component, and then selects a communication stage task without pre-dependency to construct transmission simulation of flow; meanwhile, feedback information from the packet receiver assembly is received through the monitoring port, and if a flow end confirmation message of the packet receiver is received, the dependency represented by the communication stage is deleted to update the topological structure of the simulation task until the simulation task is ended;
(3) the software defines a network receiving and dispatching strategy, analyzes the message sent by the packet sender assembly, and then performs message recombination and forwarding to realize the network transmission of the message;
the step (3) is specifically as follows: after receiving the scheduling strategy, a control plane in the software defined network generates a flow table and sends the flow table to a data plane according to a strict priority multi-queue mechanism; after receiving a message sent by a packet sender, the data plane analyzes the message to obtain a task number and a flow number of the message so as to match a forwarding strategy under a strict priority mode, and the message is recombined and forwarded to a corresponding port to realize network transmission of the message;
(4) the packet receiving device receives the message forwarded by the software defined network and analyzes the message; and after receiving the flow transmission end message transmitted by the packet transmitter, the packet receiver transmits a flow end confirmation message to the monitoring port of the packet transmitter.
2. The cluster computing flow simulation method based on packet-sending coordination according to claim 1, wherein the simulation task is specifically constructed by self-definition as follows: defining the dependency relationship and data characteristics between different calculation stages and communication stages, detecting whether a topological structure describing the dependency relationship has an illegal structure of an outlier and a loop, associating a flow template with the communication stage, and outputting the topological structure of the simulation task, so that the customized simulation task is obtained.
3. The cluster computing flow simulation method based on the packet-sending collaboration as claimed in claim 2, wherein the flow template is constructed by self-definition or based on constraint conditions; the process of constructing the flow template through the constraint conditions specifically comprises the following steps: and outputting a flow template meeting the constraint condition under the constraint condition comprising the flow size, the parallel flow quantity range, the parallel flow variance, the mean value of the size of a single message and the variance of the mean value.
4. The cluster computing flow simulation method based on packet-sending coordination according to claim 1, wherein the construction of the simulation task through the constraint condition is specifically; under the constraint conditions including the flow size, the range of the quantity of the parallel flows, the variance of the parallel flows, the mean value and the variance of the size of a single message, the quantity of the computing nodes and the quantity of the communication stages, generating a task topological structure without outliers and loops based on the inverse process of topological sorting, generating a flow template according with the flow constraint conditions according to the quantity of the communication stages, and outputting the topological structure describing the task, so that the simulation task constructed based on the constraint conditions is obtained.
5. The method for cluster computational flow simulation based on packet-sending coordination according to claim 1, wherein said step (2) is specifically said step
(2.1) checking whether simulation task description data including topological structure data, mapping relation and associated flow template data output by task construction are complete or not; if the flow template data is complete, transmitting a preset scheduling strategy to a software defined network;
(2.2) processing matrix data describing a topological structure in a simulation task to obtain a communication stage without pre-dependency at present for task scheduling, and obtaining flow characteristics; starting a simulation task of the communication stage according to the flow size, the flow parallelism and the variance thereof, and the average message size and the variance in the flow characteristics; generating the flow of the cooperative flow neutron flow based on the flow, the number of the packet sending threads and the parallel flow variance, and generating the size of each message based on the average value and the variance value of the message by Gaussian distribution; starting threads corresponding to the number of the packet sending threads, and performing packet packaging; if all the packet sending threads are finished, sending a traffic transmission representation message in the communication stage;
and (2.3) monitoring the state feedback port in real time, deleting the dependency represented by the communication stage to update the topological structure of the simulation task if a flow end confirmation message of the packet receiver is received, and repeating the step (2.2) until all elements in the matrix data of the topological structure are zero and the simulation task is ended.
6. The method for cluster computation flow simulation based on packet-sending coordination according to claim 1, wherein the step (4) comprises the following substeps:
(4.1) receiving a message forwarded from a software defined network by monitoring a port in a specified range in real time in a packet receiver, and analyzing the content of the message to obtain information including a task number and flow rate coding;
(4.2) the packet receiver counts the transmission completion time of each flow and the total task completion time according to the task number-flow number as a statistic dimension;
and (4.3) after receiving the flow transmission end message transmitted by the packet transmitter, the packet receiver transmits a flow transmission end confirmation message containing the task number, the flow end identifier and the task end identifier to a packet transmitter feedback port.
7. A cluster computation flow simulation system based on packet receiving and sending coordination is characterized by comprising a task construction component, a packet sender component, a packet receiver component, a software defined network component, an inter-component communication module, a message packaging module and a message unpacking module;
the task construction component is used for constructing a simulation task through self-definition or constraint conditions;
the packet sending component is used for detecting the integrity of task description data, selecting a simulation task from a simulation task library, starting the simulation task, transmitting a preset scheduling strategy to the software defined network component, and then selecting a communication stage task without pre-dependency at present to construct transmission simulation of flow;
the packet receiving component is used for receiving the message forwarded by the software defined network;
the software defined network component is used for receiving the scheduling strategy, analyzing the message sent by the packet sender component, and then recombining and forwarding the message to realize the network transmission of the message;
the inter-component communication module is used for receiving feedback information from the packet receiver component through the monitoring port; transmitting a preset scheduling strategy to a software defined network; after receiving the flow transmission end message transmitted by the packet transmitter, the packet receiver transmits a flow end confirmation message to a monitoring port of the packet transmitter;
the message packaging module is used for packaging the message;
the message unpacking module is used for unpacking the message.
8. A cluster computation flow simulation device based on packet transmission and reception cooperation, which is characterized by comprising one or more processors and is used for realizing the cluster computation flow simulation method based on packet transmission and reception cooperation of any one of claims 2 to 6.
9. A computer-readable storage medium having a program stored thereon, where the program is to implement the method for computing traffic simulation based on a cluster for packet-based collaboration as claimed in any one of claims 2 to 6 when the program is executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210383737.0A CN114465941B (en) | 2022-04-13 | 2022-04-13 | Cluster computing flow simulation method, system and device based on packet receiving and transmitting cooperation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210383737.0A CN114465941B (en) | 2022-04-13 | 2022-04-13 | Cluster computing flow simulation method, system and device based on packet receiving and transmitting cooperation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114465941A CN114465941A (en) | 2022-05-10 |
| CN114465941B true CN114465941B (en) | 2022-07-15 |
Family
ID=81418474
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210383737.0A Active CN114465941B (en) | 2022-04-13 | 2022-04-13 | Cluster computing flow simulation method, system and device based on packet receiving and transmitting cooperation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114465941B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114900472B (en) * | 2022-07-12 | 2022-11-08 | 之江实验室 | Method and system for realizing cooperative flow scheduling by control surface facing to multiple tasks |
| CN114884893B (en) * | 2022-07-12 | 2022-10-25 | 之江实验室 | Forwarding and control definable cooperative traffic scheduling method and system |
| CN115951989B (en) * | 2023-03-15 | 2023-06-20 | 之江实验室 | Collaborative flow scheduling numerical simulation method and system based on strict priority |
| CN116996443B (en) * | 2023-09-25 | 2024-01-23 | 之江实验室 | Network collaborative traffic scheduling method and system combining GNN and SAC models |
| CN117221126B (en) * | 2023-11-09 | 2024-02-13 | 之江实验室 | Network collaboration flow-oriented route scheduling method and system |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113992590A (en) * | 2021-10-25 | 2022-01-28 | 天津职业技术师范大学(中国职业培训指导教师进修中心) | Link load balancing method based on software defined network |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080198742A1 (en) * | 2007-02-18 | 2008-08-21 | Gideon Kaempfer | Method and system for testing stateful network communications devices |
| WO2015121864A1 (en) * | 2014-02-16 | 2015-08-20 | B.G. Negev Technologies And Applications Ltd., At Ben-Gurion University | A system and method for integrating legacy flow-monitoring systems with sdn networks |
| US9871730B2 (en) * | 2014-08-22 | 2018-01-16 | Futurewei Technologies, Inc. | Network element configured to operate in an information centric network |
| US9769060B2 (en) * | 2015-07-20 | 2017-09-19 | Schweitzer Engineering Laboratories, Inc. | Simulating, visualizing, and searching traffic in a software defined network |
| CN109921947B (en) * | 2019-03-26 | 2022-02-11 | 东软集团股份有限公司 | Network flow simulation method, device, equipment and network equipment test system |
| US11469942B2 (en) * | 2019-08-15 | 2022-10-11 | At&T Intellectual Property I, L.P. | System and method for SDN orchestration validation |
| CN111835579B (en) * | 2020-06-04 | 2021-12-21 | 南瑞集团有限公司 | Method and system for testing effectiveness of network traffic scheduling simulation |
| CN113518012B (en) * | 2021-09-10 | 2021-12-10 | 之江实验室 | Distributed cooperative flow simulation environment construction method and system |
-
2022
- 2022-04-13 CN CN202210383737.0A patent/CN114465941B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113992590A (en) * | 2021-10-25 | 2022-01-28 | 天津职业技术师范大学(中国职业培训指导教师进修中心) | Link load balancing method based on software defined network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114465941A (en) | 2022-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114465941B (en) | Cluster computing flow simulation method, system and device based on packet receiving and transmitting cooperation | |
| US9450895B2 (en) | Context-aware dynamic policy selection for messaging behavior | |
| CN112187632B (en) | Industrial equipment integration model and construction method, edge gateway device and industrial equipment integration system | |
| Agha et al. | PMaude: Rewrite-based specification language for probabilistic object systems | |
| US9602380B2 (en) | Context-aware dynamic policy selection for load balancing behavior | |
| WO2017054531A1 (en) | Yang model-based compiling method, and corresponding interface, component, and system | |
| Wen et al. | A RESTful framework for Internet of things based on software defined network in modern manufacturing | |
| Amoretti et al. | DEUS: a discrete event universal simulator | |
| CN106528169B (en) | A kind of Web system exploitation reusable method based on AnGo Dynamic Evolution Model | |
| CN115150274B (en) | Unified configuration method, system and storage medium for time-sensitive network equipment | |
| CN113518012B (en) | Distributed cooperative flow simulation environment construction method and system | |
| WO2021098824A1 (en) | Network slice creation method, basic network controller, system, and storage medium | |
| US11868361B2 (en) | Data distribution process configuration method and apparatus, electronic device and storage medium | |
| Chen et al. | Introduction to OPNET network simulation | |
| CN114372084A (en) | Real-time processing system for sensing stream data | |
| EP3637690A1 (en) | Service configuration-driven flow table method and device | |
| Xiao et al. | A hybrid task crash recovery solution for edge computing in IoT-based manufacturing | |
| Iglesias-Urkia et al. | IEC 61850 meets CoAP: towards the integration of smart grids and IoT standards | |
| US20220350685A1 (en) | Data distribution system and data distribution method | |
| CN109150583A (en) | A kind of management method and device of northbound interface | |
| Ölveczky et al. | Specification and analysis of the AER/NCA active network protocol suite in Real-Time Maude | |
| Fummi et al. | Modeling of communication infrastructure for design-space exploration | |
| CN115774573A (en) | Application integration method and device, electronic equipment and storage medium | |
| CN112235367B (en) | Method, system, terminal and storage medium for subscribing entity behavior relation message | |
| CN108989406A (en) | Software shelf realization method and system based on micro services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |