CN115774580A - Cloud-side data transmission control system, method and storage medium - Google Patents
Cloud-side data transmission control system, method and storage medium Download PDFInfo
- Publication number
- CN115774580A CN115774580A CN202111054930.1A CN202111054930A CN115774580A CN 115774580 A CN115774580 A CN 115774580A CN 202111054930 A CN202111054930 A CN 202111054930A CN 115774580 A CN115774580 A CN 115774580A
- Authority
- CN
- China
- Prior art keywords
- data
- edge
- module
- cloud
- data channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
Landscapes
- Computer And Data Communications (AREA)
Abstract
The disclosure relates to the technical field of artificial intelligence, and discloses a cloud-side data transmission control system and method, a storage medium and an electronic device. The cloud layer comprises a cloud data channel module connected with a cloud application and used for acting the cloud layer to perform data transmission with the edge layer; the edge layer comprises a data channel interface module, an edge data channel module and a service discovery module; the data channel interface module is used for establishing connection between the edge application and the edge data channel module; the edge data channel module is used for acting an edge layer and a cloud layer or other edge layers to perform data transmission and processing transmission data; the service discovery module is used for routing when data transmission is carried out between the cloud end layer and each module in the edge layer, monitoring the states of nodes where the cloud end data channel module and the edge data channel module are located, and regulating and controlling transmission data according to the states. The method and the device not only realize the transmission regulation and control of the cloud-edge data, but also realize the transmission regulation and control of the data between the edge ends.
Description
Technical Field
The present disclosure relates to the field of artificial intelligence technologies, and in particular, to a cloud-side data transmission control system, a cloud-side data transmission control method, and a storage medium.
Background
With the development of the intelligent edge computing Technology field, data transmission from cloud end to edge, from edge to cloud end, or even between edge and edge is almost a common requirement for all edge business applications, while with the diversification and complication of 5G (5 th Generation Mobile Communication Technology, fifth Generation Mobile Communication Technology) era intelligent identification scale deployment and AI (Artificial Intelligence) applications, the number of network edge devices and the rapid growth of generated data, a centralized cloud computing (i.e. gathering, processing and storing data at the cloud end) solution will face various challenges, and therefore, a new Generation of edge-cloud-coordinated universal computing solution needs to move forward the computing power and process the business logic of user applications at the network edge side closer to the data source.
In the related art, a system architecture includes a cloud center layer, an Edge layer, and a device layer, and manages Edge devices and tasks by deploying a central scheduling application in the cloud center portion of the system architecture, and an Edge end of a KubeEdge (an open source system) is installed in the Edge layer to manage various loads with a cloud end. Then, in the related art, data transmission cannot be performed between edges, and scheduling and control of data traffic between cloud edges are also lacking.
It is to be noted that the information invented in the background section above is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The present disclosure aims to provide a cloud-side data transmission control system and method, a computer storage medium, and an electronic device, so as to overcome the problems that scheduling and control of data traffic cannot be implemented, and data cannot be transmitted at edges and edges in cloud-side data transmission, at least to a certain extent. In order to solve the above problems, the present disclosure adopts the following technical solutions.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to one aspect of the disclosure, a cloud-side data transmission control system is provided, which includes a cloud end layer and an edge layer, wherein the cloud end layer includes a cloud end data channel module connected with a cloud end application, and is used for acting the cloud end layer and the edge layer to perform data transmission; the edge layer comprises a data channel interface module, an edge data channel module and a service discovery module; the data channel interface module is used for establishing connection between the edge application and the edge data channel module; the edge data channel module is used for acting the edge layer and the cloud end layer or other edge layers to perform data transmission and process transmission data; the service discovery module is used for routing when data transmission is carried out between the cloud end layer and each module in the edge layer, monitoring the states of nodes where the cloud end data channel module and the edge data channel module are located, and regulating and controlling the transmission data according to the states.
In an exemplary embodiment of the present disclosure, the data channel interface module includes: the address acquisition unit is used for sending a service request to the service discovery module and receiving the address of the edge data channel module returned by the service discovery module based on the service request; and the connection establishing unit is used for sending a connection request to the edge data channel module through the address so as to establish the connection between the edge application and the edge data channel module.
In an exemplary embodiment of the present disclosure, the data channel interface module further includes: the error detection unit is used for detecting the state of the node where the edge data channel module is located; the cache unit is used for caching the data to be transmitted when the state is the offline state; and the connection establishing unit is used for establishing connection with other edge data channel modules when the state is the disconnection state.
In an exemplary embodiment of the present disclosure, the system further includes a plug-in management module, which is respectively connected to the edge data channel module and the cloud data channel module; the edge data channel module includes: a data receiving unit, configured to receive a data packet sent by the edge application via the data channel interface module; the data packet analyzing unit is used for analyzing the data packet to obtain analyzed data; and the data dispatching unit is used for dispatching the analysis data to the plug-in management module.
In an exemplary embodiment of the present disclosure, the plug-in management module includes a plurality of plug-in sub-modules, which form a directed acyclic structure in a parallel or serial manner; the plurality of daughter card modules includes: the first sub plug-in module is used for receiving the analysis data distributed by the data distribution unit; the second sub-plug-in module is used for processing the analysis data and packaging the processed analysis data to obtain packaged data; and the third sub-plug-in module is used for uploading the packet data to the cloud data channel module.
In an exemplary embodiment of the present disclosure, the second daughter card module includes: the data aggregation unit is used for carrying out aggregation processing on the analysis data; the flow statistic unit is used for carrying out statistics on transmission flow according to the analysis data; a transmission scheduling unit, configured to schedule transmission of the analysis data according to a convergence processing result and/or a transmission traffic statistical result; and the information leakage detection unit is used for detecting the privacy information in the analysis data so as to regulate and control the transmission of the analysis data according to the detection result.
In an exemplary embodiment of the present disclosure, the service discovery module includes: the health state detection unit is used for detecting the health states of nodes where the cloud data channel module and the edge data channel module are located; the cluster construction unit is used for forming a node cluster by nodes with the same function type according to the function type of the node where the cloud data channel module or the edge data dynamic module is located; and the transmission data regulation and control unit is used for carrying out transmission regulation and control on the transmission data according to the health state and the formed node cluster.
In an exemplary embodiment of the present disclosure, the cloud data channel module is deployed in a cluster manner, and the cloud data channel module uses a message queue manner as a transmission data bearer to provide acquisition and transmission of transmission data for the cloud application.
According to an aspect of the present disclosure, there is provided a cloud-side data transmission control method applied to an edge data channel module in the cloud-side data transmission control system described in any one of the above, the method including: acquiring and processing transmission data sent by the edge application through a data channel interface; sending the processed transmission data to a cloud data channel module of a cloud layer so as to send the processed transmission data to a cloud application through the cloud data channel module; or sending the processed transmission data to other edge data channel modules of other edge layers, so as to send the processed transmission data to corresponding other edge applications via the other edge data channel modules; acquiring and processing transmission data sent by the cloud application through the cloud data channel module, and sending the processed transmission data to the edge application through the data channel interface module; in the transmission data sending process, routing among the modules is carried out through a service discovery module, the states of nodes where the cloud data channel module and the other edge data channel modules are located are monitored through the service discovery module, and the transmission data are regulated and controlled according to the states.
According to an aspect of the present disclosure, there is provided a computer storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the cloud-side data transmission control method described above.
According to an aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the above-mentioned cloud-edge data transmission control method via execution of the executable instructions.
The cloud-side data transmission control system in the exemplary embodiment of the disclosure comprises a cloud end layer and an edge layer, wherein the cloud end layer comprises a cloud end data channel module connected with a cloud end application and used for acting the cloud end layer to perform data transmission with the edge layer; the edge layer comprises a data channel interface module, an edge data channel module and a service discovery module, wherein the data channel interface module is used for establishing connection between the edge application and the edge data channel module; the edge data channel module is used for acting an edge layer and performing data transmission and processing transmission data with a cloud layer or other edge layers; the service discovery module is used for monitoring the states of the nodes where the cloud data channel module and the edge data channel module are located, and regulating and controlling the transmission data according to the states. On one hand, the data channel interface modules are respectively arranged on the cloud layer and each edge layer and are used as communication agents of the cloud layer or the edge layer, so that bidirectional duplex data transmission between cloud edges and between different edge local area networks is realized; on the other hand, the transmission data processed at the edge layer can be sent to the cloud end layer through the processing of the edge data channel module on the transmission data, so that the large-scale transmission of the data can be realized; on the other hand, in the data transmission process between the cloud edge or the edge layer and the edge layer, the node state discovery function of the service discovery module is used for detecting and transmitting data through the healthy nodes, so that the regulation and control of transmitted data are realized, the efficient utilization of the cloud edge bandwidth is facilitated, and the load balance is realized.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description, which proceeds with reference to the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:
fig. 1 schematically illustrates a structural diagram of a cloud-edge data transmission control system according to an exemplary embodiment of the present disclosure;
FIG. 2 schematically illustrates a structural schematic of a data lane interface module according to an exemplary embodiment of the present disclosure;
FIG. 3 schematically illustrates a structural schematic of a data lane interface module according to an exemplary embodiment of the present disclosure;
fig. 4 schematically shows a structural diagram of a cloud-edge data transmission control system according to an exemplary embodiment;
FIG. 5 schematically illustrates a structural diagram of a plug-in management module according to an exemplary embodiment of the present disclosure;
FIG. 6 schematically illustrates a structural diagram of a plug-in management module of a directed acyclic structure according to an exemplary embodiment of the present disclosure;
fig. 7 schematically illustrates a structural diagram of a service discovery module according to an exemplary embodiment of the present disclosure;
fig. 8 schematically illustrates a flowchart of a cloud-edge data transmission control method according to an exemplary embodiment of the present disclosure;
FIG. 9 schematically illustrates an edge application establishing connection with an edge data channel module in accordance with an exemplary embodiment of the present disclosure;
FIG. 10 schematically illustrates an interaction flow diagram for initializing CEC-C and CEC-E according to an exemplary embodiment of the present disclosure;
FIG. 11 schematically illustrates a schematic diagram of a storage medium according to an exemplary embodiment of the present disclosure; and
fig. 12 schematically shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
Exemplary embodiments will now be described more fully with reference to the accompanying drawings. The exemplary embodiments, however, may be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their detailed description will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the embodiments of the disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, and so forth. In other instances, well-known structures, methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.
The block diagrams shown in the figures are functional entities only and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, or in one or more software-hardened modules, or in different networks and/or processor devices and/or microcontroller devices.
With the large-scale deployment of intelligent identification in the 5G era and the diversification and complication of AI applications, the number of network edge devices and generated data are rapidly increased, a centralized cloud computing (i.e., aggregating, processing and storing data at a cloud end) solution will face various challenges such as real-time performance, bandwidth, energy consumption and data privacy protection, and it is necessary to move the computing power forward to process the business logic of user applications at the network edge side closer to the data source. However, a new problem is brought about, namely, the services of the edge and the cloud end need to be further coordinated; secondly, the reliability of data transmission of the edge and the cloud needs to be further improved; thirdly, the problem of unnecessary transmission of data possibly exists; and fourthly, the development and deployment difficulty caused by the complex environment and the diversified ecology of the edge end.
In the related art, docker (an open source application container engine) is a container virtualization technology for developing an application, delivering the application, and running the application. The container virtualizes the operating system layer and eliminates the difference of different architectures through a set of standards of the container, so that the container can be deployed to any place regardless of the difference of the architectures. The Kubernetes framework is a suite of open source systems for automated deployment, expansion, and management of containerized applications, intended to provide a platform for automated deployment, expansion, and running of application containers across a host cluster, that supports a range of container tools, including Docker. The KubeEdge framework is a framework for managing and arranging edge nodes and equipment based on cloud containers provided by Kubernets, and the edge nodes and the equipment are incorporated into the Kubernets system to realize edge-cloud integrated node and equipment management. Kubeeedge defines abstraction of edge equipment, then monitors running state of the equipment, acquires equipment data and transmits control data to the equipment by a general mapper on an edge node, and provides abstraction and control capability of the equipment to a cloud end in a proxy mode.
Based on the above technology, in the related art, the cloud edge communication system framework includes a cloud center layer, an edge layer, and a device layer. The cloud center layer adopts Docker + Kubernetes as a system infrastructure, uses Kubernetes to establish an AI-Edge cloud center part, deploys a central scheduling application and manages Edge devices and tasks; the Edge layer installs a Docker environment, configures operating system parameters, then installs an Edge end of KubeEdge, receives and executes instructions issued by the Cloud part, manages various loads, and synchronizes the state and events of the Edge part load to the Cloud part; devices of the device layer may be connected to edge nodes of the edge layer.
Accordingly, the related art has the following drawbacks: firstly, when data is transmitted to the cloud end at the edge, the related technology can only transmit a small amount of formatted data, but cannot transmit a large amount of formatted data or unformatted data, namely large-scale transmission of the data cannot be realized; secondly, the related art does not implement the service discovery mechanism of the edge, so that the edge device cannot acquire the addresses of other devices at the edge and cannot perform data transmission between the edge devices; finally, because there are few channels between the edge and the cloud and the speed is slow, if repeated data is repeatedly transmitted between the edge and the cloud due to improper scheduling, and the channel efficiency is reduced due to fragmentation of data packets, the value of edge computation power is greatly reduced.
In view of the defects in the related art, in the present exemplary embodiment, a cloud-side data transmission control system is first provided. Referring to fig. 1, the cloud-edge data transmission control system 100 includes a cloud layer 110 and an edge layer 120, where the cloud layer 110 includes a cloud data channel module 1101 connected to a cloud application, and is configured to proxy the cloud layer 110 for data transmission with the edge layer 120; the edge layer 120 includes a data channel interface module 1201, an edge data channel module 1202, and a service discovery module 1203;
the data channel interface module 1201 is configured to establish a connection between the edge application and the edge data channel module 1202;
the edge data channel module 1202 is configured to proxy the edge layer 120 to perform data transmission with the cloud end layer 110 or other edge layers and process transmission data;
the service discovery module 1203 is configured to perform routing during data transmission between the cloud end layer and the edge layer, monitor states of nodes where the cloud end data channel module 1101 and the edge data channel module 1201 are located, and regulate and control transmission data according to the states.
According to the cloud-edge data transmission control system in the present exemplary embodiment, on one hand, data channel interface modules are respectively deployed in a cloud end layer and each edge layer to serve as communication agents of the cloud end layer or the edge layer, so that bidirectional duplex data transmission between cloud edges and between different edge local area networks is realized; on the other hand, the transmission data processed at the edge layer can be sent to the cloud end layer through the processing of the edge data channel module on the transmission data, so that the large-scale transmission of the data is realized; on the other hand, in the data transmission process between the cloud edge or the edge layer and the edge layer, the node state discovery function of the service discovery module is used, data transmission is carried out through the healthy nodes, regulation and control of transmitted data are achieved, efficient utilization of cloud edge bandwidth is facilitated, and load balancing is achieved.
The cloud-side data transmission control system in the present exemplary embodiment is further described below.
In an exemplary embodiment of the present disclosure, as shown in fig. 2, the data channel interface module 1201 may include:
an address acquisition unit 1211, configured to send a service request to the service discovery module 1203, and receive an address of the edge data channel module 1202 returned by the service discovery module 1203 based on the service request;
a connection establishing unit 1212, configured to send a connection request to the edge data channel module 1202 through the address acquired by the address acquiring unit 1211 to establish a connection between the edge application and the edge data channel module 1202.
Optionally, the address of the edge data channel module 1202 returned by the service discovery module 1203 based on the service request may be the real address of the edge data channel module 1202; optionally, the address of the edge data channel module 1202 returned by the service discovery module 1203 based on the service request may be a virtual address of the edge data channel module 1202, so as to establish a connection between the edge application and the edge data channel module 1202 according to the virtual address, based on which, when the address of the edge data channel module 1202 changes, the virtual address does not change correspondingly, and the data channel interface module 1201 does not need to track the change of a specific address, thereby improving connection effectiveness and data transmission stability.
In some possible embodiments, the service discovery module 1203 may also provide a corresponding relationship between the list of edge data channel modules 1202 and the virtual addresses, may select different virtual addresses of the edge data channel modules 1202 from the corresponding relationship based on a preset default selection mechanism through the address acquisition unit 1211, or may select different virtual addresses of the edge data channel modules 1202 from the corresponding relationship by a user, so as to achieve load balancing to some extent.
In an exemplary embodiment of the disclosure, as shown in fig. 3, the data channel interface module 1201 may further include:
an error detection unit 1213, configured to detect a state of a node where the edge data channel module 1202 is located;
a cache unit 1214, configured to cache data to be transmitted when the status is an offline status;
a connection unit 1215, configured to establish a connection with another edge data channel module when the status is the dropped status.
Specifically, the edge data tunnel module 1202 is a stateless proxy service module, and when the edge data tunnel module 1202 is in a dropped state, the error detection unit 1213 in the data tunnel interface module 1201 connected to the edge data tunnel module 1202 detects the dropped state, and performs transmission data caching through the cache unit 1214, or establishes a connection with another edge data tunnel module through the connection establishment unit 1215, so that it can be ensured that the edge data tunnel module in the edge layer is in a high availability state, thereby ensuring the fault tolerance and stability of cloud-edge data transmission or transmission between the edge layer and the edge layer data.
In some possible embodiments, since the data channel interface module 1201 is deployed in the application container, when the edge data channel module 1202 is in a drop state, the drop problem may also be detected by a kubernets + KubeEdge container management system located on an upper layer, and a restart node service is selected or an edge data channel module service is started at another node, so as to further ensure a high availability state of the edge data channel module.
In an exemplary embodiment of the present disclosure, as shown in fig. 4, the cloud-side data transmission control system further includes a plug-in management module 1204, which is connected to the edge data channel module 1202 and the cloud data channel module 1101 respectively; the edge data lane module 1202 may include: a data receiving unit, configured to receive a data packet sent by an edge application via the data channel interface module 1201; the data packet analyzing unit is used for analyzing the data packet to obtain analyzed data; and a data dispatching unit, configured to dispatch the analysis data to the plug-in management module 1204.
In an exemplary embodiment of the disclosure, the plug-in management module 1204 includes a plurality of plug-in sub-modules, which form a directed acyclic structure in a parallel or serial manner, etc.; wherein, as shown in fig. 5, the plurality of sub-plug-in modules comprise:
a first sub plug-in module 1241, configured to receive analysis data distributed by the data distribution unit;
a second sub-plug-in module 1242, configured to process the analysis data, and package the processed analysis data to obtain package data;
the third sub-plug-in module 1243 is configured to upload the packet data to the cloud data channel module 1101.
Specifically, when the edge application sends transmission data to the edge data channel module 1202 through the data channel interface module 1201, the data packet is received through the data receiving unit, the data packet is analyzed through the data packet analyzing unit to obtain analysis data, and finally the analysis data is distributed to the plug-in management module 1204 through the data distributing unit to be processed through analysis data. The first sub-plug-in module 1241 receives the analysis data, the second sub-plug-in module 1242 processes the analysis data, packages the processed analysis data, and finally uploads the packaged data to the cloud data channel module 1101 through the third sub-plug-in module 1243.
In some possible embodiments, the second daughter card module 1242 may include: the data aggregation unit is used for carrying out aggregation processing on the analysis data; the flow statistic unit is used for carrying out statistics on the transmission flow according to the analysis data; and the transmission scheduling unit is used for scheduling the transmission of the analysis data according to the convergence processing result and/or the transmission flow statistic result. The information leakage detection unit is used for detecting the privacy information in the analytic data so as to regulate and control the transmission of the analytic data according to the detection result; for example, a feature value extraction ratio peer-to-peer approach may be used to detect possible privacy leaks in parsed data.
It should be noted that the plug-in management module 1204 can analyze the transmission data packet in real time, for example, process the data packet according to the pre-configured processing logic, and process the data, including processing, forwarding, or discarding, so as to implement the customizable data scheduling.
Fig. 6 shows a schematic structural diagram of a plug-in management module 1204 according to an exemplary embodiment of the disclosure, and as shown in fig. 6, each sub plug-in module may have multiple combinations of parallel, serial, or parallel-serial combination, and data processing, including but not limited to data aggregation, scheduling, and the like, is implemented by combining sub plug-in modules with different basic functions.
It should be noted that the components in the plug-in management module 1204 according to the exemplary embodiment of the disclosure are not limited to the first sub-plug-in module 1241, the second sub-plug-in module 1242, and the third sub-plug-in module 1243, and each sub-plug-in module may also be composed of different modules (or units), and it is within the scope of the disclosure that the data processing function is realized by different sub-plug-in modules in different combination manners.
In an exemplary embodiment of the present disclosure, as shown in fig. 7, the service discovery module 1203 includes:
a health status detection unit 1231, configured to detect health statuses of nodes where the cloud data channel 1101 module and the edge data channel 1202 are located; the cluster construction unit 1232 is configured to form a node cluster from nodes with the same function type according to the function type of the node where the cloud data channel module 1101 or the edge data channel module 1202 is located; and a transmission data regulating and controlling unit 1233, configured to perform transmission regulation and control on transmission data according to the health state and the formed node cluster.
Specifically, the health state detection unit 1231 detects the health states of the nodes where the cloud data channel 1101 module and the edge data channel 1202 are located according to a preset period, and forms a plurality of nodes with the same function into a node cluster with the same name through the cluster construction unit 1232. When data transmission is performed, the transmission data regulation and control unit 1233 can be used for performing data transmission regulation and control distribution on all nodes in the same node cluster according to the health state, so that load balancing is achieved. In addition, when abnormal conditions such as temporary failure of node service occur, the transmission data are cached, and therefore the disaster tolerance performance is improved.
In an exemplary embodiment of the present disclosure, the cloud data channel module 1101 is deployed in a cluster manner, and the cloud data channel module 1101 uses a message queue manner as a transmission data bearer to provide acquisition and transmission of transmission data for a cloud application; each node in the message queue is deployed in a master-slave mode, when the master node fails, the slave node is elected as the master node, and data loss is reduced to the maximum extent due to real-time synchronization of the data of the master node and the slave node. The cluster node storage service uses a persistent infrastructure provided by Kubernetes, and can retransmit transmission data after the node is on-line again.
In some possible embodiments, the cloud data channel module 1101 deletes the transmission data in the message queue only after receiving the response message that the transmission data in the message queue has been consumed, and does not delete unconsumed data, thereby ensuring reliable transmission and disaster tolerance of the transmission data.
It should be noted that although several modules or units of the cloud-side data transmission control system are mentioned in the above detailed description, such division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
According to the cloud-edge data transmission control system in the embodiment of the present invention, the data channel interface modules are respectively deployed in the cloud end layer and each edge layer to serve as communication agents of the cloud end layer or the edge layer, so that bidirectional duplex data transmission between cloud edges and between different edge local area networks is realized; meanwhile, the transmission data processed at the edge layer can be sent to the cloud end layer through the processing of the edge data channel module on the transmission data, so that the large-scale transmission of the data is realized; in addition, in the data transmission process between the cloud edge or the edge layer and the edge layer, the node state discovery function of the service discovery module is used, data transmission is carried out through the healthy nodes, regulation and control of transmitted data are achieved, efficient utilization of cloud edge bandwidth is facilitated, and load balancing is achieved.
In addition, in an exemplary embodiment of the present disclosure, a cloud-side data transmission control method is further provided, which is applied to the edge data channel module in the cloud-side data transmission control system. Referring to fig. 8, the cloud-edge data transmission control method may include the steps of:
in step S810, transmission data sent by the edge application through the data channel interface module is acquired and processed;
in step S820, the processed transmission data is sent to a cloud data channel module of the cloud layer, so as to send the processed transmission data to the cloud application via the cloud data channel module;
or, the processed transmission data is sent to other edge data channel modules of other edge layers, so that the processed transmission data is sent to corresponding other edge applications through the other edge data channel modules;
in step S830, transmission data sent by the cloud application through the cloud data channel module is obtained and processed, and the processed transmission data is sent to the edge application through the data channel interface module;
in the transmission data sending process, routing among the modules is carried out through the service discovery module, the states of the nodes where the cloud data processing module and other edge data processing modules are located are monitored through the service discovery module, and transmission data are regulated and controlled according to the states.
In an exemplary embodiment of the present disclosure, the edge application needs to establish a connection with the edge data channel module through the data channel interface module; in some possible embodiments, the edge application uses an SDK (software development kit) provided by the data channel interface module to connect to the edge data channel module. Specifically, fig. 9 shows an interaction flowchart of establishing a connection between an edge application and an edge data channel module according to an exemplary embodiment of the present disclosure, where UAI is a data channel interface module, CEC-E is the edge data channel module, and CEC-C is a cloud data channel module. As shown in fig. 9, the process includes the steps of:
in step S910, the edge application sends a service request using the SDK provided by the UAI; and will send the address to find the module and obtain the request from step S920 to the service;
in step S930, the service discovery module receives the address returned by CEC-E;
in step S940, a connection establishment request is sent to CEC-E via the address;
in step S950, the CEC-E returns connection success response information is received.
Based on the method, when the data are sent, the UAI sends the data to the CEC-E, then uploads the data to the CEC-C in the cloud, and finally reaches the cloud application. If CEC-E connection interruption and other conditions occur, the UAI temporarily performs caching and performs retransmission after connection is recovered.
In an exemplary embodiment of the present disclosure, before performing cloud-side data transmission regulation, CEC-C and CEC-E need to be initialized, and fig. 10 shows an interaction flowchart for initializing CEC-C and CEC-E according to an exemplary embodiment of the present disclosure, such as fig. 10, where the process includes the following steps:
in step S1001, CEC-C is deployed at the cloud layer through kubernets API Server (K8S application program interface service);
in step S1002, CEC-E1 and CEC-E2 are deployed at the edge layer through Kubernets API Server; the CEC-E1 and the CEC-E2 may be deployed in the same edge layer, or may be deployed in different edge layers (that is, one or more CEC-es may be deployed in the same edge layer), and a corresponding number of CEC-es may be deployed in different edge layers according to an actual data transmission requirement, which is not particularly limited in this disclosure.
In step S1003, CEC-E1 and CEC-E2 respectively send registration requests to CEC-C;
in step S1004, CEC-E1 and CEC-E2 receive response information of registration completion returned from CEC-C, respectively, and complete registration of CEC-E1 and CEC-E2 to CEC-C;
in step S1005, the CEC-E service discovery service is deployed at the service discovery module through the kubernets API Server, so as to implement load balancing by scheduling data transmission of CEC-E1 and CEC-E2 through the service discovery module, respectively.
It should be noted that, since each functional module in the cloud-side data transmission control method according to the exemplary embodiment of the present disclosure is the same as each functional module (or unit) in the invention embodiment of the cloud-side data transmission control system, it is not described herein again.
In addition, in the exemplary embodiments of the present disclosure, a computer storage medium capable of implementing the above method is also provided. On which a program product capable of implementing the above-described method of the present specification is stored. In some possible embodiments, aspects of the present disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present disclosure described in the "exemplary methods" section above of this specification, when the program product is run on the terminal device.
Referring to fig. 11, a program product 1100 for implementing the above method according to an exemplary embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not so limited, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method or program product. Accordingly, various aspects of the present disclosure may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 1200 according to such an embodiment of the present disclosure is described below with reference to fig. 12. The electronic device 1200 shown in fig. 12 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present disclosure.
As shown in fig. 12, the electronic device 1200 is embodied in the form of a general purpose computing device. The components of the electronic device 1200 may include, but are not limited to: the at least one processing unit 1210, the at least one memory unit 1220, a bus 1230 connecting different system components (including the memory unit 1220 and the processing unit 1210), and a display unit 1240.
Wherein the storage unit stores program code that can be executed by the processing unit 1210 to cause the processing unit 1210 to perform the steps according to various exemplary embodiments of the present disclosure described in the above section "exemplary method" of this specification.
The storage unit 1220 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM) 1221 and/or a cache memory unit 1222, and may further include a read only memory unit (ROM) 1223.
The bus 1230 may be any bus representing one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
The electronic device 1200 can also communicate with one or more external devices 1300 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 1200, and/or any device (e.g., router, modem, etc.) that enables the electronic device 1200 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interfaces 1250. Also, the electronic device 1200 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 1260. As shown, the network adapter 1260 communicates with the other modules of the electronic device 1200 via a bus 1230. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, to name a few.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
Furthermore, the above-described figures are merely schematic illustrations of processes included in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.
Claims (10)
1. A cloud-edge data transmission control system comprises a cloud end layer and an edge layer, and is characterized in that the cloud end layer comprises a cloud end data channel module connected with a cloud end application and used for acting the cloud end layer to perform data transmission with the edge layer; the edge layer comprises a data channel interface module, an edge data channel module and a service discovery module;
the data channel interface module is used for establishing connection between the edge application and the edge data channel module;
the edge data channel module is used for acting the edge layer and the cloud end layer or other edge layers to perform data transmission and process transmission data;
the service discovery module is used for routing when data transmission is carried out between the cloud end layer and each module in the edge layer, monitoring the states of nodes where the cloud end data channel module and the edge data channel module are located, and regulating and controlling the transmission data according to the states.
2. The system of claim 1, wherein the data channel interface module comprises:
the address acquisition unit is used for sending a service request to the service discovery module and receiving the address of the edge data channel module returned by the service discovery module based on the service request;
and the connection establishing unit is used for sending a connection request to the edge data channel module through the address so as to establish the connection between the edge application and the edge data channel module.
3. The system of claim 2, wherein the data channel interface module further comprises:
the error detection unit is used for detecting the state of the node where the edge data channel module is located;
the cache unit is used for caching the data to be transmitted when the state is the offline state;
and the connection establishing unit is used for establishing connection with other edge data channel modules when the state is the disconnection state.
4. The system of claim 1, further comprising a plug-in management module respectively connected to the edge data channel module and the cloud data channel module;
the edge data channel module includes:
a data receiving unit, configured to receive a data packet sent by the edge application via the data channel interface module;
the data packet analyzing unit is used for analyzing the data packet to obtain analyzed data;
and the data dispatching unit is used for dispatching the analysis data to the plug-in management module.
5. The system of claim 4, wherein the plug-in management module comprises a plurality of plug-in sub-modules, the plurality of plug-in sub-modules forming a directed acyclic structure in a parallel or serial manner; the plurality of daughter card modules includes:
the first sub-plug-in module is used for receiving the analysis data distributed by the data distribution unit;
the second sub-plug-in module is used for processing the analysis data and packaging the processed analysis data to obtain packaged data;
and the third sub-plug-in module is used for uploading the packet data to the cloud data channel module.
6. The system of claim 5, wherein the second daughter card module comprises:
the data aggregation unit is used for carrying out aggregation processing on the analysis data;
the flow statistic unit is used for carrying out statistics on transmission flow according to the analysis data;
a transmission scheduling unit, configured to schedule transmission of the analysis data according to a convergence processing result and/or a transmission traffic statistical result;
and the information leakage detection unit is used for detecting the privacy information in the analysis data so as to regulate and control the transmission of the analysis data according to the detection result.
7. The system of claim 1, wherein the service discovery module comprises:
the health state detection unit is used for detecting the health states of the nodes where the cloud data channel module and the edge data channel module are located;
the cluster construction unit is used for forming nodes with the same function type into a node cluster according to the function type of the node where the cloud data channel module or the edge data channel module is located;
and the transmission data regulation and control unit is used for carrying out transmission regulation and control on the transmission data according to the health state and the formed node cluster.
8. The system according to any one of claims 1 to 7, wherein the cloud data channel modules are deployed in a cluster manner, and the cloud data channel modules are used as transmission data bearers in a message queue manner to provide acquisition and transmission of transmission data for the cloud application.
9. A cloud-side data transmission control method applied to an edge data channel module in the cloud-side data transmission control system according to any one of claims 1 to 8, the method comprising:
acquiring and processing transmission data sent by the edge application through the data channel interface module;
the processed transmission data are sent to a cloud data channel module of a cloud layer, so that the processed transmission data are sent to a cloud application through the cloud data channel module; or sending the processed transmission data to other edge data channel modules of other edge layers so as to send the processed transmission data to corresponding other edge applications through the other edge data channel modules;
the transmission data sent by the cloud application through the cloud data channel module is obtained and processed, and the processed transmission data is sent to the edge application through the data channel interface module;
in the transmission data sending process, routing among the modules is carried out through a service discovery module, the states of the nodes where the cloud data channel module and the other edge data channel modules are located are monitored through the service discovery module, and the transmission data are regulated and controlled according to the states.
10. A storage medium having stored thereon a computer program which, when executed by a processor, implements the cloud-edge data transmission control method according to claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111054930.1A CN115774580A (en) | 2021-09-09 | 2021-09-09 | Cloud-side data transmission control system, method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111054930.1A CN115774580A (en) | 2021-09-09 | 2021-09-09 | Cloud-side data transmission control system, method and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115774580A true CN115774580A (en) | 2023-03-10 |
Family
ID=85387957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111054930.1A Pending CN115774580A (en) | 2021-09-09 | 2021-09-09 | Cloud-side data transmission control system, method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115774580A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116389584A (en) * | 2023-05-17 | 2023-07-04 | 南方电网调峰调频发电有限公司储能科研院 | Cloud edge data transmission method for electrochemical energy storage station |
-
2021
- 2021-09-09 CN CN202111054930.1A patent/CN115774580A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116389584A (en) * | 2023-05-17 | 2023-07-04 | 南方电网调峰调频发电有限公司储能科研院 | Cloud edge data transmission method for electrochemical energy storage station |
| CN116389584B (en) * | 2023-05-17 | 2023-08-15 | 南方电网调峰调频发电有限公司储能科研院 | Cloud edge data transmission method for electrochemical energy storage station |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109088820B (en) | Cross-device link aggregation method and device, computing device and storage medium | |
| US10887019B2 (en) | Multi-layer system optimization | |
| CN111880902A (en) | Pod creation method, device, equipment and readable storage medium | |
| CN113742031A (en) | Node state information acquisition method and device, electronic equipment and readable storage medium | |
| CN110476154B (en) | Proxy server apparatus and method for data collection | |
| KR20140141053A (en) | System and method for accelerating mapreduce operation | |
| CN111885439B (en) | Optical network integrated management and duty management system | |
| CN111338893A (en) | Process log processing method and device, computer equipment and storage medium | |
| US10362120B2 (en) | Distributed gateways with centralized data center for high throughput satellite (HTS) spot beam network | |
| US20230137879A1 (en) | In-flight incremental processing | |
| Liu et al. | Sn-vne: A virtual network embedding algorithm for satellite networks | |
| CN106063205B (en) | The device and method that client traffic controls in remote access environment | |
| CN106330994A (en) | User message publishing method and system | |
| CN115774580A (en) | Cloud-side data transmission control system, method and storage medium | |
| US10666359B2 (en) | Multi-layer system self-optimization | |
| CN117194562A (en) | Data synchronization method and device, electronic equipment and computer readable medium | |
| CN116755799A (en) | Service arrangement system and method | |
| CN114697334B (en) | Method and device for executing scheduling task | |
| CN114615268B (en) | Service network, monitoring node, container node and equipment based on Kubernetes cluster | |
| CN116016307A (en) | Method and device for transmitting calculation network parameters and electronic equipment | |
| CN112398668B (en) | IaaS cluster-based cloud platform and node switching method | |
| CN114143903A (en) | Method, device, equipment and medium for configuring network slice PDU session | |
| US10784963B2 (en) | Multi-layer system capacity planning | |
| Adedokun et al. | Development of a modified East-West interface for distributed control plane network | |
| Chang et al. | An M2M computing model for improving the performance among devices |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |