CN118074861A

CN118074861A - Method, device, equipment and storage medium for eliminating transmission redundancy

Info

Publication number: CN118074861A
Application number: CN202410187122.XA
Authority: CN
Inventors: 王占营; 唐宝玲; 汪晓雨
Original assignee: Hefei Lianbao Information Technology Co Ltd
Current assignee: Hefei Lianbao Information Technology Co Ltd
Priority date: 2024-02-20
Filing date: 2024-02-20
Publication date: 2024-05-24

Abstract

The present disclosure provides a transmission redundancy elimination method, apparatus, device, and storage medium, where the method includes: acquiring data information of data transmitted by a server, and preprocessing the data information based on noise filtering to obtain first data information; based on the data information, judging whether the transmission signal corresponding to the data information is redundant or not, and collecting corresponding redundant data; determining a first elimination strategy or a second elimination strategy to eliminate the redundant data based on the generation node and the data information of the redundant data to obtain data to be optimized; and taking the data to be optimized as training data of an optimization model, creating and optimizing the optimization model based on the data to be optimized and a loss function, and optimizing the data to be optimized based on the optimization model to obtain optimized data.

Description

Method, device, equipment and storage medium for eliminating transmission redundancy

Technical Field

The present disclosure relates to the field of data transmission, and in particular, to a method, an apparatus, a device, and a storage medium for eliminating transmission redundancy.

Background

Currently, many servers employ data redundancy and signal integrity checking techniques that are mainly retrospective, i.e., redundancy and error checking after data transmission is completed. However, this technique has a slow reaction rate and cannot find and deal with problems in time. In addition, the probability of false alarm and false alarm is high, so that the efficiency of the server is reduced, and the server is limited by storage and calculation resources, so that large-scale data redundancy and error checking cannot be performed. In addition, even if redundancy and signal problems are handled, the existing technology cannot guarantee the quality of data transmission. In many cases, the signal after redundancy and error has been eliminated still presents quality problems. This is related to both the nature of the data itself (e.g., uncertainty in the data distribution and dynamic changes in the data) and to the uncertainty in the transmission process (e.g., network delays and packet losses). Thus, the quality of the data transmission cannot be ensured by merely eliminating redundancy and errors.

Disclosure of Invention

The present disclosure provides a transmission redundancy elimination method, apparatus, device, and storage medium, so as to at least solve the above technical problems in the prior art.

According to a first aspect of the present disclosure, there is provided a transmission redundancy elimination method, the method comprising:

Acquiring data information of data transmitted by a server, and preprocessing the data information based on noise filtering to obtain first data information;

based on the first data information, judging whether a transmission signal corresponding to the data information is redundant or not, and collecting corresponding redundant data;

Determining a first elimination strategy or a second elimination strategy to eliminate the redundant data based on the generation node and the data information of the redundant data to obtain data to be optimized;

And taking the data to be optimized as training data of an optimization model, creating and optimizing the optimization model based on the data to be optimized and a loss function, and optimizing the data to be optimized based on the optimization model to obtain optimized data.

In an embodiment, the determining, based on the first data information, whether redundancy occurs in the transmission signal corresponding to the data information includes:

setting a first threshold of the data information based on a signal type in the first data information;

Establishing a self-encoder, processing the first data information based on the encoder to obtain second data information, and comparing the second data information with the first threshold to obtain a first comparison result;

based on the first comparison result, whether the data information is redundant data is determined.

In an embodiment, the building self-encoder includes:

acquiring real-time data information transmitted by the server, and performing feature extraction on the real-time data information based on a hash function to obtain first feature information;

establishing an encoder model, wherein the encoder model consists of an encoder and a decoder, and training the encoder model by taking the first characteristic information as encoder model training data;

Discarding the decoder in the trained encoder model to obtain the self-encoder.

In an embodiment, the optimizing the data to be optimized based on the optimization model to obtain optimized data includes:

Acquiring data information of the data to be optimized, and setting initialization parameters of the optimization model based on the data information;

calculating transmission quality loss corresponding to the initialization parameter based on the loss function, and calculating a loss gradient corresponding to the initialization parameter based on the transmission quality loss;

and updating the initialization parameters by using the loss gradient based on the optimization model to obtain optimization data.

In an embodiment, after the obtaining the optimized data, the method further includes:

Inputting the optimization data into the optimization model, and optimizing the model by the optimization model based on the optimization data.

According to a second aspect of the present disclosure, there is provided a transmission redundancy elimination apparatus, characterized in that the apparatus includes:

the data collection unit is used for obtaining data information of data transmitted by the server, preprocessing the data information based on noise filtering, and obtaining first data information;

A redundancy determination unit, configured to determine, based on the data information, whether redundancy occurs in a transmission signal corresponding to the data information, and collect corresponding redundancy data;

The redundancy elimination unit is used for determining a first elimination strategy or a second elimination strategy to eliminate the redundancy data based on the generation node and the data information of the redundancy data to obtain data to be optimized;

The optimizing unit is used for taking the data to be optimized as training data of an optimizing model, creating and optimizing the optimizing model based on the data to be optimized and the loss function, and optimizing the data to be optimized based on the optimizing model to obtain optimizing data.

In an embodiment, the redundancy determining unit is further configured to set a first threshold of the data information based on a signal type in the data information; establishing a self-encoder, processing the first data information based on the encoder to obtain second data information, and comparing the second data information with the first threshold to obtain a first comparison result; based on the first comparison result, judging whether the data information is redundant data or not;

The optimizing unit is further used for acquiring data information of the data to be optimized and setting initialization parameters of the optimizing model based on the data information; calculating transmission quality loss corresponding to the initialization parameter based on the loss function, and calculating a loss gradient corresponding to the initialization parameter based on the transmission quality loss; updating the initialization parameters by using the loss gradient based on the optimization model to obtain optimization data; inputting the optimization data into the optimization model, and optimizing the model by the optimization model based on the optimization data.

In an embodiment, the redundancy determination unit further includes:

The encoder building unit is used for acquiring the real-time data information transmitted by the server, and extracting the characteristics of the real-time data information based on a hash function to obtain first characteristic information; establishing an encoder model, wherein the encoder model consists of an encoder and a decoder, and training the encoder model by taking the first characteristic information as encoder model training data; discarding the decoder in the trained encoder model to obtain the self-encoder.

According to a third aspect of the present disclosure, there is provided an electronic device comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the present disclosure.

According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of the present disclosure.

The method, the device, the equipment and the storage medium for eliminating transmission redundancy are characterized in that the integrity of a data signal is detected in a server, signal data is collected and processed, whether redundancy or distortion occurs in the signal transmission process is judged by a novel self-encoder algorithm based on deep learning, and the result is fed back. Then, a redundancy elimination mechanism is introduced, so that real-time monitoring, analysis and identification can be realized, redundancy transmission can be effectively eliminated, and high-efficiency transmission of data is ensured. Finally, an optimization module is introduced into the server architecture, and the signal optimization is performed on the analysis result after redundancy elimination, so that the performance of the server is improved. By the method, the accuracy and efficiency of the server on data transmission can be greatly improved, and the method has high application value.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Fig. 1 is a schematic diagram of an implementation flow of a transmission redundancy elimination method according to an embodiment of the disclosure;

Fig. 2 shows a second implementation flow diagram of a transmission redundancy elimination method according to an embodiment of the disclosure;

FIG. 3 illustrates a schematic diagram of a transmission redundancy elimination apparatus according to an embodiment of the disclosure;

Fig. 4 shows a schematic diagram of a composition structure of an electronic device according to an embodiment of the disclosure.

Detailed Description

In order to make the objects, features and advantages of the present disclosure more comprehensible, the technical solutions in the embodiments of the present disclosure will be clearly described in conjunction with the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. Based on the embodiments in this disclosure, all other embodiments that a person skilled in the art would obtain without making any inventive effort are within the scope of protection of this disclosure.

Fig. 1 shows a schematic implementation flow diagram of a transmission redundancy elimination method according to an embodiment of the disclosure, and as shown in fig. 1, the implementation flow of the transmission redundancy elimination method according to the embodiment of the disclosure includes the following steps:

Step 101, obtaining data information of data transmitted by a server, and preprocessing the data information based on noise filtering to obtain first data information.

In an embodiment of the present disclosure, obtaining data information of transmission data in a server transmission environment, where the data information includes: the central processing unit (Central Processing Unit, CPU) of the server uses the situation, the memory uses the situation, can also be the network bandwidth uses the situation. The original signal of the transmitted data is preprocessed by noise filtering, principal component analysis preprocessing, in preparation for subsequent integrity analysis. The first data information is the data information after preprocessing.

In an embodiment of the present disclosure, the signal preprocessing operation includes: dimension reduction techniques based on principal component analysis (PRINCIPAL COMPONENTS ANALYSIS, PCA). Through PCA, high-dimensional data can be converted into lower-dimensional data, and main characteristics of original data are reserved as much as possible, so that curse of the dimensions is avoided.

Step 102, based on the first data information, determining whether redundancy occurs in the transmission signal corresponding to the data information, and collecting corresponding redundancy data.

In the embodiment of the disclosure, a first threshold value of the data information is set based on a signal type in the data information; the first threshold may be an upper limit and a lower limit of the signal strength, or a frequency range of the signal. And establishing a self-encoder, and processing the first data information based on the encoder to obtain second data information, wherein the second data information is obtained by performing logic calculation by using the encoder based on the first data. Comparing the second data information with the first threshold value to obtain a first comparison result; based on the first comparison result, whether the data information is redundant data is determined. Wherein, the first comparison result is: when the second data information exceeds the first threshold range, the signal is considered to have redundancy or distortion.

In the embodiment of the disclosure, the operation of creating the self-encoder is specifically: acquiring real-time data information transmitted by the server, wherein the implementation data information is transmitted to the server through a monitoring system: the network layer, the server and the database layer are used for monitoring and acquiring, and the monitoring system mainly comprises the following components: a data collection component: the system is responsible for collecting and recording various data in the server architecture in real time, such as network packets, system logs, system indexes and the like; a data analysis component: the collected data is analyzed in real time, and the analysis mainly comprises the identification of redundant data, the statistics of data flow and the like. Performing feature extraction on the real-time data information based on a hash function to obtain first feature information; establishing an encoder model, wherein the encoder model consists of an encoder and a decoder, and training the encoder model by taking the first characteristic information as encoder model training data; discarding the decoder in the trained encoder model to obtain the self-encoder.

And 103, determining a first elimination strategy or a second elimination strategy to eliminate the redundant data based on the generation node of the redundant data and the data information, so as to obtain the data to be optimized.

In an embodiment of the present disclosure, the first cancellation strategy is: changing the data transmission mode to reduce network load; the second cancellation strategy is: and changing the routing mode to optimize the data transmission sequence. If redundancy occurs in transmitting the same data to a plurality of receiving points, a first elimination strategy is adopted, and a plurality of unicast is replaced by multicast so as to reduce network load; or a second elimination strategy is adopted, and the elimination of redundant data is realized by modifying the data routing mode of the server and optimizing the data sending sequence. And the data after redundancy elimination is the data to be optimized.

And 104, taking the data to be optimized as training data of an optimization model, creating and optimizing the optimization model based on the data to be optimized and a loss function, and optimizing the data to be optimized based on the optimization model to obtain optimized data.

In the embodiment of the disclosure, first, evaluating the signal quality of the data to be optimized includes: and detecting the strength, stability and integrity of the signal, and taking the evaluation result as a basis for data optimization. Preferably, an optimization model is created based on a gradient descent method and specific application scenes and requirements, and data information of the data to be optimized is obtained, wherein the data information comprises transmission frequency, transmission power and the like, and initialization parameters of the optimization model are set based on the data information; calculating transmission quality loss corresponding to the initialization parameter based on the loss function, and calculating a loss gradient corresponding to the initialization parameter based on the transmission quality loss; and updating the initialization parameters by using the loss gradient based on an optimization model to obtain optimization data. Inputting the optimization data into the optimization model, and optimizing the model by the optimization model based on the optimization data.

Fig. 2 shows a second implementation flow chart of a transmission redundancy elimination method according to an embodiment of the disclosure, and as shown in fig. 2, the implementation flow chart of the transmission redundancy elimination method according to the embodiment of the disclosure includes the following steps:

In step 201, signal data in a transmission environment of a server is collected.

In the embodiment of the disclosure, the data signals are collected in real time through various sensors or data collection software in a server transmission environment. The data may be the CPU usage of the server, the memory usage, or the network bandwidth usage. The collected data signals are subjected to signal preprocessing, wherein the data signals are preprocessed by noise filtering and principal component analysis preprocessing to prepare for subsequent integrity analysis.

In the embodiments of the present disclosure, to avoid the problem of "curse by dimension", a dimension reduction technique such as Principal Component Analysis (PCA) is introduced. Through PCA, high-dimensional data can be converted into lower-dimensional data, and meanwhile, main characteristics of original data are reserved as much as possible, and the method specifically comprises the following steps: firstly, constructing a data matrix X, setting a data signal set as an m-by-n dimensional matrix X, wherein each row represents one sample, each column represents one feature, and if the number of the samples is too small, carrying out row replication to construct a new data matrix X'; then, carrying out data normalization, averaging each characteristic column, and subtracting the corresponding average value from each sample data, so as to obtain a normalized data matrix X'; then, the covariance matrix is solved, and the normalized data matrix X' is multiplied by the transpose thereof to obtain a covariance matrix C. The formula is as follows: c= (X ") ^T; then, calculating eigenvalues and eigenvectors of covariance matrix, converting data from n-dimensional space to k-dimensional space, combining new k-dimensional space base by baseline of original space, and giving linear combination coefficient by eigenvectors; then, sorting the characteristic values, selecting the characteristic values with the previous k large, and finding out the corresponding characteristic vectors to form a matrix P; finally, dimension reduction is carried out, the data matrix after the normalization of the matrix P is carried out, and linear transformation is carried out on each column, so that the conversion from n-dimensional space to k-dimensional space is realized. The normalized data matrix is: xnew=p ^T ×x ", where X _new is the last obtained reduced-dimension data matrix, and each column is the coordinates of the original data in the new space.

Step 202, a redundancy elimination mechanism is established based on a custom algorithm.

In the embodiment of the disclosure, the monitoring system is deployed in the server architecture to collect and analyze the data transmission information of the server in real time; wherein, monitored control system deploys at: the monitoring system comprises a data collection component, a data analysis component and a data analysis component, wherein the data collection component is used for collecting and recording various data in a server architecture, such as network packets, system logs, system indexes and the like, and the data analysis component is used for carrying out real-time analysis on the collected data and mainly comprises redundant data identification, data traffic statistics and the like.

In the embodiment of the disclosure, based on the collected data transmission information, a self-encoder is built, specifically: and extracting characteristics of the collected data transmission information based on a hash function, wherein the mathematical formula of the hash function is as follows: h (k) =k mod m, H (k) is a hash function, k is the input (i.e., the collected data transfer information), m is the number of buckets, and redundant data in the collected data transfer information will be hashed into the same bucket; a self-encoder model is built based on an encoder for scaling down an input to an encoding and a decoder for restoring the encoding to the size of the input data and training the self-encoding based on a back-propagation method and a gradient descent method, wherein the training process of the self-encoder can be expressed by the following mathematical formula: min Σ|x ⁱ-x'ⁱ||²＝||xⁱ-g(f(xⁱ))||², where x ⁱ is the input data, f (x ⁱ) is the code obtained by encoder f, x' ⁱ is the code obtained by decoder g, the minimum reconstruction error of the target can be optimized based on the above formula, and the self-encoder is trained, when the self-encoder training is completed, the decoder therein is discarded, and the input is converted into a low-dimensional code by the encoder only. Observing the behavior of its code in the training set for a given input based on the redundancy of the trained self-encoder identification; an input may be considered redundant if its code is extremely similar or identical to other codes.

Step 203, the integrity of the signal data is analyzed.

In the embodiment of the disclosure, a specific signal data threshold is set according to the specific application scene and performance requirement of the server; for example, for certain specific signal types, the threshold may be set as an upper and lower limit for signal strength, or a range of signal frequencies, etc.

In an embodiment of the present disclosure, the pre-processed signal data is processed based on the trained self-encoder model, where the trained self-encoder is expressed as: min||X-F (G (X))| ², X is input data, F (X) and G (X) represent coding and decoding functions, processed data are compared with the set threshold, when the difference of the processed data exceeds the set threshold, the signal is considered to have redundancy or distortion condition, the signal with redundancy or distortion is marked, and subsequent processing or alarming is carried out.

Step 204, eliminating redundant data in the signal data.

In the disclosed embodiment, once a certain data packet is identified as redundant, an optimal redundancy elimination strategy is determined; if redundancy occurs in transmitting the same data to multiple receiving points, the network load is reduced by multicasting instead of multicasting.

In the embodiment of the disclosure, according to the analysis result, implementing a corresponding redundancy elimination strategy; the method comprises the steps of stopping sending redundant data, modifying a data routing mode of a server, and optimizing the sending sequence of the data. The data routing mode of the modification server is to consider that the path of the data packet is adjusted to the shortest path or the path with the least route under the condition of redundant data transmission. This process may be performed by dynamic routing protocols, such as the network state routing protocol (Open short PATH FIRST, OSPF), routing information protocol (Routing Information Protocol, RIP), and the like. These protocols dynamically calculate the optimal path to the destination address based on network conditions (e.g., link cost, network congestion, etc.), and then forward the packet along this path. A fixed path and fixed bandwidth problem for changing conventional static routing protocols; optimizing the sending sequence of the data can optimize the workload and network congestion problems of the server, sorts the data according to the priority, importance, data type and other factors of the data, and then sends the data according to the optimized sequence. Algorithms for optimizing the transmission order include first in/first out (First Input First Output, FIFO), play Priority (Priority), round Robin (Round Robin), etc.

Step 205, optimizing the transmission signal.

In the embodiment of the disclosure, in order to adapt to the change of the service architecture, the signal after redundancy elimination needs to be subjected to quality optimization. Firstly, the quality of the signal is evaluated, including the strength, stability, integrity and the like of the signal, so that the basis is provided for subsequent optimization. The signal strength can be generally estimated by signal received strength (RECEIVED SIGNAL STRENGTH Indicator, RSSI). RSSI is a measure of the power of a received signal and is typically expressed in dBm. For example, when the RSSI value is-40 dBm, the representative signal strength is high; when the RSSI value is-90 dBm, the signal strength represented is very low. The stability of a signal can be assessed by measuring the extent of signal quality fluctuations. Such as standard deviation or variance of signal received strength (RSSI), may measure the stability of the signal. If the variance of the RSSI is small, the signal is relatively stable; if the variance is large, the signal quality fluctuation is large, and the stability is poor. The integrity of a signal can generally be assessed by comparing a transmitted signal with a received signal. The transmitted signal may be encoded using a hash function, and after the receiving end receives the signal, the same hash function is used to encode the transmitted signal, and then the two encodings are compared. If the two codes agree, then the signal integrity can be considered to be relatively high; if the codes are not identical, then the signal is lost or erroneous.

In the embodiment of the disclosure, an optimization algorithm module is required to be established according to specific application scenes and requirements, and a proper optimization model and parameters are selected according to actual conditions; first, the parameter x needs to be initialized. This parameter represents the transmission power, frequency, etc. of the signal; then, defining a loss function f (x) for measuring the quality loss of signal transmission under the current parameter x; next, the gradient of the loss function f (x) with respect to the parameter x needs to be calculated, denoted as ∈f (x); updating x based on the gradient, wherein the updating formula is as follows: x=x- η, f (x), where η is the learning rate, which determines the step size of the parameter update. The above steps are repeated until the loss function f (x) drops to some acceptable level or a preset number of iterations is reached. The gradient descent method finds the value of x that minimizes the loss function f (x) by continuously adjusting the parameter x, i.e., the parameter setting that minimizes the loss of signal transmission quality.

In the embodiment of the disclosure, data to be optimized is input into the trained optimization model, and an optimization target is set, wherein the optimization target can be to reduce the distortion of signals or improve the stability of signals, and the optimization of signals is completed by calculating loss through the optimization model and adjusting parameter values according to gradients. And feeding back the optimized result to the optimized model, and updating and optimizing the optimized model so that the optimized model can adapt to the change of the server architecture.

Fig. 3 is a schematic diagram of a transmission redundancy elimination apparatus according to an embodiment of the disclosure, and as shown in fig. 3, the transmission redundancy elimination apparatus according to an embodiment of the disclosure includes the following units:

the data collection unit 301 is configured to obtain data information of data transmitted by the server, and perform preprocessing on the data information based on noise filtering to obtain first data information.

A redundancy determination unit 302, configured to determine, based on the first data information, whether redundancy occurs in a transmission signal corresponding to the data information, and collect corresponding redundancy data.

The redundancy determining unit 302 is further configured to set a first threshold of the data information based on a signal type in the first data information; establishing a self-encoder, processing the first data information based on the encoder to obtain second data information, and comparing the second data information with the first threshold to obtain a first comparison result; based on the first comparison result, whether the data information is redundant data is determined.

And the redundancy elimination unit 303 is configured to determine, based on the generation node and the data information of the redundancy data, a first elimination policy or a second elimination policy to eliminate the redundancy data, so as to obtain data to be optimized.

The optimizing unit 304 is configured to create an optimizing model based on the data to be optimized and the loss function, optimize the data to be optimized based on the optimizing model, and obtain optimized data.

The optimizing unit 304 is further configured to obtain data information of the data to be optimized, and set an initialization parameter of the optimization model based on the data information; calculating transmission quality loss corresponding to the initialization parameter based on the loss function, and calculating a loss gradient corresponding to the initialization parameter based on the transmission quality loss; updating the initialization parameters by using the loss gradient based on the optimization model to obtain optimization data; inputting the optimization data into the optimization model, and optimizing the model by the optimization model based on the optimization data.

The encoder establishing unit 305 is configured to obtain real-time data information transmitted by the server, and perform feature extraction on the real-time data information based on a hash function to obtain first feature information; establishing an encoder model, wherein the encoder model consists of an encoder and a decoder, and training the encoder model by taking the first characteristic information as encoder model training data; discarding the decoder in the trained encoder model to obtain the self-encoder.

In an exemplary embodiment, the data collection unit 301, the redundancy determination unit 302, the redundancy elimination unit 303, the optimization unit 304, the encoder creation unit 305, and the like may be implemented by one or more central processing units (CPU, central Processing Unit), graphics processors (GPU, graphics Processing Unit), application Specific Integrated Circuits (ASIC), application SPECIFIC INTEGRATED Circuit, DSP, programmable logic device (PLD, programmable Logic Device), complex Programmable logic device (CPLD, complex Programmable Logic Device), field-Programmable gate array (FPGA), general purpose processor, controller, microcontroller (MCU, micro Controller Unit), microprocessor (Microprocessor), or other electronic components.

The specific manner in which the various modules and units perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.

According to embodiments of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

Fig. 4 shows a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 4, the apparatus 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to the bus 804.

Various components in device 800 are connected to I/O interface 805, including: an input unit 806 such as a keyboard, mouse, etc.; an output unit 807 such as various types of displays, speakers, and the like; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, or the like. The communication unit 809 allows the device 800 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.

The computing unit 801 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 801 performs the respective methods and processes described above, for example, a transmission redundancy elimination method. For example, in some embodiments, a method of transmission redundancy elimination may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and/or installed onto device 800 via ROM 802 and/or communication unit 809. When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of one transmission redundancy elimination method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a transmission redundancy cancellation method in any other suitable way (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.

The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel or sequentially or in a different order, provided that the desired results of the technical solutions of the present disclosure are achieved, and are not limited herein.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present disclosure, the meaning of "a plurality" is two or more, unless explicitly defined otherwise.

The foregoing is merely specific embodiments of the disclosure, but the protection scope of the disclosure is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the disclosure, and it is intended to cover the scope of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A method for eliminating transmission redundancy, the method comprising:

2. The method of claim 1, wherein determining whether redundancy occurs in the transmission signal corresponding to the data information based on the first data information comprises:

3. The method of claim 2, wherein the establishing a self-encoder comprises:

Discarding the decoder in the trained encoder model to obtain the self-encoder.

4. The method according to claim 1, wherein optimizing the data to be optimized based on the optimization model to obtain optimized data comprises:

5. The method of claim 1, wherein after the obtaining the optimized data, the method further comprises:

6. An apparatus for eliminating transmission redundancy, the apparatus comprising:

A redundancy determination unit, configured to determine, based on the first data information, whether redundancy occurs in a transmission signal corresponding to the data information, and collect corresponding redundancy data;

7. The apparatus of claim 6, wherein the device comprises a plurality of sensors,

The redundancy determination unit is further configured to set a first threshold of the data information based on a signal type in the first data information; establishing a self-encoder, processing the first data information based on the encoder to obtain second data information, and comparing the second data information with the first threshold to obtain a first comparison result; based on the first comparison result, judging whether the data information is redundant data or not;

8. The apparatus according to claim 7, wherein the redundancy determination unit further includes:

9. An electronic device, comprising:

at least one processor; and

A memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1-5.