CN112465141B - Model compression method, device, electronic equipment and medium - Google Patents
Model compression method, device, electronic equipment and medium Download PDFInfo
- Publication number
- CN112465141B CN112465141B CN202011501677.5A CN202011501677A CN112465141B CN 112465141 B CN112465141 B CN 112465141B CN 202011501677 A CN202011501677 A CN 202011501677A CN 112465141 B CN112465141 B CN 112465141B
- Authority
- CN
- China
- Prior art keywords
- data
- model
- compressed
- preset
- simulation data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006835 compression Effects 0.000 title claims abstract description 54
- 238000007906 compression Methods 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000004088 simulation Methods 0.000 claims abstract description 124
- 230000004913 activation Effects 0.000 claims abstract description 78
- 238000012545 processing Methods 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 81
- 230000015654 memory Effects 0.000 claims description 21
- 239000013598 vector Substances 0.000 claims description 21
- 230000007787 long-term memory Effects 0.000 claims description 8
- 230000006403 short-term memory Effects 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000012549 training Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 2
- 101150043283 ccdA gene Proteins 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000013140 knowledge distillation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T90/00—Enabling technologies or technologies with a potential or indirect contribution to GHG emissions mitigation
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a data processing technology, and discloses a model compression method, which comprises the following steps: performing data fitting on random noise data by using a pre-constructed fitter to obtain simulation data, calculating activation loss values of the simulation data and the noise data, and adjusting parameters of the fitter when the activation loss values are larger than a preset activation threshold until the activation loss values are smaller than or equal to the preset activation threshold, and inputting the simulation data into a model to be compressed to obtain output data; calculating a sparse loss value of the output data and the simulation data, adjusting internal parameters of the fitting device when the sparse loss value is larger than a preset sparse threshold value until the sparse loss value is smaller than or equal to the preset sparse threshold value, outputting the simulation data and compressing the model to be compressed to obtain a compressed model. The invention also discloses a model compression device, electronic equipment and a storage medium. The invention can realize the compression of the model without obtaining training data, network structure, parameters and the like.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a model compression method, apparatus, electronic device, and computer readable storage medium.
Background
In order to apply the deep learning model to small-sized devices such as mobile devices and sensors, sometimes the deep learning model must be compressed and cut to be deployed to the small-sized devices.
Currently, the mainstream deep learning compression method is to compress a model based on an original training data set, a network structure, parameters and the like, such as a knowledge distillation method and a metadata-based method, wherein the former method needs a large amount of original training data, and the latter method needs the network structure and parameters of the model, but the training data, the network structure and the parameters are generally difficult to obtain due to legal, privacy and other reasons.
Disclosure of Invention
The invention provides a model compression method, a device, electronic equipment and a computer readable storage medium, and mainly aims to provide a scheme for carrying out model compression without acquiring training data, network structures and parameters.
In order to achieve the above object, the present invention provides a model compression method, including:
performing data fitting operation on random noise data by using a pre-constructed fitter to obtain simulation data;
Calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value, and returning to perform data fitting operation on random noise data by using a prefabricated fitter to obtain the simulation data, wherein the simulation data is input into a model to be compressed until the activation loss value is smaller than or equal to the preset activation threshold value to obtain output data;
Calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, adjusting internal parameters of the fitter when the sparse loss value is larger than a preset sparse threshold value, and returning to perform data fitting operation on random noise data by using a fitter constructed in advance to obtain the simulation data, and outputting the simulation data until the sparse loss value is smaller than or equal to the preset sparse threshold value;
and carrying out compression treatment on the model to be compressed according to the simulation data to obtain a compressed model.
Optionally, the performing data fitting operation on the random noise data by using a pre-built fitter to obtain simulation data includes:
predicting the noise data by using a long-term and short-term memory network in the fitter to obtain a fitting data set;
Compressing the fitting data set by using an activation function to obtain a compressed data set;
And vectorizing the compressed data set to obtain simulation data.
Optionally, the vectorizing the compressed data set to obtain simulation data includes:
Mapping the compressed data in the compressed data set into feature vectors by using a Word2Vec algorithm;
And splicing the feature vectors according to the sequences of the feature vectors to obtain the simulation data.
Optionally, the calculating an activation loss value between the simulation data and the noise data by using a preset first loss function includes:
Calculating an activation loss value between the simulation data and the noise data using a first loss function:
Wherein, For the activation loss value, n is the number of samples of the noise data,For the mth data in the simulation data, and 1 is the L1 norm.
Optionally, the calculating the sparse loss value between the output data and the simulation data by using a preset second loss function includes:
Calculating a sparse loss value between the output data and the simulation data using a second loss function:
Wherein, For the sparse loss value, x is the number of samples of the simulation data,Is the mth data of the output data,In order to set the parameters to be the preset parameters,Is a softmax loss function.
Optionally, the compressing the model to be compressed according to the simulation data to obtain a compressed model includes:
Inputting the simulation data into a preset standard compression model to perform vector operation to obtain a first feature output by the standard compression model, and inputting the simulation data into the model to be compressed to perform vector operation to obtain a second feature output by the model to be compressed;
Determining a loss function of the model to be compressed according to the first characteristic and the second characteristic;
And carrying out back propagation on the model to be compressed according to the loss function to obtain a compressed model.
Optionally, the determining the loss function of the model to be compressed according to the first feature and the second feature includes:
performing difference calculation according to the first characteristic and the second characteristic to obtain a difference function;
and carrying out norm conversion processing on the difference function and squaring the difference function to obtain a loss function.
In order to solve the above problems, the present invention also provides a model compression apparatus, the apparatus comprising:
the data fitting module is used for performing data fitting operation on the random noise data by utilizing a pre-constructed fitter to obtain simulation data;
The activation loss module is used for calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value until the activation loss value is smaller than or equal to the preset activation threshold value, and inputting the simulation data into a model to be compressed to obtain output data;
The sparse loss module is used for calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, and when the sparse loss value is larger than a preset sparse threshold value, adjusting internal parameters of the fitter until the sparse loss value is smaller than or equal to the preset sparse threshold value, and outputting the simulation data;
and the model compression module is used for compressing the model to be compressed according to the simulation data to obtain a compressed model.
In order to solve the above-mentioned problems, the present invention also provides an electronic apparatus including:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the model compression method described above.
In order to solve the above-mentioned problems, the present invention also provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the above-mentioned model compression method.
According to the embodiment of the invention, the random noise data is subjected to data fitting operation by utilizing a prefabricated fitter to obtain simulation data; calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, and adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value until the activation loss value is smaller than or equal to the preset activation threshold value, and inputting the simulation data into a model to be compressed to obtain output data; calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, and adjusting internal parameters of the fitter when the sparse loss value is larger than a preset sparse threshold value until the sparse loss value is smaller than or equal to the preset sparse threshold value, and outputting the simulation data; and verifying the data simulated by the fitter by using the two loss functions to obtain simulation data closest to noise data, and compressing the model to be compressed according to the simulation data to obtain a compressed model. Therefore, the model compression method, the device and the computer readable storage medium can solve the problem that training data, network structures and parameters which are difficult to acquire are required to be acquired for model compression.
Drawings
FIG. 1 is a flow chart of a model compression method according to an embodiment of the present invention;
FIG. 2 is a schematic block diagram of a model compressing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device for implementing a model compression method according to an embodiment of the present invention;
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The embodiment of the application provides a model compression method. The execution subject of the model compression method includes, but is not limited to, at least one of a server, a terminal, and the like, which can be configured to execute the method provided by the embodiment of the application. In other words, the model compression method may be performed by software or hardware installed in a terminal device or a server device, and the software may be a blockchain platform. The service end includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.
Referring to fig. 1, a flow chart of a model compression method according to an embodiment of the present invention is shown. In this embodiment, the model compression method includes:
S1, performing data fitting operation on random noise data by using a pre-built fitter to obtain simulation data.
In the embodiment of the invention, the random noise data is random Gaussian noise obtained by sampling from Gaussian distribution. The fitter is used for continuously carrying out linear fitting processing on the noise data to generate simulation data approximate to real data.
Specifically, the performing data fitting operation on random noise data by using a pre-built fitter to obtain simulation data includes:
predicting the noise data by using a long-term and short-term memory network in the fitter to obtain a fitting data set;
Compressing the fitting data set by using an activation function to obtain a compressed data set;
And vectorizing the compressed data set to obtain simulation data.
The long-term and short-term memory network can train the mapping of random noise from Gaussian distribution to fitting distribution, and in order to prevent overfitting, each layer of neural network of the long-term and short-term memory network is added with a dropout mechanism. The activation function may be a tanh function with which data in the fitting dataset is compressed between-1 and 1 for subsequent vectorization operations.
Further, the vectorizing the compressed data set to obtain simulation data includes:
Mapping the compressed data in the compressed data set into feature vectors by using a Word2Vec algorithm;
And splicing the feature vectors according to the sequences of the feature vectors to obtain the simulation data.
The Word2Vec algorithm can map data into a vector with uniform dimension, and the Word2Vec algorithm is suitable for analyzing data more generally under the condition that data of one sequence have strong correlation among sequence local data.
In detail, the data fitting operation is performed on the random noise data by using a pre-constructed fitter, so that simulation data close to the random noise data can be obtained and used for replacing the random noise data to perform subsequent model compression.
S2, calculating an activation loss value between the simulation data and the noise data by using a preset first loss function.
In the embodiment of the present invention, the first loss function:
Wherein, For the activation loss value, n is the number of samples of the noise data,For the mth data in the simulation data, and 1 is the L1 norm. The L1 norm is mainly for sparsity, and the negative sign is for not sparsity as much as possible, letAs much as possible is activated.
When the activation loss value is larger than a preset activation threshold, the embodiment of the invention adjusts the parameters of the fitter and returns to the S1, and the data fitting operation is performed on the random noise data by using the fitter constructed in advance again to obtain the simulation data.
Preferably, the parameters of the fitter may be the weight, gradient, etc. of the fitter.
And when the activation loss value is smaller than or equal to a preset activation threshold value, executing S3, and inputting the simulation data into a model to be compressed to obtain output data.
The first loss function calculates an activation loss value between the simulation data and the noise data, compares the activation loss value with a preset activation threshold value, and further adjusts parameters of the fitter until the activation loss value between the simulation data and the noise data converges, and at the moment, the adjusted fitter meets the standard without adjusting the parameters.
S4, calculating a sparse loss value between the output data and the simulation data by using a preset second loss function.
In an embodiment of the present invention, the second loss function may be
Wherein,For the sparse loss value, x is the number of samples of the simulation data,Is the mth data of the output data,In order to set the parameters to be the preset parameters,Is a softmax loss function.
When the sparseness loss value is larger than a preset sparseness threshold value, the embodiment of the invention adjusts the internal parameters of the fitter and returns to the S1, and the data fitting operation is carried out on the random noise data again by using the fitter constructed in advance to obtain simulation data.
And when the sparsity loss value is smaller than or equal to a preset sparsity threshold value, executing S5, outputting the simulation data, and carrying out compression processing on the model to be compressed according to the simulation data to obtain a compressed model.
In the embodiment of the present invention, the compressing the model to be compressed according to the simulation data to obtain a compressed model includes:
Inputting the simulation data into a preset standard compression model to perform vector operation to obtain a first feature output by the standard compression model, and inputting the simulation data into the model to be compressed to perform vector operation to obtain a second feature output by the model to be compressed;
Determining a loss function of the model to be compressed according to the first characteristic and the second characteristic;
And carrying out back propagation on the model to be compressed according to the loss function to obtain a compressed model.
Specifically, the determining the loss function of the model to be compressed according to the first feature and the second feature includes:
performing difference calculation according to the first characteristic and the second characteristic to obtain a difference function;
and carrying out norm conversion processing on the difference function and squaring the difference function to obtain a loss function.
FIG. 2 is a schematic block diagram of the model compressing apparatus of the present invention.
The model compressing apparatus 100 of the present invention may be installed in an electronic device. Depending on the implemented functionality, the model compression device 100 may include a data fitting module 101, an activation loss module 102, a sparseness loss module 103, a model compression module 104. The module of the invention, which may also be referred to as a unit, refers to a series of computer program segments, which are stored in the memory of the electronic device, capable of being executed by the processor of the electronic device and of performing a fixed function.
In the present embodiment, the functions concerning the respective modules/units are as follows:
the data fitting module 101 is configured to perform a data fitting operation on random noise data by using a pre-constructed fitter, so as to obtain simulation data;
the activation loss module 102 is configured to calculate an activation loss value between the simulation data and the noise data by using a preset first loss function, and adjust parameters of the fitter when the activation loss value is greater than a preset activation threshold until the activation loss value is less than or equal to the preset activation threshold, and input the simulation data into a model to be compressed to obtain output data;
The sparse loss module 103 is configured to calculate a sparse loss value between the output data and the simulation data by using a preset second loss function, and when the sparse loss value is greater than a preset sparse threshold, adjust internal parameters of the fitter until the sparse loss value is less than or equal to the preset sparse threshold, and output the simulation data;
the model compression module 104 is configured to compress the model to be compressed according to the simulation data, to obtain a compressed model.
In detail, when each module in the model compression apparatus 100 is executed by a processor of an electronic device, a model compression method may be implemented, and the specific implementation steps of the model compression method are as follows:
step one, the data fitting module 101 performs data fitting operation on random noise data by using a pre-constructed fitter, so as to obtain simulation data.
In the embodiment of the invention, the random noise data is random Gaussian noise obtained by sampling from Gaussian distribution. The fitter is used for continuously carrying out linear fitting processing on the noise data to generate simulation data approximate to real data.
Specifically, the data fitting module 101 performs a data fitting operation on random noise data by using a pre-constructed fitter to obtain simulation data, including:
predicting the noise data by using a long-term and short-term memory network in the fitter to obtain a fitting data set;
Compressing the fitting data set by using an activation function to obtain a compressed data set;
And vectorizing the compressed data set to obtain simulation data.
The long-term and short-term memory network can train the mapping of random noise from Gaussian distribution to fitting distribution, and in order to prevent overfitting, each layer of neural network of the long-term and short-term memory network is added with a dropout mechanism. The activation function may be a tanh function with which data in the fitting dataset is compressed between-1 and 1 for subsequent vectorization operations.
Further, the vectorizing the compressed data set to obtain simulation data includes:
Mapping the compressed data in the compressed data set into feature vectors by using a Word2Vec algorithm;
And splicing the feature vectors according to the sequences of the feature vectors to obtain the simulation data.
And step two, the activation loss module 102 calculates an activation loss value between the simulation data and the noise data by using a preset first loss function.
In the embodiment of the present invention, the first loss function:
Wherein, For the activation loss value, n is the number of samples of the noise data,For the mth data in the simulation data, and 1 is the L1 norm. The L1 norm is mainly for sparsity, and the negative sign is for not sparsity as much as possible, letAs much as possible is activated.
When the activation loss value is larger than a preset activation threshold, the embodiment of the invention adjusts the parameters of the fitter and returns to the step one, and the data fitting operation is performed on the random noise data by using the fitter constructed in advance again to obtain simulation data.
Preferably, the parameters of the fitter may be the weight, gradient, etc. of the fitter.
And when the activation loss value is smaller than or equal to a preset activation threshold value, executing a third step, and inputting the simulation data into a model to be compressed to obtain output data.
And step four, the sparse loss module 103 calculates a sparse loss value between the output data and the simulation data by using a preset second loss function.
In an embodiment of the present invention, the second loss function may be
Wherein,For the sparse loss value, x is the number of samples of the simulation data,Is the mth data of the output data,In order to set the parameters to be the preset parameters,Is a softmax loss function.
When the sparse loss value is larger than a preset sparse threshold, the embodiment of the invention adjusts the internal parameters of the fitter and returns to the step one, and the data fitting operation is carried out on the random noise data again by using the fitter constructed in advance to obtain simulation data.
And when the sparse loss value is smaller than or equal to a preset sparse threshold value, executing a fifth step, outputting the simulation data, and carrying out compression processing on the model to be compressed according to the simulation data to obtain a compressed model.
In the embodiment of the present invention, the compressing the model to be compressed according to the simulation data to obtain a compressed model includes:
Inputting the simulation data into a preset standard compression model to perform vector operation to obtain a first feature output by the standard compression model, and inputting the simulation data into the model to be compressed to perform vector operation to obtain a second feature output by the model to be compressed;
Determining a loss function of the model to be compressed according to the first characteristic and the second characteristic;
And carrying out back propagation on the model to be compressed according to the loss function to obtain a compressed model.
Specifically, the determining the loss function of the model to be compressed according to the first feature and the second feature includes:
performing difference calculation according to the first characteristic and the second characteristic to obtain a difference function;
and carrying out norm conversion processing on the difference function and squaring the difference function to obtain a loss function.
Fig. 3 is a schematic structural diagram of an electronic device implementing the model compression method according to the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a model compaction program 12, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may in other embodiments also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only for storing application software installed in the electronic device 1 and various types of data, such as code of the model compression program 12, but also for temporarily storing data that has been output or is to be output.
The processor 10 may be comprised of integrated circuits in some embodiments, for example, a single packaged integrated circuit, or may be comprised of multiple integrated circuits packaged with the same or different functions, including one or more central processing units (Central Processing unit, CPU), microprocessors, digital processing chips, graphics processors, combinations of various control chips, and the like. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects respective components of the entire electronic device using various interfaces and lines, executes or executes programs or modules (e.g., executes a model compression program, etc.) stored in the memory 11, and invokes data stored in the memory 11 to perform various functions of the electronic device 1 and process data.
The bus may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus, or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. The bus may be classified as an address bus, a data bus, a control bus, etc. The bus is arranged to enable a connection communication between the memory 11 and at least one processor 10 etc.
Fig. 3 shows only an electronic device with components, it being understood by a person skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or may combine certain components, or may be arranged in different components.
For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for supplying power to each component, and preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management, and the like are implemented through the power management device. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device 1 may further include various sensors, bluetooth modules, wi-Fi modules, etc., which will not be described herein.
Further, the electronic device 1 may also comprise a network interface, optionally the network interface may comprise a wired interface and/or a wireless interface (e.g. WI-FI interface, bluetooth interface, etc.), typically used for establishing a communication connection between the electronic device 1 and other electronic devices.
The electronic device 1 may optionally further comprise a user interface, which may be a Display, an input unit, such as a Keyboard (Keyboard), or a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device 1 and for displaying a visual user interface.
It should be understood that the embodiments described are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.
The model compression program 12 stored in the memory 11 in the electronic device 1 is a combination of instructions that, when executed in the processor 10, may implement:
performing data fitting operation on random noise data by using a pre-constructed fitter to obtain simulation data;
Calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value, and returning to perform data fitting operation on random noise data by using a prefabricated fitter to obtain the simulation data, wherein the simulation data is input into a model to be compressed until the activation loss value is smaller than or equal to the preset activation threshold value to obtain output data;
Calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, adjusting internal parameters of the fitter when the sparse loss value is larger than a preset sparse threshold value, and returning to perform data fitting operation on random noise data by using a fitter constructed in advance to obtain the simulation data, and outputting the simulation data until the sparse loss value is smaller than or equal to the preset sparse threshold value;
and carrying out compression treatment on the model to be compressed according to the simulation data to obtain a compressed model.
Further, the modules/units integrated in the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. The computer readable storage medium may be volatile or nonvolatile, for example, the computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM).
The present invention also provides a computer readable storage medium storing a computer program which, when executed by a processor of an electronic device, can implement:
performing data fitting operation on random noise data by using a pre-constructed fitter to obtain simulation data;
Calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value, and returning to perform data fitting operation on random noise data by using a prefabricated fitter to obtain the simulation data, wherein the simulation data is input into a model to be compressed until the activation loss value is smaller than or equal to the preset activation threshold value to obtain output data;
Calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, adjusting internal parameters of the fitter when the sparse loss value is larger than a preset sparse threshold value, and returning to perform data fitting operation on random noise data by using a fitter constructed in advance to obtain the simulation data, and outputting the simulation data until the sparse loss value is smaller than or equal to the preset sparse threshold value;
and carrying out compression treatment on the model to be compressed according to the simulation data to obtain a compressed model.
Further, the computer-usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created from the use of blockchain nodes, and the like.
In the several embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be other manners of division when actually implemented.
The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any accompanying diagram representation in the claims should not be considered as limiting the claim concerned.
Furthermore, it is evident that the word "comprising" does not exclude other elements or steps, and that the singular does not exclude a plurality. A plurality of units or means recited in the system claims can also be implemented by means of software or hardware by means of one unit or means. The terms second, etc. are used to denote a name, but not any particular order.
Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.
Claims (7)
1. A method of model compression, the method comprising:
Step A: performing data fitting operation on random noise data by using a pre-constructed fitter to obtain simulation data;
And (B) step (B): calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value, and returning to the step A until the activation loss value is smaller than or equal to the preset activation threshold value, and inputting the simulation data into a model to be compressed to obtain output data;
Step C: calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, adjusting internal parameters of the fitter when the sparse loss value is larger than a preset sparse threshold value, and returning to the step A until the sparse loss value is smaller than or equal to the preset sparse threshold value, and outputting the simulation data;
Step D: compressing the model to be compressed according to the simulation data to obtain a compressed model;
Wherein the calculating an activation loss value between the simulation data and the noise data by using a preset first loss function includes: calculating an activation loss value between the simulation data and the noise data using a first loss function:
Wherein, For the activation loss value, n is the number of samples of the noise data,For the mth data in the simulation data, the L 1 is the L1 norm;
The calculating the sparse loss value between the output data and the simulation data by using a preset second loss function comprises the following steps: calculating a sparse loss value between the output data and the simulation data using a second loss function:
Wherein, For the sparse loss value, x is the number of samples of the simulation data,Is the mth data of the output data,In order to set the parameters to be the preset parameters,Is a softmax loss function;
The compressing the model to be compressed according to the simulation data to obtain a compressed model, which comprises the following steps: inputting the simulation data into a preset standard compression model to perform vector operation to obtain a first feature output by the standard compression model, and inputting the simulation data into the model to be compressed to perform vector operation to obtain a second feature output by the model to be compressed; determining a loss function of the model to be compressed according to the first characteristic and the second characteristic; and carrying out back propagation on the model to be compressed according to the loss function to obtain a compressed model.
2. The method for compressing a model of claim 1, wherein said performing a data fitting operation on random noise data using a pre-constructed fitter to obtain simulated data comprises:
predicting the noise data by using a long-term and short-term memory network in the fitter to obtain a fitting data set;
Compressing the fitting data set by using an activation function to obtain a compressed data set;
And vectorizing the compressed data set to obtain simulation data.
3. The method for compressing a model of claim 2, wherein said vectorizing said compressed data set to obtain simulation data comprises:
Mapping the compressed data in the compressed data set into feature vectors by using a Word2Vec algorithm;
And splicing the feature vectors according to the sequences of the feature vectors to obtain the simulation data.
4. The model compression method of claim 1, wherein the determining the loss function of the model to be compressed from the first feature and the second feature comprises:
performing difference calculation according to the first characteristic and the second characteristic to obtain a difference function;
and carrying out norm conversion processing on the difference function and squaring the difference function to obtain a loss function.
5. Model compression apparatus for implementing a model compression method according to any one of claims 1 to 4, characterized in that the apparatus comprises:
the data fitting module is used for performing data fitting operation on the random noise data by utilizing a pre-constructed fitter to obtain simulation data;
The activation loss module is used for calculating an activation loss value between the simulation data and the noise data by using a preset first loss function, adjusting parameters of the fitter when the activation loss value is larger than a preset activation threshold value until the activation loss value is smaller than or equal to the preset activation threshold value, and inputting the simulation data into a model to be compressed to obtain output data;
The sparse loss module is used for calculating a sparse loss value between the output data and the simulation data by using a preset second loss function, and when the sparse loss value is larger than a preset sparse threshold value, adjusting internal parameters of the fitter until the sparse loss value is smaller than or equal to the preset sparse threshold value, and outputting the simulation data;
and the model compression module is used for compressing the model to be compressed according to the simulation data to obtain a compressed model.
6. An electronic device, the electronic device comprising:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein,
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the model compression method of any one of claims 1 to 4.
7. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the model compression method according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011501677.5A CN112465141B (en) | 2020-12-18 | 2020-12-18 | Model compression method, device, electronic equipment and medium |
| PCT/CN2021/083080 WO2022126902A1 (en) | 2020-12-18 | 2021-03-25 | Model compression method and apparatus, electronic device, and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011501677.5A CN112465141B (en) | 2020-12-18 | 2020-12-18 | Model compression method, device, electronic equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112465141A CN112465141A (en) | 2021-03-09 |
| CN112465141B true CN112465141B (en) | 2024-06-28 |
Family
ID=74803596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011501677.5A Active CN112465141B (en) | 2020-12-18 | 2020-12-18 | Model compression method, device, electronic equipment and medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN112465141B (en) |
| WO (1) | WO2022126902A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112465141B (en) * | 2020-12-18 | 2024-06-28 | 平安科技（深圳）有限公司 | Model compression method, device, electronic equipment and medium |
| CN112508194B (en) * | 2021-02-02 | 2022-03-18 | 支付宝(杭州)信息技术有限公司 | Model compression method, system and computing equipment |
| CN115269940B (en) * | 2022-09-30 | 2022-12-13 | 佳卓智能科技(南通)有限责任公司 | Data compression method of ERP management system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108564129A (en) * | 2018-04-24 | 2018-09-21 | 电子科技大学 | A kind of track data sorting technique based on generation confrontation network |
| CN109063742A (en) * | 2018-07-06 | 2018-12-21 | 平安科技（深圳）有限公司 | Butterfly identifies network establishing method, device, computer equipment and storage medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10832139B2 (en) * | 2018-06-22 | 2020-11-10 | Moffett Technologies Co. Limited | Neural network acceleration and embedding compression systems and methods with activation sparsification |
| CN110874550A (en) * | 2018-08-31 | 2020-03-10 | 华为技术有限公司 | Data processing method, device, equipment and system |
| CN109784474B (en) * | 2018-12-24 | 2020-12-11 | 宜通世纪物联网研究院(广州)有限公司 | Deep learning model compression method and device, storage medium and terminal equipment |
| CN109919864A (en) * | 2019-02-20 | 2019-06-21 | 重庆邮电大学 | A kind of compression of images cognitive method based on sparse denoising autoencoder network |
| US11556796B2 (en) * | 2019-03-25 | 2023-01-17 | Nokia Technologies Oy | Compressing weight updates for decoder-side neural networks |
| CN111612143B (en) * | 2020-05-22 | 2023-12-19 | 中国科学院自动化研究所 | Compression method and system of deep convolutional neural network |
| CN111814962B (en) * | 2020-07-09 | 2024-05-10 | 平安科技（深圳）有限公司 | Parameter acquisition method and device for identification model, electronic equipment and storage medium |
| CN112465141B (en) * | 2020-12-18 | 2024-06-28 | 平安科技（深圳）有限公司 | Model compression method, device, electronic equipment and medium |
-
2020
- 2020-12-18 CN CN202011501677.5A patent/CN112465141B/en active Active
-
2021
- 2021-03-25 WO PCT/CN2021/083080 patent/WO2022126902A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108564129A (en) * | 2018-04-24 | 2018-09-21 | 电子科技大学 | A kind of track data sorting technique based on generation confrontation network |
| CN109063742A (en) * | 2018-07-06 | 2018-12-21 | 平安科技（深圳）有限公司 | Butterfly identifies network establishing method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112465141A (en) | 2021-03-09 |
| WO2022126902A1 (en) | 2022-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112465141B (en) | Model compression method, device, electronic equipment and medium | |
| CN114330713B (en) | Convolutional neural network model pruning method and device, electronic equipment and storage medium | |
| CN110135437A (en) | For the damage identification method of vehicle, device, electronic equipment and computer storage medium | |
| CN113705462B (en) | Face recognition method, device, electronic equipment and computer readable storage medium | |
| CN111783974A (en) | Model construction and image processing method and device, hardware platform and storage medium | |
| CN113378970B (en) | Sentence similarity detection method and device, electronic equipment and storage medium | |
| CN113627361B (en) | Training method and device for face recognition model and computer program product | |
| CN116701574A (en) | Text semantic similarity calculation method, device, equipment and storage medium | |
| CN116681082A (en) | Discrete text semantic segmentation method, device, equipment and storage medium | |
| CN116701635A (en) | Training video text classification method, training video text classification device, training video text classification equipment and storage medium | |
| CN116401602A (en) | Event detection method, device, equipment and computer readable medium | |
| CN116564322A (en) | Voice conversion method, device, equipment and storage medium | |
| CN116741358A (en) | Inquiry registration recommendation method, inquiry registration recommendation device, inquiry registration recommendation equipment and storage medium | |
| CN114792097B (en) | Method and device for determining prompt vector of pre-training model and electronic equipment | |
| CN116468025A (en) | Electronic medical record structuring method and device, electronic equipment and storage medium | |
| CN113515591B (en) | Text defect information identification method and device, electronic equipment and storage medium | |
| CN111859985B (en) | AI customer service model test method and device, electronic equipment and storage medium | |
| CN113344415A (en) | Deep neural network-based service distribution method, device, equipment and medium | |
| CN112434157A (en) | Document multi-label classification method and device, electronic equipment and storage medium | |
| CN116578690B (en) | Insurance customer service method and system based on artificial intelligence | |
| CN114781833B (en) | Capability assessment method, device and equipment based on business personnel and storage medium | |
| CN115098644B (en) | Image and text matching method and device, electronic equipment and storage medium | |
| CN113723554B (en) | Model scheduling method, device, electronic equipment and computer readable storage medium | |
| CN118197620A (en) | Peripheral T cell lymphoma prognosis evaluation method and system | |
| CN116757197A (en) | Text theme segmentation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40041514 Country of ref document: HK |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |