CN110580523B - Error calibration method and device for analog neural network processor - Google Patents

Error calibration method and device for analog neural network processor Download PDF

Info

Publication number
CN110580523B
CN110580523B CN201810580960.8A CN201810580960A CN110580523B CN 110580523 B CN110580523 B CN 110580523B CN 201810580960 A CN201810580960 A CN 201810580960A CN 110580523 B CN110580523 B CN 110580523B
Authority
CN
China
Prior art keywords
processor
weight parameters
error
phase
trainable
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201810580960.8A
Other languages
Chinese (zh)
Other versions
CN110580523A (en
Inventor
贾凯歌
乔飞
魏琦
樊子辰
刘辛军
杨华中
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to CN201810580960.8A priority Critical patent/CN110580523B/en
Priority to PCT/CN2018/104781 priority patent/WO2019232965A1/en
Publication of CN110580523A publication Critical patent/CN110580523A/en
Application granted granted Critical
Publication of CN110580523B publication Critical patent/CN110580523B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G06N3/065—Analogue means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Stored Programmes (AREA)

Abstract

The embodiment of the invention provides an error calibration method and device for a simulated neural network processor, wherein the method comprises the following steps: if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure; training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain; replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations; and storing the learned weight parameters for the NN to calibrate the error of the processor according to the weight parameters. The device performs the above method. The error calibration method and device for the simulated neural network processor provided by the embodiment of the invention can reduce the energy and resource consumption of the simulated NN processor, thereby improving the efficiency of the simulated NN processor.

Description

Error calibration method and device for analog neural network processor
Technical Field
The embodiment of the invention relates to the technical field of error calibration, in particular to an error calibration method and device of an analog neural network processor.
Background
Compared with a digital circuit, the analog circuit has the characteristics of low power consumption, small chip size and the like due to the clock-free characteristic, so that an analog Neural Network (NN) processor is widely applied.
However, although the analog NN processor has high energy efficiency, the nonlinearity of the system and the device still leaves room for improvement in the problem of processing accuracy, resulting in errors mainly from variations in the manufacturing process, voltage or temperature offsets, and various noise sources. Although the simulated NN algorithm can tolerate these defects to some extent, the simulated NN processor still has a series of problems such as non-negligible accuracy degradation due to error accumulation.
Neural network algorithms have some error-tolerant learning capabilities, and they can obtain some fixed pattern of input data, even computational errors. Therefore, retraining the network is an effective method to reduce the effect of various errors, such as Process Variations ("PV"), on the system without increasing the extra consumption of the inference channel. However, on-line retraining all the network layers is a very complex task, requires high-precision computation, and requires a large number of additional computation units to achieve convergence, resulting in excessive energy and resource consumption of the simulated NN processor, thereby greatly reducing the efficiency of the simulated NN processor.
Therefore, how to avoid the above-mentioned drawbacks, reduce the energy and resource consumption of the simulated NN processor, and improve the efficiency of the simulated NN processor becomes a problem to be solved.
Disclosure of Invention
Aiming at the problems in the prior art, the embodiment of the invention provides an error calibration method and device for an analog neural network processor.
In a first aspect, an embodiment of the present invention provides an error calibration method for an analog neural network processor, where the method includes:
if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure;
training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain;
replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations;
and storing the learned weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
In a second aspect, an embodiment of the present invention provides an error calibration apparatus for an analog neural network processor, the apparatus including:
an obtaining unit, configured to, if algorithm update and/or error parameter adjustment is detected, analyze the NN network structure to obtain trainable weight parameters of a full connection layer in the NN network structure;
the quantization unit is used for training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain;
the operation unit is used for replacing multiplication operations used by back propagation and trainable weight parameter updating in the learning process by adopting shift operation;
and the storage unit is used for storing the learned weight parameters so that the NN can calibrate the error of the processor according to the weight parameters.
In a third aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a bus, wherein,
the processor and the memory are communicated with each other through the bus;
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform a method comprising:
if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure;
training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain;
replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations;
and storing the trained weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, including:
the non-transitory computer readable storage medium stores computer instructions that cause the computer to perform a method comprising:
if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure;
training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain;
replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations;
and storing the learned weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
According to the error calibration method and device for the simulated neural network processor, provided by the embodiment of the invention, the logarithmic quantization loss value and the gradient are adopted in the process of training the weight parameters of the full connection layer, and the multiplication operation used by back propagation and trainable weight parameter updating is replaced by the shift operation after logarithmic quantization, so that the energy and resource consumption of the simulated NN processor can be reduced, and the efficiency of the simulated NN processor is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart illustrating an error calibration method for a simulated neural network processor according to an embodiment of the present invention;
FIG. 2(a) is a schematic diagram of a prior art backpropagation and trainable weight parameter update using multiplication;
FIG. 2(b) is a schematic diagram of an embodiment of the present invention employing a shift operation for back propagation and trainable weight parameter update;
FIG. 3 is a schematic diagram of an error calibration apparatus for a simulated neural network processor according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic flow chart of an error calibration method for a simulated neural network processor according to an embodiment of the present invention, and as shown in fig. 1, an error calibration method for a simulated neural network processor according to an embodiment of the present invention includes the following steps:
s101: and if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure.
Specifically, if the device detects algorithm update and/or error parameter adjustment, the device analyzes the NN network structure to obtain trainable weight parameters of a full connection layer in the NN network structure. An analog neural network processor refers to a processor that computes in the analog domain for neural network algorithms. The algorithm update may be understood as an update of the neural network NN, and the error parameter may be a PV parameter, but is not particularly limited. The network structure may include a convolutional layer, a pooling layer, a full connection, and other connection relationships, and it should be noted that: the trainable weight parameters in the embodiment of the present invention may be understood as weight parameters of a full connection layer, that is: the embodiment of the invention only trains the weight parameters of the full connection layer, and the weight parameters of each layer such as the convolution layer do not need to be trained.
S102: training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain.
Specifically, the device trains the trainable weight parameters by using a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain. Neuron values computed by the analog neural network processor may be converted from the analog domain to the digital domain by low-precision ADCs (i.e., ADCs below a preset precision threshold). Logarithmic quantization may present a wider data range than linear quantization when the same bit width is used. For example: the range of 4 bits is [ -128,128] based on 2 logarithmic quantization, and the range of [ -128,127] for integers of not less than 8 bits. Considering that the loss value and gradient always vary greatly and they can be approximated in a Stochastic Gradient Descent (SGD) algorithm, a logarithmic quantization method is used to quantize the loss value and gradient. Training the trainable weight parameters by using a random gradient descent SGD algorithm is a mature technique in the art and is not described in detail.
S103: a shift operation is used instead of the multiplication operations used for back propagation and trainable weight parameter updates in the learning process.
In particular, the device employs a shift operation instead of a multiplication operation used for back propagation and trainable weight parameter updates in the learning process. FIG. 2(a) is a schematic diagram of a prior art backpropagation and trainable weight parameter update using multiplication; fig. 2(b) is a schematic diagram of an embodiment of the present invention that employs a shift operation for inverse propagation and trainable weight parameter updating, and as shown in fig. 2(a) and 2(b), S102 may employ 2-based logarithmic quantization, and after 2-based logarithmic quantization is completed, multiplication operations in the inverse propagation and trainable weight updating stages may be replaced by a shift operation. Logarithmic quantization is less power consuming because it can be easily obtained by calculating where the highest bit (for positive data) or the lowest bit (for negative data) of the high level is located.
S104: and storing the trained weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
Specifically, the device stores the trained weight parameters, so that the NN can calibrate the error of the processor according to the weight parameters. It should be noted that: after learning is completed, the new parameters already contain the error information of the simulated neural network processor, and no additional error compensation is needed.
During the learning phase, on average, about 66.7% less memory is available compared to the prior art, as shown in table 1 (percentage of memory saved). This can relieve the storage pressure and reduce the data consumption of the embedded terminal.
TABLE 1
Neuron and its use&Loss of power Gradient of gradient Weight of Synthesis of
Prior Art 32bit 32bit 32bit 32bit
Examples of the invention 6bit 6bit 20bit 10.67bit
Saving memory 81.25％ 81.25％ 37.5％ 66.7％
After logarithmic quantization in the embodiment of the present invention, almost all multiplication operations in the learning process can be replaced by displacement operations, and if the learning process is implemented by using the existing FPGA or ASIC integration in the system, the power consumption will be significantly reduced, as shown in table 2 (energy consumption calculation). Wherein PDP (Power Delay product) is the product of power Delay. The combination of log quantization and shift operations can achieve approximately a 50-fold energy efficiency improvement over the prior art.
Table 2 (32 bit fixed point number is selected as the reference in the prior art; data in Table 2 is shown in
Obtained by simulation under the SMIC180nm process
Operations Time delay (ns) Power (mw) PDP(pJ)
Prior Art 32bit multiplication 6.53 15.7 102.52
Examples of the invention 32bit shift 1.62 1.22 1.97
According to the error calibration method for the simulated neural network processor, provided by the embodiment of the invention, the logarithmic quantization loss value and the gradient are adopted in the process of training the weight parameters of the full connection layer, and the multiplication operation used by back propagation and trainable weight parameter updating is replaced by the shift operation after logarithmic quantization, so that the energy and resource consumption of the simulated NN processor can be reduced, and the efficiency of the simulated NN processor is improved.
On the basis of the above embodiment, the method further includes:
storing the trainable weight parameters and the weight parameters with a resolution greater than a preset precision.
In particular, the device stores said trainable weight parameters and said weight parameters with a resolution higher than a preset precision. The specific numerical value of the preset precision can be set independently according to the actual condition. Non-trainable weights (e.g., convolutional layers) are saved at low resolution (i.e., resolution lower than a preset precision), which may be bit wide; whereas trainable weights (e.g., fully connected layers) are maintained at high resolution (i.e., resolution higher than a preset precision), which may be bit wide. The high resolution trainable weights are only used to update the weights in the learning phase (learning phase) to ensure convergence.
According to the error calibration method for the simulated neural network processor, provided by the embodiment of the invention, the trainable weight parameters and the weight parameters are stored by adopting the resolution which is higher than the preset precision, so that the convergence of the algorithm can be ensured.
On the basis of the above embodiment, the method further includes:
and acquiring the weight parameters in the convolution layer in the network structure, and storing the weight parameters in the convolution layer with a resolution lower than a preset precision.
Specifically, the device acquires a weight parameter in a convolutional layer in the network structure, and stores the weight parameter in the convolutional layer at a resolution lower than a preset precision. Reference may be made to the above embodiments, which are not described in detail.
According to the error calibration method for the simulated neural network processor, provided by the embodiment of the invention, the weight parameters in the convolutional layer are stored by adopting the resolution which is lower than the preset precision, so that the energy and resource consumption of the simulated NN processor can be reduced, and the efficiency of the simulated NN processor is improved.
On the basis of the above embodiment, the use phase of the NN includes an inference phase and a learning phase; correspondingly, the method further comprises the following steps:
after the step of storing the learned weight parameters, the learning phase is frozen and the reasoning phase is activated.
Specifically, after the step of storing the learned weight parameters, the device freezes the learning phase and activates the reasoning phase. That is, in most cases, only the inference phase is active, so that only a small amount of additional energy consumption is required.
According to the error calibration method for the simulated neural network processor, provided by the embodiment of the invention, after the step of storing the learned weight parameters, the learning stage is frozen, so that the energy and resource consumption of the simulated NN processor can be further reduced, and the efficiency of the simulated NN processor is improved.
On the basis of the above embodiment, the method further includes:
if an algorithm update and/or an error parameter adjustment is detected, the inference phase is frozen and the method described above is performed to activate the learning phase.
Specifically, the device freezes the inference phase if it detects an algorithm update and/or an error parameter adjustment, and executes the method described above to activate the learning phase. That is, in a few cases, only the learning phase is active, so only a small amount of additional energy consumption is required.
According to the error calibration method for the simulated neural network processor, provided by the embodiment of the invention, the energy and resource consumption of the simulated NN processor can be further reduced by freezing the inference stage if the algorithm update and/or the error parameter adjustment are detected, so that the efficiency of the simulated NN processor is improved.
On the basis of the above embodiment, the method further includes:
the reverse phase of the NN is executed in a processor of the terminal except the processor; wherein the terminal is loaded with the processor and the other processors.
Specifically, the reverse phase of the NN in the device is executed in a processor other than the processor of the terminal; wherein the terminal is loaded with the processor and the other processors. The terminal may be a PC or the like, included within the range of the device. The other processors can comprise a CPU or an FPGA and the like, and the steps of weight updating and the like are transferred to other processors for processing, so that the chip area and the energy consumption of the simulation NN processor are further reduced.
According to the error calibration method for the simulated neural network processor provided by the embodiment of the invention, the steps of updating the weight and the like are transferred to other processors for processing, so that the energy and resource consumption of the simulated NN processor can be further reduced, and the efficiency of the simulated NN processor is improved.
On the basis of the above embodiment, the other processors include a CPU or an FPGA.
In particular, the other processors in the device include a CPU or FPGA. Reference may be made to the above embodiments, which are not described in detail.
According to the error calibration method for the analog neural network processor provided by the embodiment of the invention, other processors are selected as CPUs or FPGAs, so that the other processors can be ensured to normally process the steps of weight updating and the like.
Fig. 3 is a schematic structural diagram of an error calibration apparatus for an analog neural network processor according to an embodiment of the present invention, and as shown in fig. 3, an embodiment of the present invention provides an error calibration apparatus for an analog neural network processor, including an obtaining unit 301, a quantizing unit 302, and an arithmetic unit 303, and a storage unit 304, where:
the obtaining unit 301 is configured to, if algorithm update and/or error parameter adjustment is detected, analyze the network structure of the NN to obtain trainable weight parameters of a full connection layer in the network structure; the quantization unit 302 is configured to train the trainable weight parameters by using a random gradient descent SGD algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain; a replacement unit 303 for replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with a shift operation; the storage unit 304 is configured to store the learned weight parameters, so that the NN may calibrate an error of the processor according to the weight parameters.
Specifically, the obtaining unit 301 is configured to, if it is detected that the algorithm is updated and/or the error parameter is adjusted, analyze the network structure of the NN to obtain trainable weight parameters of a full connection layer in the network structure; the quantization unit 302 is configured to train the trainable weight parameters by using a random gradient descent SGD algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain; a replacement unit 303 for replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with a shift operation; the storage unit 304 is configured to store the learned weight parameters, so that the NN may calibrate an error of the processor according to the weight parameters.
According to the error calibration device of the simulated neural network processor, provided by the embodiment of the invention, the logarithmic quantization loss value and gradient are adopted in the process of training the weight parameters of the full connection layer, and the multiplication operation used by inverse propagation and trainable weight parameter updating is replaced by the shift operation after logarithmic quantization, so that the energy and resource consumption of the simulated NN processor can be reduced, and the efficiency of the simulated NN processor is improved.
The error calibration apparatus for an analog neural network processor provided in the embodiments of the present invention may be specifically configured to execute the processing procedures of the above-described method embodiments, and its functions are not described herein again, and refer to the detailed description of the above-described method embodiments.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 4, the electronic device includes: a processor (processor)401, a memory (memory)402, and a bus 403;
the processor 401 and the memory 402 complete communication with each other through a bus 403;
the processor 401 is configured to call the program instructions in the memory 402 to execute the methods provided by the above-mentioned method embodiments, for example, including: if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure; training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain; replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations; and storing the trained weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
The present embodiment discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the method provided by the above method embodiments, for example, including: if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure; training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain; replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations; and storing the learned weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
The present embodiments provide a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the methods provided by the above method embodiments, for example, including: if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure; training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain; replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations; and storing the learned weight parameters for the NN to calibrate the error of the processor according to the weight parameters.
Those of ordinary skill in the art will understand that: all or part of the steps of implementing the method embodiments may be implemented by hardware related to program instructions, and the program may be stored in a computer-readable storage medium, and when executed, executes the steps including the method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
The above-described embodiments of the electronic device and the like are merely illustrative, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may also be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the embodiments of the present invention, and are not limited thereto; although embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.

Claims (6)

1. A method of error calibration for an analog neural network processor, the method being performed in an analog neural network, NN, processor, comprising:
if algorithm updating and/or error parameter adjustment are detected, analyzing the NN network structure to obtain trainable weight parameters of a full connection layer in the network structure;
training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain;
replacing the multiplication operations used for back propagation and trainable weight parameter updating in the learning process with shift operations;
storing the learned weight parameters for the NN to calibrate the error of the processor according to the weight parameters;
storing the trainable weight parameters and the weight parameters with a resolution higher than a preset precision;
acquiring weight parameters in a convolutional layer in the network structure, and storing the weight parameters in the convolutional layer with a resolution lower than a preset precision;
the NN using phase comprises an inference phase and a learning phase; correspondingly, the method further comprises the following steps:
after the step of storing the learned weight parameters, freezing the learning phase and activating the reasoning phase;
the method further comprises the following steps:
if an algorithm update and/or an error parameter adjustment is detected, freezing the reasoning phase and performing the above step of calibrating the error of the processor to activate the learning phase.
2. The method of claim 1, further comprising:
the reverse phase of the NN is executed in a processor of the terminal except the processor; wherein the terminal is loaded with the processor and the other processors.
3. The method of claim 2, wherein the other processor comprises a CPU or FPGA.
4. An error calibration apparatus for an analog neural network processor, the apparatus comprising an analog Neural Network (NN) processor, the apparatus comprising:
an obtaining unit, configured to, if algorithm update and/or error parameter adjustment is detected, analyze the NN network structure to obtain trainable weight parameters of a full connection layer in the NN network structure;
the quantization unit is used for training the trainable weight parameters by adopting a random gradient descent (SGD) algorithm; wherein, the loss value and the gradient in the learning process adopt logarithmic quantification; the learning process is performed in the digital domain;
the operation unit is used for replacing multiplication operations used by back propagation and trainable weight parameter updating in the learning process by adopting shift operation;
the storage unit is used for storing the learned weight parameters so that the NN can calibrate the error of the processor according to the weight parameters; storing the trainable weight parameters and the weight parameters with a resolution higher than a preset precision; acquiring weight parameters in a convolutional layer in the network structure, and storing the weight parameters in the convolutional layer with a resolution lower than a preset precision;
the NN using phase comprises an inference phase and a learning phase; correspondingly, the device is also used for:
after the step of storing the learned weight parameters, freezing the learning phase and activating the reasoning phase; and if the algorithm updating and/or the error parameter adjustment are detected, freezing the reasoning stage, and executing the units to activate the learning stage.
5. An electronic device, comprising: a processor, a memory, and a bus, wherein,
the processor and the memory are communicated with each other through the bus;
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of any of claims 1 to 3.
6. A non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the method of any one of claims 1 to 3.
CN201810580960.8A 2018-06-07 2018-06-07 Error calibration method and device for analog neural network processor Active CN110580523B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201810580960.8A CN110580523B (en) 2018-06-07 2018-06-07 Error calibration method and device for analog neural network processor
PCT/CN2018/104781 WO2019232965A1 (en) 2018-06-07 2018-09-10 Error calibration method and device for analog neural network processor

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810580960.8A CN110580523B (en) 2018-06-07 2018-06-07 Error calibration method and device for analog neural network processor

Publications (2)

Publication Number Publication Date
CN110580523A CN110580523A (en) 2019-12-17
CN110580523B true CN110580523B (en) 2022-08-02

Family

ID=68769907

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810580960.8A Active CN110580523B (en) 2018-06-07 2018-06-07 Error calibration method and device for analog neural network processor

Country Status (2)

Country Link
CN (1) CN110580523B (en)
WO (1) WO2019232965A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113705776B (en) * 2021-08-06 2023-08-08 山东云海国创云计算装备产业创新中心有限公司 Method, system, equipment and storage medium for realizing activation function based on ASIC

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104899641A (en) * 2015-05-25 2015-09-09 杭州朗和科技有限公司 Deep neural network learning method, processor and deep neural network learning system
CN107665364A (en) * 2016-07-28 2018-02-06 三星电子株式会社 Neural net method and equipment
CN108009640A (en) * 2017-12-25 2018-05-08 清华大学 The training device and its training method of neutral net based on memristor

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8352400B2 (en) * 1991-12-23 2013-01-08 Hoffberg Steven M Adaptive pattern recognition based controller apparatus and method and human-factored interface therefore
US7409372B2 (en) * 2003-06-20 2008-08-05 Hewlett-Packard Development Company, L.P. Neural network trained with spatial errors
CN105279554B (en) * 2015-09-29 2017-12-15 东方网力科技股份有限公司 The training method and device of deep neural network based on Hash coding layer
EP3497624A1 (en) * 2016-08-13 2019-06-19 Intel Corporation Apparatuses, methods, and systems for neural networks
CN106355248A (en) * 2016-08-26 2017-01-25 深圳先进技术研究院 Deep convolution neural network training method and device
US10255910B2 (en) * 2016-09-16 2019-04-09 Apptek, Inc. Centered, left- and right-shifted deep neural networks and their combinations
CN107092960A (en) * 2017-04-17 2017-08-25 中国民航大学 A kind of improved parallel channel convolutional neural networks training method
CN107395211B (en) * 2017-09-12 2020-12-01 苏州浪潮智能科技有限公司 Data processing method and device based on convolutional neural network model
CN107944458A (en) * 2017-12-08 2018-04-20 北京维大成科技有限公司 A kind of image-recognizing method and device based on convolutional neural networks
CN107992842B (en) * 2017-12-13 2020-08-11 深圳励飞科技有限公司 Living body detection method, computer device, and computer-readable storage medium

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104899641A (en) * 2015-05-25 2015-09-09 杭州朗和科技有限公司 Deep neural network learning method, processor and deep neural network learning system
CN107665364A (en) * 2016-07-28 2018-02-06 三星电子株式会社 Neural net method and equipment
CN108009640A (en) * 2017-12-25 2018-05-08 清华大学 The training device and its training method of neutral net based on memristor

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Analog in-memory subthreshold deep neural network accelerator;L. Fick 等;《2017 IEEE Custom Integrated Circuits Conference (CICC)》;20170727;1-4 *

Also Published As

Publication number Publication date
CN110580523A (en) 2019-12-17
WO2019232965A1 (en) 2019-12-12

Similar Documents

Publication Publication Date Title
KR20220086694A (en) Memristor-based neural network training method and training device therefor
CN111353579A (en) Method and system for selecting quantization parameters for a deep neural network using back propagation
US20240046086A1 (en) Quantization method and quantization apparatus for weight of neural network, and storage medium
CN113642711B (en) Processing method, device, equipment and storage medium of network model
CN110580523B (en) Error calibration method and device for analog neural network processor
CN113887845A (en) Extreme event prediction method, apparatus, device, medium, and computer program product
US20190377548A1 (en) Arithmetic processing apparatus, control method, and recording medium
CN109102067B (en) Method for increasing and decreasing neural network nodes, computer device and storage medium
CN112016702A (en) Medical data processing method, device, equipment and medium based on transfer learning
US20220405561A1 (en) Electronic device and controlling method of electronic device
CN111401569A (en) Hyper-parameter optimization method and device and electronic equipment
CN115577491A (en) Parameter correction method and device, electronic equipment and storage medium
CN115392441A (en) Method, apparatus, device and medium for on-chip adaptation of quantized neural network model
EP3742283A1 (en) Arithmetic processing device, method for controlling arithmetic processing device, and program for controlling arithmetic processing device
CN111476356B (en) Memristive neural network training method, device, equipment and storage medium
US20220036190A1 (en) Neural network compression device
TWI732467B (en) Method of training sparse connected neural network
CN110929849A (en) Neural network model compression method and device
CN110852361B (en) Image classification method and device based on improved deep neural network and electronic equipment
CN109981327B (en) Method and system for predicting traffic
CN111092602A (en) Modeling method and device of power amplifier, computer equipment and storage medium
US20230008014A1 (en) Data processing device, data-processing method and recording media
CN116432735A (en) Data processing method and device and edge computing equipment
CN116702861B (en) Compression method, training method, processing method and device of deep learning model
US20210081783A1 (en) Information processing apparatus, method of processing information, and non-transitory computer-readable storage medium for storing information processing program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant