CN112712170B - Neuromorphic visual target classification system based on input weighted impulse neural network - Google Patents
Neuromorphic visual target classification system based on input weighted impulse neural network Download PDFInfo
- Publication number
- CN112712170B CN112712170B CN202110025992.3A CN202110025992A CN112712170B CN 112712170 B CN112712170 B CN 112712170B CN 202110025992 A CN202110025992 A CN 202110025992A CN 112712170 B CN112712170 B CN 112712170B
- Authority
- CN
- China
- Prior art keywords
- event
- pulse
- module
- input
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 55
- 230000000007 visual effect Effects 0.000 title claims abstract description 20
- 238000007781 pre-processing Methods 0.000 claims abstract description 30
- 238000010276 construction Methods 0.000 claims abstract description 21
- 238000000034 method Methods 0.000 claims abstract description 18
- 230000001537 neural effect Effects 0.000 claims abstract description 10
- 210000002569 neuron Anatomy 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 21
- 230000002776 aggregation Effects 0.000 claims description 12
- 238000004220 aggregation Methods 0.000 claims description 12
- 238000003062 neural network model Methods 0.000 claims description 12
- 230000002123 temporal effect Effects 0.000 claims description 9
- 238000009825 accumulation Methods 0.000 claims description 7
- 230000008859 change Effects 0.000 claims description 6
- 230000006835 compression Effects 0.000 claims description 6
- 238000007906 compression Methods 0.000 claims description 6
- 238000001208 nuclear magnetic resonance pulse sequence Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 5
- 238000011176 pooling Methods 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 4
- 238000006116 polymerization reaction Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 abstract 1
- 210000005036 nerve Anatomy 0.000 description 7
- 238000013459 approach Methods 0.000 description 5
- 230000004931 aggregating effect Effects 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 239000012528 membrane Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000012421 spiking Methods 0.000 description 1
- 210000000225 synapse Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a neuromorphic visual target classification system based on an input weighted impulse neural network, which belongs to the technical field of artificial neural networks and comprises the following four method modules: the system comprises a data preprocessing module, a network construction module, a learning module and an reasoning module. The data preprocessing module is used for converting the acquired event camera asynchronous output space-time pulse event stream into an event frame sequence; the network construction module is used for constructing an input weighted impulse neural network by the input weighted unit and the impulse neural layer unit according to a certain network connection mode; the learning module is used for learning the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generating a model file; the reasoning module is used for loading the network model file output by the learning module to perform feedforward network calculation. The invention can ensure that the neuromorphic vision classification impulse neural network has low time delay and simultaneously keeps higher performance.
Description
Technical Field
The invention belongs to the field of deep learning in machine learning, and particularly relates to a neuromorphic visual target classification system based on an input weighted impulse neural network.
Background
An event camera is an asynchronous neuromorphic vision sensor that produces a paradigm shift in the way visual information is acquired. Unlike traditional vision sensor that samples light at fixed moment, the event camera samples light according to scene dynamics, produces pulse event stream through the luminance change of asynchronous measurement every pixel, and pulse event stream encodes the time, position and the luminance change polarity of luminance change. The impulse neural network (Spiking Neural Networks, SNNs) is a new generation artificial neural network inspired by brain operation mechanism, takes an impulse sequence as a data transmission form, and has the advantages of ultra-low time delay, low energy consumption and the like compared with the traditional artificial neural network (Artificial Neural Networks, ANNs). The event stream output with high time resolution (microsecond level) of the event camera has great potential in some computer vision application scenes with requirements on efficiency and power consumption by combining with a pulse neural network with ultra-low time delay.
In theory, a pulse Event stream output by the neuromorphic vision sensor is processed by Event-by-Event, and the input of each Event causes the change of the internal state of the pulse neural network, so that the output with the minimum time delay can be obtained. However, the output at the lowest latency tends to be poor due to the very small amount of information contained in a single event. Another approach is to aggregate the stream of pulsed events over a period of time in some way into a new Event Frame (Event Frame) that contains much more information than a single Event, which allows the network to achieve better performance, but will introduce some delay. Thus, new neuromorphic vision algorithms need to be researched and developed to achieve a balance between low latency and high performance.
Target recognition and classification are an important task for neuromorphic vision. The pulse frequency coding-based neuromorphic visual target classification supervised learning method mainly comprises two steps:
(1) Conversion-based methods: the trained deep neural network (such as convolutional neural network and fully connected neural network) is converted into a pulse neural network by using a weight adjustment and normalization method, and the pulse neural network is applied to the classification of the neuromorphic visual target. This approach can achieve accuracy comparable to deep neural networks, but has inherent limitations, such as limited use of activation functions and not utilization of time information in the event stream.
(2) Pulse-based methods: the learning of the weights of synapses is done by back-propagation of training errors in Time and space, a representative algorithm being the back-propagation-Through-Time (BPTT) based on Time. This approach is computationally more efficient, but the accuracy in neuromorphic visual target classification cannot exceed the transform-based approach.
The main difficulty in solving the task of classifying and identifying the target of the neuromorphic vision is how to effectively extract the space-time characteristics from the pulse event stream space-time information to obtain higher classification accuracy. In the prior art, the impulse neural network based on the LIF neuron model uses a time-space error back propagation learning algorithm to update the weights, so that researchers can use a GPU to perform accelerated training and can also use training tools such as Pytorch which are very mature in deep learning. However, this approach does not distinguish between inputs to the network, which can affect the performance of the network to some extent. In fact, inputs at different times contain different signal-to-noise ratios, giving the same input weights to all input times weakens the ability of the network to extract valid spatiotemporal features.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides a neuromorphic vision target classification system based on an input weighted pulse neural network, which is characterized in that pulse event stream data are firstly aggregated into event frame sequence data, then an input weight is provided for each event frame according to the difference of information quantity contained in the event frames, and finally the weighted event frames are used as new inputs of the pulse neural network, so that the pulse neural network can adaptively pay attention to the event frames with great influence on classification results in the event frame sequence, and the accuracy of target classification and recognition tasks of neuromorphic vision can be improved while only a small amount of data is needed.
In order to achieve the above purpose, the present invention provides the following technical solutions:
the neuromorphic visual target classification system based on the input weighted impulse neural network comprises a data preprocessing module, a network construction module, a learning module and an reasoning module which are connected in sequence;
the data preprocessing module is used for acquiring a space-time pulse event stream asynchronously output by the event camera, wherein space-time pulse events in the space-time pulse event stream are described by adopting an address event expression protocol; the time-space pulse event stream is used for converging the time-space pulse event stream into an event frame sequence according to the time resolution dt' of an event camera, and the event frame sequence is described by tensors; according to the set time resolution dt, aggregating the event frame sequence with the time resolution dt' into a new event frame sequence, wherein the event frame data is described by tensors; the event frame sequence tensor data is used as the output of a data preprocessing module;
the network construction module is composed of an input weighting unit and a pulse neural layer unit and is used for constructing an input weighting pulse neural network;
the learning module learns the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generates a model file;
the inference module reads the input weighted impulse neural network structure configured by the network construction module, loads the model file generated by the learning module to obtain input weighted impulse neural network parameters, obtains a trained input weighted impulse neural network model, and takes a plurality of event frames output by the data preprocessing module as the input of the input weighted impulse neural network model to obtain an inference result.
The invention further improves that the data preprocessing module aggregates the pulse event stream into an event frame sequence according to the time-space pulse event stream output by the event camera and the set time resolution dt, and the method specifically comprises the following steps:
the stream of spatiotemporal pulse events is represented by the set e= { E i |e i ＝[x i ,y i ,t′ i ,p i ]Determining; wherein e i For the ith pulse event in the pulse event stream, (x) i ,y i ) Pixel coordinates, t 'for the ith pulse event' i For the time stamp of the ith pulse event in the whole time stream, p i The polarity of the light intensity change for the ith pulse event; the time resolution of the asynchronous pulse transmission event stream of the event camera is dt', and the spatial resolution is H multiplied by W; then, new spatiotemporal event frame data are aggregated according to the set temporal resolution dt.
The invention is further improved in that the data aggregation process is performed in two steps, specifically comprising:
first, based on the time resolution dt' of the event camera, generating a plurality of events E at the time t t′ Assembled into tensor X t′； wherein ,Et′ ＝{e i |e i ＝[x i ,y i ,t′,p i ]}，X t′ ∈R H×W×2 ；
Second, based on the set time resolutionRate dt, using formula X t ＝f(X′ t ) Generating event frame tensor X at time t t ∈ R H×W×2 The method comprises the steps of carrying out a first treatment on the surface of the Where dt=β×dt', β is a polymerization time factor; x'. t ＝{X t′ |t′∈[β×t,β×(t+1)-1]-a }; f is an accumulation operation, a weighted accumulation operation, or an and or operation.
The invention is further improved in that the network construction module consists of one or more input weighting units, a pulse neural layer unit and one or more perceptron neuron output layers; the impulse neuron adopts an LIF neuron model; the input weighting unit is composed of three steps, and specifically comprises:
first, an event frame X (x= { X) obtained by the data preprocessing module is processed 1 ,X 2 ,...,X t ,...,X T },X∈R H ×W×2×T ) As input, it is compressed into a vector z (z= { z) using a compression function f 1 ,z 2 ,...,z t ,...z T },z∈R T ) The method comprises the steps of carrying out a first treatment on the surface of the Wherein, when the compression function f is an average pooling function, the t-th tensor X in X t Is compressed to a value z t The specific formula is as follows:
the second step, the vector z is input into a two-layer nonlinear fully connected network to obtain an output vector s (s E R T )：
s＝σ(W 2 δ(W 1 z))
Where delta is the ReLU activation function,
weight matrix W 1 and W2 R is an optional super parameter, which is a trainable parameter;
third, the value s in the vector s is calculated t As event frame X t Multiplying the weight of each element in the event frame to obtain a new event frame
As an output of the input weighting unit:
the invention further improves that the learning module comprises a feedforward network computing unit, an error back propagation unit and a weight updating unit, and specifically comprises:
the feedforward network computing unit selects T event frames from an event frame sequence converted from single event stream data by using a random time clipping method to be input by the feedforward network computing unit, wherein the T event frames are described by tensors; specifically, in single spatio-temporal pulse event stream data, T is generated at a temporal resolution dt total From which T event frames are randomly extracted as inputs to a feed forward network computation unit, T<T total ；
Calculating an output pulse sequence according to the input weighting unit, the pulse neural layer unit and the network connection mode in sequence, and then calculating an output target according to the output pulse sequence by the perceptron neuron output layer;
the error back propagation unit calculates an error between the output target and the set target according to the set loss function and performs back propagation;
the weight updating unit updates the weight according to the set learning rate and the error.
The invention is further improved in that the reasoning module consists of a model loading unit and a feedforward network calculating unit; the model loading unit reads the input weighted impulse neural network structure configured by the network construction module, loads the model file generated by the learning module and obtains a trained input weighted impulse neural network model; the feedforward network computing unit computes an input weighting and a pulse neuron layer in the pulse neural network model according to an event stream or an event frame provided by the data preprocessing module as input, and then an output target is obtained by the perceptron neuron output layer according to output pulses of the pulse neuron layer.
Compared with the prior art, the invention has at least the following beneficial technical effects:
aiming at the problems of high classification delay and low accuracy of the pulse neural network in the conventional nerve morphology visual target classification method, the invention improves the structure of the conventional pulse neural network, namely, an event frame with great influence on the nerve morphology visual classification result is adaptively given with higher weight by introducing an input weighting unit, so that the space-time characteristic extraction capability of pulse event streams is improved, the required data quantity is reduced, and the nerve morphology visual classification accuracy of the pulse neural network is effectively improved.
Drawings
FIG. 1 is a schematic diagram of a neuromorphic visual target classification system based on an input weighted impulse neural network in accordance with the present invention.
Fig. 2 shows an output of an event stream data (waving motion) in the neuromorphic vision dataset DVS128 ges after passing through the data preprocessing module. That is, a stream of 200,000 microsecond pulsed events is aggregated into 10 event frames with 20 ms resolution.
FIG. 3 is a network connection of an input weighted impulse neural network.
Fig. 4 is a schematic diagram of an input weighting unit.
Fig. 5 is a schematic diagram of a unit of a pulse nerve layer.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the neuromorphic visual target classification system based on the weighted impulse neural network provided by the invention is composed of a data preprocessing module S101, a network construction module S102, a learning module S103 and an reasoning module S104 which are sequentially connected.
The data preprocessing module S101 is configured by a pulse event stream aggregation unit and an event frame aggregation unit, and is configured to aggregate a spatio-temporal pulse event stream asynchronously output by an event camera into an event frame sequence.
The input of the data preprocessing module is a space-time pulse event stream output by an event camera, and the space-time pulse event is described by adopting an address event expression protocol; the pulse event stream aggregation unit is used for aggregating the pulse event streams into an event frame sequence according to the time resolution dt' of the event camera; the event frame aggregation unit is used for aggregating the event frame sequence with the resolution ratio dt' into new event frame sequence data according to the set time resolution ratio dt, and the event frame data are described by tensors; the event frame sequence output by the event frame aggregation unit is used as the output of the data preprocessing module.
The spatiotemporal pulse event stream is represented by the set e= { E i |e i ＝[x i ,y i ,t′ i ,p i ]Determining; wherein e i For the ith pulse event in the pulse event stream, (x) i ,y i ) Pixel coordinates, t 'for the ith pulse event' i For the time stamp of the ith pulse event in the whole time stream, p i The light intensity for the ith pulse event changes polarity. The time resolution of the asynchronous pulse transmission event stream of the event camera is dt', and the spatial resolution is H multiplied by W. Typically, the temporal resolution of an event camera is in the order of microseconds, i.e., dt' =1 μs.
The pulse event stream aggregation unit is used for aggregating the pulse event streams into an event frame sequence according to the time resolution dt' of the event camera, and specifically comprises the following steps: based on the time resolution dt' of the event camera, generating a plurality of events E at the moment t t′ Assembled into tensor X t′. wherein ,Et′ ＝{e i |e i ＝[x i ,y i ,t′,p i ]}，X t′ ∈R H×W×2 。
The event frame aggregation unit is used for setting a sequence of event frames with the resolution of dt' =1μsThe time resolution dt is aggregated into new event frame sequence data, which specifically comprises: based on the set time resolution dt, formula X is used t ＝f(X′ t ) Generating event frame tensor X at time t t ∈R H×W×2 . Where dt=β×dt', β is a polymerization time factor; x'. t ＝{X t′ |t′∈[β×t,β× (t+1)-1]-a }; f may be an accumulation operation, a weighted accumulation operation, or an and or operation.
FIG. 2 illustrates one data (waving action) in the data set DVS128 Gestme, the output of the data preprocessing module when the input is a pulse event stream of 200,000 microseconds, the aggregation method is an OR operation, and the aggregation time factor is 20,000. That is, 10 event frames with 20 ms resolution are shown in FIG. 2, aggregated from event streams.
When dt' =1 μs, β=3, one example of an "or" polymerization method is as follows:
when dt' =1μs, β=3, one example of the accumulation operation is as follows:
the network construction module S102 is composed of an input weighting unit and a impulse neural layer unit, and is configured to configure an input weighted impulse neural network. The network construction module consists of one or more input weighting units and pulse nerve layer units (pulse nerve layers are formed by fully connected or convolved pulse nerve cells), and one or more perceptron nerve element output layers; the impulse neuron adopts an LIF neuron model. Fig. 3 illustrates a network connection formed by an input weighting unit and a pulsed neuron layer unit.
The input weighting unit is shown in fig. 4. The unit consists of three steps, and specifically comprises:
first step, willEvent frame X (x= { X) obtained by data preprocessing module 1 ,X 2 ,...,X t ,...,X T },X∈R H ×W×2×T ) As input, it is compressed into a vector z (z= { z) using a compression function f 1 ,z 2 ,...,z t ,...z T },z∈R T ). The compression function f may be an average pooling function, a maximum pooling function, etc. When f is the average pooling function, the t-th tensor X in X t Is compressed to a value z according to the following formula t ：
The second step, the vector z is input into a two-layer nonlinear fully connected network to obtain an output vector s (s E R T )：
s＝σ(W 2 δ(W 1 z))
Where delta is the ReLU activation function, sigma is the Sigmoid function,
is a trainable weight matrix, +.>
Is a trainable weight matrix, r is an optional parameter.
Third, the value s in the vector s is calculated t As event frame X t Multiplying the weight of each element in the event frame to obtain a new event frame
As an output of the input weighting unit:
the impulse neural layer unit is composed of LIF neuron model as shown in fig. 5. Specifically, the LIF neuron layer expression is as follows:
wherein, the function g is a step function,
representing Hadamard product, x t,n-1 An event frame input for the n-1 th layer at the t moment, h t-1,n The internal state quantity of the nth layer at the time t-1; u (u) t,n Is a membrane potential; membrane potential and neuron threshold u th Compare and compare x t,n Pass on to the next layer as spatial output, h t,n As the time output of the LIF neuron is passed on to the next moment. The weight matrix->
And spatial input x t,n-1 When matrix multiplication is adopted, the LIF neuron model is a fully connected LIF neuron model; the weight matrix->
And spatial input x t,n-1 When convolution operation is adopted, the LIF neuron model is a convolution LIF neuron model.
The learning module S103 is configured by a feedforward network computing unit, an error back propagation unit, and a weight updating unit, and is configured to learn the input weighted impulse neural network obtained by the network building module according to the event frame sequence obtained by the preprocessing module, and generate a model file.
The feedforward network computing unit selects T event frames from the event frame sequence converted from single event stream data by using a random time clipping method as input of the feedforward network computing unit, wherein the T event frames are selected from the event frame sequenceFrames are described using tensors; specifically, in single spatio-temporal pulse event stream data, T is generated at a temporal resolution dt total From which T (T< T total ) The event frames are used as inputs of a feedforward network computing unit; calculating an output pulse sequence according to the input weighting unit, the pulse neural layer unit and the network connection mode in sequence, and then calculating an output target according to the output pulse sequence by the perceptron neuron output layer;
the error back propagation unit calculates an error between the output target and the set target according to the set loss function, and performs back propagation.
The weight updating unit updates the weight according to the set learning rate and the error.
The reasoning module S104 is composed of a model loading unit and a feed-forward network computing unit. The method comprises the steps of reading an input weighted impulse neural network structure configured by a network construction module, loading a model file generated by a learning module to obtain input weighted impulse neural network parameters, obtaining a trained input weighted impulse neural network model, and taking an event frame output by a data preprocessing module as input of the input weighted impulse neural network model to obtain an inference result.
The model loading unit reads the impulse neural network structure configured by the network construction module, loads the model file generated by the learning module and obtains a trained input weighted impulse neural network model.
The feedforward network computing unit computes an input weighting and a pulse neuron layer in the pulse neural network model according to the event stream provided by the data preprocessing module as input, and then an output target is obtained by the perceptron neuron output layer according to the output pulse of the pulse neuron layer.
The data preprocessing module, the network construction module, the learning module and the reasoning module are sequentially connected.
To better illustrate the beneficial effects of the present invention, experiments of the method of the present invention on a neuromorphic visual-target-classification dataset DVS128 testure are presented below. In experiments we set 60 event frames as inputs and tested the influence of the input weighting unit on the final result on the basis of different time resolutions dt:
convolution-based input weighted impulse neural network experimental result on DVS128 Gestm
From the above table, it can be seen that, when the neuromorphic visual target classification is performed, the input weighted impulse neural network can achieve higher classification performance while only requiring a small amount of data, compared with the conventional impulse neural network. For example, when the time resolution is 1 millisecond, only 60 milliseconds of data is needed to achieve 91.28% accuracy. In all time resolutions, the input weighted impulse neural network can improve the network performance.
Claims (5)
1. The neuromorphic visual target classification system based on the input weighted impulse neural network is characterized by comprising a data preprocessing module, a network construction module, a learning module and an reasoning module which are connected in sequence;
the data preprocessing module is used for acquiring a space-time pulse event stream asynchronously output by the event camera, wherein space-time pulse events in the space-time pulse event stream are described by adopting an address event expression protocol; for adapting the stream of spatio-temporal pulse events to the temporal resolution dt of the event camera ′ Aggregation into an event frame sequence, wherein the event frame sequence is described by tensors; according to the set time resolution dt, the time resolution is dt ′ The event frame sequences of the event frames are aggregated into a new event frame sequence, and the event frame data is described by tensors; the event frame sequence tensor data is used as the output of a data preprocessing module;
the network construction module is composed of an input weighting unit and a pulse neural layer unit and is used for constructing an input weighting pulse neural network; the network construction module consists of one or more input weighting units, a pulse neural layer unit and one or more perceptron neuron output layers; the impulse neuron adopts an LIF neuron model; the input weighting unit is composed of three steps, and specifically comprises:
firstly, taking an event frame X obtained by a data preprocessing module as input, and compressing the event frame X into a vector z by utilizing a compression function f; wherein x= { X 1 ,X 2 ,…,X t ,…,X T ' X.epsilon.R H×W×2×T ，z＝{z 1 ,z 2 ,…,z t ,…z T Z e R T When the compression function f is an average pooling function, the t-th tensor X in X t Is compressed to a value z t The specific formula is as follows:
secondly, inputting the vector z into a two-layer nonlinear fully-connected network to obtain an output vector s:
s＝σ(W 2 δ(W 1 z))
wherein ,s∈RT Delta is the function of the ReLU activation,
weight matrix W 1 and W2 R is an optional super parameter, which is a trainable parameter;
third, the value s in the vector s is calculated t As event frame X t Multiplying the weight of each element in the event frame to obtain a new event frame
As an output of the input weighting unit:
wherein ,
the learning module learns the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generates a model file;
the inference module reads the input weighted impulse neural network structure configured by the network construction module, loads the model file generated by the learning module to obtain input weighted impulse neural network parameters, obtains a trained input weighted impulse neural network model, and takes a plurality of event frames output by the data preprocessing module as the input of the input weighted impulse neural network model to obtain an inference result.
2. The neuromorphic visual target classification system of claim 1 wherein the data preprocessing module aggregates the pulse event streams into a sequence of event frames according to a set temporal resolution dt from the spatio-temporal pulse event streams output by the event camera, comprising:
the stream of spatiotemporal pulse events is represented by the set e= { E i |e i ＝[x i ,y i ,t i ′ ,p i ]Determining; wherein e i For the ith pulse event in the pulse event stream, (x) i ,y i ) Pixel coordinates, t, for the ith pulse event i ′ For the time stamp of the ith pulse event in the whole time stream, p i The polarity of the light intensity change for the ith pulse event; the time resolution of asynchronous pulse-delivering event stream of event camera is dt ′ The spatial resolution is H×W; then, new spatiotemporal event frame data are aggregated according to the set temporal resolution dt.
3. The neuromorphic visual target classification system based on an input weighted impulse neural network of claim 2, wherein the data aggregation process is performed in two steps, comprising:
first step, based on the temporal resolution dt of the event camera ′ Will t ′ Several events E generated at the moment t′ Assembled into tensor X t′； wherein ,Et′ ＝{e i |e i ＝[x i ,y i ,t ′ ,p i ]}，X t′ ∈R H×W×2 ；
Second, based on the set time resolution dt, using formula X t ＝f(X ′ t ) Generating event frame tensor X at time t t ∈R H ×W×2 The method comprises the steps of carrying out a first treatment on the surface of the Wherein dt=β×dt ′ Beta is a polymerization time factor; x is X ′ t ＝{X t′ |t′∈[β×t,β×(t+1)-1]-a }; f is an accumulation operation, a weighted accumulation operation, or an and or operation.
4. The neuromorphic visual target classification system based on an input weighted impulse neural network of claim 1, wherein the learning module comprises a feed-forward network computing unit, an error back propagation unit and a weight updating unit, and specifically comprises:
the feedforward network computing unit selects T event frames from an event frame sequence converted from single event stream data by using a random time clipping method to be input by the feedforward network computing unit, wherein the T event frames are described by tensors; specifically, in single spatio-temporal pulse event stream data, T is generated at a temporal resolution dt total From which T event frames are randomly extracted as inputs to a feed forward network computation unit, T<T total ；
Calculating an output pulse sequence according to the input weighting unit, the pulse neural layer unit and the network connection mode in sequence, and then calculating an output target according to the output pulse sequence by the perceptron neuron output layer;
the error back propagation unit calculates an error between the output target and the set target according to the set loss function and performs back propagation;
the weight updating unit updates the weight according to the set learning rate and the error.
5. The neuromorphic visual target classification system based on the input weighted impulse neural network of claim 1, wherein the reasoning module is composed of a model loading unit and a feedforward network computing unit; the model loading unit reads the input weighted impulse neural network structure configured by the network construction module, loads the model file generated by the learning module and obtains a trained input weighted impulse neural network model; the feedforward network computing unit computes an input weighting and a pulse neuron layer in the pulse neural network model according to an event stream or an event frame provided by the data preprocessing module as input, and then an output target is obtained by the perceptron neuron output layer according to output pulses of the pulse neuron layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110025992.3A CN112712170B (en) | 2021-01-08 | 2021-01-08 | Neuromorphic visual target classification system based on input weighted impulse neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110025992.3A CN112712170B (en) | 2021-01-08 | 2021-01-08 | Neuromorphic visual target classification system based on input weighted impulse neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112712170A CN112712170A (en) | 2021-04-27 |
| CN112712170B true CN112712170B (en) | 2023-06-20 |
Family
ID=75548618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110025992.3A Active CN112712170B (en) | 2021-01-08 | 2021-01-08 | Neuromorphic visual target classification system based on input weighted impulse neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112712170B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113408671B (en) * | 2021-08-18 | 2021-11-16 | 成都时识科技有限公司 | Object identification method and device, chip and electronic equipment |
| CN114429491B (en) * | 2022-04-07 | 2022-07-08 | 之江实验室 | Pulse neural network target tracking method and system based on event camera |
| CN114723009B (en) * | 2022-04-12 | 2023-04-25 | 重庆大学 | Data representation method and system based on asynchronous event stream |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111462268A (en) * | 2020-03-31 | 2020-07-28 | 北京市商汤科技开发有限公司 | Image reconstruction method and device, electronic equipment and storage medium |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150278685A1 (en) * | 2014-03-31 | 2015-10-01 | Qualcomm Incorporated | Probabilistic representation of large sequences using spiking neural network |
| CN107403154B (en) * | 2017-07-20 | 2020-10-16 | 四川大学 | Gait recognition method based on dynamic vision sensor |
| US10992887B2 (en) * | 2017-09-28 | 2021-04-27 | Apple Inc. | System and method for event camera data processing |
| CN111582445B (en) * | 2020-04-24 | 2023-05-26 | 浙江大学 | Efficient learning system and method based on impulse neural network |
| CN111612136B (en) * | 2020-05-25 | 2023-04-07 | 之江实验室 | Neural morphology visual target classification method and system |
| CN112001353B (en) * | 2020-09-03 | 2023-02-17 | 杭州云栖智慧视通科技有限公司 | Pedestrian re-identification method based on multi-task joint supervised learning |
-
2021
- 2021-01-08 CN CN202110025992.3A patent/CN112712170B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111462268A (en) * | 2020-03-31 | 2020-07-28 | 北京市商汤科技开发有限公司 | Image reconstruction method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112712170A (en) | 2021-04-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112712170B (en) | Neuromorphic visual target classification system based on input weighted impulse neural network | |
| CN112633497B (en) | Convolutional impulse neural network training method based on re-weighted membrane voltage | |
| Jia | Investigation into the effectiveness of long short term memory networks for stock price prediction | |
| Wu et al. | Training spiking neural networks with accumulated spiking flow | |
| Cordone et al. | Learning from event cameras with sparse spiking convolutional neural networks | |
| CN114998659B (en) | Image data classification method for training impulse neural network model on line along with time | |
| CN113205048B (en) | Gesture recognition method and system | |
| Zambrano et al. | Efficient computation in adaptive artificial spiking neural networks | |
| Xu et al. | Spike trains encoding and threshold rescaling method for deep spiking neural networks | |
| Zhang et al. | Efficient spiking neural networks with logarithmic temporal coding | |
| CN114186672A (en) | Efficient high-precision training algorithm for impulse neural network | |
| Rady | Shannon Entropy and Mean Square Errors for speeding the convergence of Multilayer Neural Networks: A comparative approach | |
| CN112288080A (en) | Pulse neural network-oriented adaptive model conversion method and system | |
| CN114266351A (en) | Pulse neural network training method and system based on unsupervised learning time coding | |
| CN109635938A (en) | A kind of autonomous learning impulsive neural networks weight quantization method | |
| CN112232440A (en) | Method for realizing information memory and distinction of impulse neural network by using specific neuron groups | |
| CN115860215A (en) | Photovoltaic and wind power generation power prediction method and system | |
| CN115346096A (en) | Pulse neural network model constructed based on memristor | |
| CN114091659A (en) | Ultralow-delay pulse neural network based on space-time information and learning method | |
| CN116822592A (en) | Target tracking method based on event data and impulse neural network | |
| CN116629332A (en) | Signal compensation method based on optical reserve tank calculation | |
| CN111797968B (en) | Reweighting sparse reconstruction method based on impulse neural network | |
| Rafik et al. | Learning and Predictive Energy Consumption Model based on LSTM recursive neural networks | |
| CN116109489A (en) | Denoising method and related equipment | |
| Gerlinghoff et al. | Desire backpropagation: A lightweight training algorithm for multi-layer spiking neural networks based on spike-timing-dependent plasticity |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |