CN112699956B - Neuromorphic visual target classification method based on improved impulse neural network - Google Patents

Abstract

The invention discloses a neuromorphic visual target classification method based on an improved impulse neural network. The method includes: S1: Obtain neuromorphic visual target classification data set; S2: Pulse event stream serialization aggregation: aggregate the spatiotemporal pulse event stream data in the data set into a new event frame sequence according to the set time resolution dt. Data; S3: Construct an improved spiking neural network model: improve the synaptic connection method of Leaky Integrity and Fire (LIF) spiking neurons in the time dimension, and build an improved spiking neural network based on the improved LIF neuron layer; S4: For the data set after serialization and aggregation of the impulse event stream, samples are randomly selected from the sequence as input to train and test the constructed improved impulse neural network; S5: Save the trained improved impulse neural network structure and network parameters. The present invention can effectively improve the network classification accuracy problem in target recognition and classification of neuromorphic vision.

Description

A neuromorphic visual target classification method based on improved spiking neural network

Technical field

The invention belongs to the field of deep learning in machine learning, and specifically relates to a neuromorphic visual target classification method based on an improved impulse neural network.

Background technique

In recent years, deep learning, represented by Artificial Neural Networks (ANNs), has achieved great success in fields such as image recognition and natural language processing. However, due to the differences between the operating mechanism of artificial neural network models and the actual observations in biology, The operating mechanism of the brain is fundamentally different, making it difficult to achieve true strong artificial intelligence. Spiking Neural Networks (SNNs) use a more biologically interpretable spiking neuron model as the basic unit. Compared with artificial neural networks based on pulse frequency encoding information, they have the advantages of low latency and low energy consumption, and can Simulating various neural signals and arbitrary continuous functions is an effective tool for complex spatiotemporal information processing.

The neuromorphic visual event stream based on "events (pulses)" is generated by an event camera designed based on bionics principles by asynchronously measuring the brightness change of each pixel. The event stream is extremely sensitive to the time, location and brightness changes of the brightness change. Sexuality is coded. Compared to traditional cameras, event cameras offer attractive performance: high temporal resolution (microsecond level), very high dynamic range (140dB vs 60dB), high pixel bandwidth (order of kHz), thus effectively reducing motion Vague. Since SNNs have ultra-low latency characteristics in processing input pulses, spiking neural networks are usually used to process the visual event stream, thereby fully utilizing the high time resolution of the visual event stream for real-time output (microsecond-level delay output). However, the existing typical spiking neural network structure cannot fully utilize the sample capacity of spiking event stream data, so its performance in visual target classification is not as good as that of artificial neural networks.

Object recognition and classification is an important task in neuromorphic vision. Currently, there are three main types of existing neuromorphic visual target recognition and classification learning algorithms:

(1) Unsupervised algorithm for pulse time-dependent plasticity based on temporal coding. The temporal encoding method allows this learning algorithm to have an output that is almost synchronized with the input. However, the time-dependent plasticity learning algorithm mainly changes synaptic weights through local neuron activity, making it difficult to achieve high performance.

(2) Non-direct training supervision algorithm from ANN to SNN based on frequency coding. By mapping the trained ANNs model to the corresponding SNNs, this learning algorithm can currently obtain network performance comparable to ANNs. However, this conversion method does not utilize the time information in the event stream, and there are many constraints. For example, the activation function can only use ReLU, etc.

(3) Error backpropagation direct training supervision algorithm based on frequency coding. Synaptic weights are learned by backpropagating training errors in time and space. The representative algorithm is the time-based backpropagation algorithm (Backpropagation-Through-Time, BPTT). However, due to reasons such as the non-differentiability of the impulse function, deep SNNs have problems such as gradient disappearance (explosion) during training, making it difficult to obtain good network performance.

The existing neuromorphic visual target classification methods based on spiking neural networks, regardless of which one is the time encoding method or the frequency encoding method, the spiking neural network structure (the network is shallow due to the gradient problem) and the training method (the conversion method cannot The inherent problems in using temporal information make it difficult to improve the accuracy of neuromorphic visual target classification. In the existing technology, the spiking neural network based on the LIF (Leaky Integrity and Fire) neuron model uses the error backpropagation algorithm for supervised learning, which allows researchers to use GPUs for accelerated training and can also be used in deep learning. Mature Pytorch and other training tools. Generally, a deeper network structure is a necessary condition to improve network performance. However, the gradient problem caused by non-differentiable impulses in training makes it difficult for deep impulse neural networks to converge during training.

Contents of the invention

In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a neuromorphic visual target classification method based on an improved spiking neural network. This method achieves high-performance neuromorphic visual target classification by re-aggregating input data and improving the basic structure of the spiking neural network neuron model. The method proposed by the present invention first aggregates the pulse event stream data into an event frame sequence, and uses a time random clipping method to determine the data input to the network when training the network, which is equivalent to increasing the amount of data available for network training; then, an improved system based on Improving the spiking neural network of LIF neurons avoids training a deep spiking neural network by strengthening the network's ability to extract spatiotemporal features in longer time series, making the shallow improved spiking neural network also available in neuromorphic visual target classification tasks Higher accuracy.

In order to achieve the above objects, the present invention adopts the following technical solutions:

A neuromorphic visual target classification method based on an improved spiking neural network. This method first aggregates the spiking event stream data into a sequence of event frames. When training the network, a time random cropping method is used to determine the data input to the network. This is equivalent to increasing the available data. The amount of data for network training; then we improved the construction of a spiking neural network based on improved LIF neurons. By strengthening the network's ability to extract spatiotemporal features in a longer time series, we avoided training a deep spiking neural network, making shallow improvements to spiking neural networks. The network can also achieve high accuracy in neuromorphic visual object classification tasks.

A further improvement of the present invention is that the method specifically includes the following steps:

S1: Obtain neuromorphic visual object classification data set;

S2: Pulse event stream serialization aggregation: Aggregate the spatiotemporal pulse event stream data in the neuromorphic visual target classification data set into new event frame sequence data according to the set time resolution dt;

S3: Construct an improved spiking neural network model: improve the synaptic connection method of leakage-accumulation-emitting LIF spiking neurons in the time dimension, and build an improved spiking neural network based on the improved LIF neuron layer;

S4: For the data set after the serialization and aggregation of the pulse event stream, samples are randomly selected from the sequence as input, and the improved pulse neural network constructed is trained and tested;

S5: Save the trained improved spiking neural network structure and network parameters. The network parameters are the parameters required for the corresponding neuromorphic visual target classification.

A further improvement of the present invention is that step S2 is specifically carried out in three steps:

The first step is to obtain the spatio-temporal pulse event stream in the neuromorphic visual target classification data set, which is determined by the set E={e _i |e _i =[x _i ,y _i ,t′ _i ,pi _] }; where e _i is the pulse The i-th pulse event in the event stream, (xi _, y _i ) is the pixel coordinate of the i-th pulse event, t′ _i is the timestamp of the i-th pulse event in the entire time stream, p _i is the i-th pulse event The light intensity change polarity of each pulse event, the time resolution of the asynchronous pulse event stream transmitted by the event camera is dt′, and the spatial resolution is H×W;

In the second step, based on the pulse event stream output by the event camera, event frames with a time resolution of dt′ are aggregated. Taking aggregation at time t′ as an example, several event sets E _t′ generated at time t′ are assembled into a tensor X _t′ , where, E _t′ = {e _i |e _i = [x _i ,y _i ,t′, p _i ]}, X _t′ ∈R ^H×W×2 ;

The third step is to generate the event frame tensor X _t ∈R ^H×W×2 at time t based on the event frame with time resolution dt′, X _t =f(X′ _t ), where dt=β×dt′ _, β is the aggregation _time factor;

A further improvement of the present invention is that in step S3, the improved mathematical expression of the LIF pulse neuron layer is:

Among them, x ^t,n-1 is the event frame input by the n-1th layer at time t, h ^t-1,n is the internal state quantity of the nth layer at time t-1, u ^t,n is the nth layer at time t The membrane potential of , x ^t,n is the spatial output from the nth layer at time t to the next layer, h ^t,n is the time output from the nth layer at time t to the next time.

A further improvement of the present invention is that the network forming the model of RNNs_Model is a recurrent neural network;

The mathematical expression of the LIF_Model is:

Among them, function g is a step function, Represents Hadamard product;

Based on the improved LIF spiking neuron layer, LIF neuron layer and fully connected layer, an improved spiking neural network is constructed.

A further improvement of the present invention is that step S4 specifically includes:

When training the improved spiking neural network, if the event frame data generates a total of T _total event frames according to the set time resolution dt, and T event frames are randomly selected from them as training input, T < T _total ;

When testing the improved spiking neural network, the sliding window method is used to preprocess the event frame sequence in the time dimension, and pre-segmented fragments of a fixed size T are obtained as test data; the pre-segmented fragments are set to m, and the same data is The pre-segmented m test data are input into the network to obtain m prediction results. When more than m/2 results are predicted correctly, the test data is judged to be correctly predicted.

Compared with the existing technology, the present invention at least has the following beneficial technical effects:

1. In the learning system of the present invention, by re-aggregating the event stream data into event frame data (tensor data), the GPU can be used as a training acceleration tool and deep learning platform languages such as Pytorch can be used in the learning and reasoning of the impulse neural network. tool.

2. In the learning system of the present invention, T event frames are obtained as training input by using the random time clipping method for network training. This is equivalent to increasing the amount of data available for training, and therefore can effectively improve the obtained network model. performance.

3. The present invention provides an improved LIF neuron model. The improved LIF neuron model is used to form a fully connected (convolutional) spiking neural network. Compared with the spiking neural network based on the LIF model, the target recognition and classification accuracy of neuromorphic vision can be greatly improved.

Description of the drawings

Figure 1 is a schematic flow chart of a neuromorphic visual target classification method based on an improved spiking neural network provided by the present invention.

Figure 2 is an output of an event stream data (fish) in the neuromorphic vision data set CIFAR10-DVS after passing through the pulse event stream serialization and aggregation module. That is, a 100,000 microsecond pulse event stream is aggregated into 9 event frames with a resolution of 10 milliseconds.

Figure 3 shows a network construction method for improving the spiking neural network.

Figure 4 is a schematic diagram of the LIF spiking neuron model.

Figure 5 is a schematic diagram of the improved LIF spiking neuron (RNNs_LIF) model.

Figure 6 is a schematic diagram of an improved LIF spiking neuron (RNN_LIF) model.

Figure 7 is a schematic diagram of an improved LIF spiking neuron (LSTM_LIF) model.

Figure 8 is a schematic diagram of an improved LIF spiking neuron (GRU_LIF) model.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

The purpose of the present invention is to provide a neuromorphic visual target classification method based on an improved impulse neural network to effectively solve the problem of neuromorphic visual target recognition and classification. In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

Figure 1 is a schematic flow chart of a neuromorphic visual target classification method based on an improved spiking neural network provided by the present invention, including:

S1: Obtain neuromorphic visual object classification data set.

S2: Pulse event stream serialization aggregation. The spatio-temporal pulse event stream data in the data set is aggregated into new event frame sequence data according to the set time resolution dt. This is done in three steps:

The first step is to obtain the spatiotemporal spike event stream in the neuromorphic visual object classification data set. The spatio-temporal pulse event stream contains multiple spatio-temporal pulse events and is described using an address event expression protocol. Specifically, the spatio-temporal pulse event stream is determined by the set E = {e _i | e _i = [xi _{, yi} , t′ _i , p _i ]}; where _ei is the i-th pulse in the pulse event stream event, (xi _, y _i ) is the pixel coordinate of the i-th pulse event, t′ _i is the timestamp of the i-th pulse event in the entire time stream, p _i is the light intensity change extreme of the i-th pulse event sex. Among them, the time resolution of the asynchronous pulse event stream transmitted by the event camera is dt′, and the spatial resolution is H×W.

In the second step, based on the pulse event stream output by the event camera, event frames with a time resolution of dt′ are aggregated. The event frame sequence is described using tensors. Specifically, taking the aggregation at time t′ as an example, several event sets E _t′ generated at time t′ are assembled into a tensor X _t′ . Among them, E _t′ ={e _i |e _i =[x _i ,y _i ,t′, _pi ]}, X _t′ ∈R ^H×W×2 .

The third step is to generate an event frame with time resolution dt based on the event frame with time resolution dt′. That is, the event frame tensor X _t ∈R ^H×w×2 at time t is generated using the formula X _t =f(X′ _t ). Among them, dt=β×dt′, β is the aggregation time factor; X′ _t ={X _t′ |t′∈[β×t,β×(t+1)-1]}; f is a general aggregation function , which can be an accumulation operation, a weighted accumulation operation, or an "AND or NOT" operation.

When dt′=1μs and β=3, an example of the “AND” aggregation method is shown in the figure below:

Figure 2 shows a piece of data (fish) in the CIFAR10-DVS data set. When the input is a pulse event stream of 900,000 microseconds, the aggregation method is the "OR" operation, and the aggregation time factor is 10,000, the pulse event stream sequence ize the output of the aggregation module. That is, Figure 2 shows the event stream aggregated into 9 event frames with a resolution of 10 milliseconds.

S3: Construct an improved spiking neural network model: improve the synaptic connection method of LIF spiking neurons in the time dimension, and build an improved spiking neural network based on the improved LIF neuron layer. By combining the improved LIF spiking neuron layer, LIF neuron layer and fully connected layer in a certain network connection method, the improved spiking neural network required for training and testing is constructed. Figure 3 shows a network connection method that improves the formation of LIF spiking neuron layer, LIF neuron layer and fully connected layer.

As shown in Figure 4, the LIF neurons constructing the classic spiking neural network can be described by the following formula:

Among them, function g is a step function, Represents the Hadamard product, x ^t,n-1 is the event frame input to the n-1th layer at time t, h ^t-1,n is the internal state quantity of the nth layer at time t-1; u ^t,n is the membrane potential; The membrane potential is compared to the neuron threshold u _th , and x ^t,n is passed to the next layer as the spatial output, and h ^t,n is passed to the next moment as the temporal output of the LIF neuron.

LIF neurons pass the formula To aggregate the spatial input x ^t,n-1 and the internal state of the neuron at the previous moment h ^t-1,n . The aggregated result u ^t,n is the membrane potential of the LIF neuron. This spatiotemporal information aggregation method of LIF neurons is actually a direct connection in time. Its disadvantage is that when aggregating the membrane potential of a single LIF neuron, only the intermediate state of this neuron at the previous moment is aggregated. , and does not consider the impact of the intermediate states of other neurons in the same layer on the neuron membrane potential.

In the present invention, by improving the acquisition method of membrane potential in LIF neurons, a type of improved LIF neuron model is obtained. As shown in Figure 5, the present invention uses Recurrent Neural Networks (RNNs) that process sequence data to aggregate the spatial input x ^t,n-1 and the internal state of the neuron at the previous moment h ^t-1,n . The mathematical expression of the improved LIF neuron layer model is:

Among them, x ^t,n-1 is the event frame input by the n-1th layer at time t, h ^t-1,n is the internal state quantity of the nth layer at time t-1; x ^t,n-1 and h ^{t -1,n} are used as the spatial input and time input of RNNs_Model respectively, and one output of RNNs_Model is selected as the membrane potential u ^t,n ; the membrane potential is compared with the neuron threshold, and x ^t,n is passed to the next as the spatial output. One layer, h ^{t ,n} is passed to the next moment as the temporal output of LIF neurons.

The RNNs_Model can be a recurrent neural network (Recurrent Neural Network, RNN), a long short-term memory network (LSTM, Long Short-Term Memory), a bidirectional recurrent neural network (Bidirectional RNN, Bi-RNN) and a gated recurrent unit network (Gated Recurrent Unit networks (GRU) and other recurrent neural network models.

The expression of the emission or leakage mechanism (LIF_Model) is:

Among them, function g is a step function, Represents the Hadamard product.

As shown in Figure 6, an RNNs_Model uses the RNN model to aggregate x ^t,n-1 and h ^t-1,n to obtain the membrane potential u ^t,n , and the RNN_LIF neuron model can be obtained:

As shown in Figure 7, a RNNs_Model uses the LSTM model to aggregate x ^t,n-1 and h ^t-1,n to obtain the membrane potential u ^t,n , and the LSTM_LIF neuron model can be obtained:

As shown in Figure 8, a RNNs_Model uses the GRU model to aggregate x ^t,n-1 and h ^t-1,n to obtain the membrane potential u ^t,n , and the GRU_LIF neuron model can be obtained:

In the model mentioned above, the operation between the weight matrix W and the input intermediate state vector h or the spatial input vector is matrix multiplication. At this time, the network model is an improved spiking neural network. When the operation between the weight matrix W and the input intermediate state vector h or spatial input vector is a convolution operation, the network model becomes an improved convolutional impulse neural network.

S4: Training and testing of the improved spiking neural network constructed on the dataset after serialization and aggregation of spiking event streams.

When training the improved spiking neural network, a random spatiotemporal clipping method is used to select T event frames from all event frame data as neural network module training inputs, and the T event frames are described by tensors. That is, in a single spatiotemporal pulse event stream data, a total of T _total event frames can be generated according to the set time resolution dt, from which T (T < T _total ) event frames are randomly selected as training input.

When testing the improved spiking neural network, the sliding window method is used to preprocess the event frame sequence in the time dimension, and pre-segmented fragments of a fixed size T are obtained as test data. The number of pre-segmentation fragments is set to m. If m pieces of test data pre-segmented from the same data are input into the network, m prediction results can be obtained. When more than m/2 results are predicted correctly, the test data prediction is judged to be correct.

S5: Save the trained improved spiking neural network structure and network parameters. The network parameters are the parameters required for the corresponding neuromorphic visual target classification.

In order to better illustrate the beneficial effects of the present invention, the experimental results of the method of the present invention on the neuromorphic visual target classification data set DVS128 Gesture are given below.

On the DVS128 Gesture data set, we set experimental results in fully connected spiking neural networks and convolutional spiking neural networks under different time resolutions and when the event frame sequence T=60. As can be seen from the above two tables, whether it is based on a fully connected spiking neural network or a convolution-based spiking neural network, the network performance of LIF-LSTM has improved at all time resolutions compared with the classic SNN. In addition, when dt is small, the improvement effect of LIF-LSTM is more obvious, which shows that LSTM-LIF can maintain high performance while having lower latency than classic SNN.

What are listed above are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments, and many modifications are possible. All modifications that a person of ordinary skill in the art can directly derive or associate from the disclosure of the present invention should be considered to be within the protection scope of the present invention.

Claims (2)

1. A neuromorphic visual target classification method based on an improved spiking neural network, which is characterized in that the method first aggregates the spiking event stream data into an event frame sequence, and uses a time random clipping method to determine the data input to the network when training the network, This is equivalent to increasing the amount of data available for network training; then we improved the construction of a spiking neural network based on improved LIF neurons, which avoids training a deep spiking neural network by strengthening the network's ability to extract spatiotemporal features in longer time series. This enables the shallow improved spiking neural network to achieve higher accuracy in neuromorphic visual target classification tasks; the method specifically includes the following steps: S1: Obtain neuromorphic visual object classification data set; S2: Pulse event stream serialization aggregation: Aggregate the spatio-temporal pulse event stream data in the neuromorphic visual target classification data set into new event frame sequence data according to the set time resolution dt; this is done in three steps: The first step is to obtain the spatio-temporal pulse event stream in the neuromorphic visual target classification data set, which is determined by the set E={e _i |e _i =[x _i ,y _i ,t _i ^′ ,pi _] }; where e _i is the pulse For the i-th pulse event in the event stream, (xi _, y _i ) is the pixel coordinate of the i-th pulse event, ^{ti ′} _is the timestamp of the i-th pulse event in the entire time stream, and p _i is the i-th pulse event. The light intensity change polarity of each pulse event, the time resolution of the asynchronous pulse event stream transmitted by the event camera is dt ^′ , and the spatial resolution is H×W; The second step is to aggregate event frames with a time resolution of dt ^′ based on the pulse event stream output by the event camera. Taking the aggregation at time t ^′ as an example, assemble several event sets E _t′ ^generated at time t ′ into a tensor X _t′ , where E _t′ ={e _i |e _i =[x _i ,y _i ,t ^′ ,p _i ]}, X _t′ ∈R ^H×W×2 ; The third step is to generate the event frame tensor X _t ∈R ^{H ×W×2} at time t based on the event frame with time resolution dt ^′ , X _t =f(X ^′ _t ), where dt=β×dt ^′ _, β _is the aggregation time factor ^; S3: Construct an improved spiking neural network model: improve the synaptic connection method of leakage-accumulation-emitting LIF spiking neurons in the time dimension, build an improved spiking neural network based on the improved LIF neuron layer; improve the mathematical expression of the LIF spiking neuron layer for: Among them, x ^t,n-1 is the event frame input by the n-1th layer at time t, h ^t-1,n is the internal state quantity of the nth layer at time t-1, u ^t,n is the nth layer at time t The ^membrane potential ^of The network forming the model of RNNs_Model is a recurrent neural network; The mathematical expression of the LIF_Model is: Among them, function g is a step function, Represents Hadamard product; Based on the improved LIF spiking neuron layer, LIF neuron layer and fully connected layer, an improved spiking neural network is constructed; S4: For the data set after the serialization and aggregation of the pulse event stream, samples are randomly selected from the sequence as input, and the improved pulse neural network constructed is trained and tested; S5: Save the trained improved spiking neural network structure and network parameters. The network parameters are the parameters required for the corresponding neuromorphic visual target classification. 2. A neuromorphic visual target classification method based on an improved spiking neural network according to claim 1, characterized in that step S4 specifically includes: When training the improved spiking neural network, if the event frame data generates a total of T _total event frames according to the set time resolution dt, and T event frames are randomly selected from them as training input, T < T _total ; When testing the improved spiking neural network, the sliding window method is used to preprocess the event frame sequence in the time dimension, and pre-segmented fragments of a fixed size T are obtained as test data; the pre-segmented fragments are set to m, and the same data is The pre-segmented m test data are input into the network to obtain m prediction results. When more than m/2 results are predicted correctly, the test data is judged to be correctly predicted.